Roadmap & Releases

Release 2024.08.0

• Breaking Changes
  • The scheduler was updated for the real time executor. When under pressure, the real time executor previously would run callbacks for a stage in a round-robin fashion. It nows runs them in publish-time order (oldest first). This produces more intuitive behavior when you have a callback with many small updates (such as pose) and another with large updates (autonomy data). The pose updates that occurred before the autonomy data was published are now received before the autonomy data in all cases.
  • The scheduler was updated for the simclock executor. Previously, stage callbacks were run in the order that they were added to the stage interface. When using expected runtimes, this could cause some callbacks to starve others. The scheduler was updated to model the real time executor’s algorithm. The pipeline was deterministic in either case, although your results may change slightly when migrating to the new executor.
  • nlohmann::json was removed from ark::serialization. Anything using nlohmann::json implicitly will need to be updated (both to include the appropriate nlohmann libraries, and to convert from the the ark::serialization JSON into nlohmann).
  • Upgraded to Clang 18.1.8 (and a number of third party dependencies got minor revisions). Most notable is an update to fmt, which improves build times for everything using fmt (including the Ark debuglog).
• Serialization Changes
  • nlohmann::json was removed in rbuf in favor of yyjson. This has a significant reduction in compile times, a significant improvement in JSON parsing performance, and now supports NaN and inf values properly. If you were depending on nlohmann’s JSON in the ark::serialization::JsonBlob class, it’s recommended to either migrate to the new API, or re-parse the JSON (via str()) into nlohmann’s API, depending on your use case.
  • Added new JSON patch and merge patch API support to ark::serialization. These APIs allow you to easily patch an object with a fragment of JSON (see RFC 6902 and 7386 for more details).
  • Added a new ‘intermediate representation’ to rbufc, making it possible to get the parsed results of an rbuf file as JSON. This can be helpful for building tools to lint or introspect rbuf files.
• Build System Improvements
  • Implicit dependencies on rbuf targets were removed in the CMake build system, significantly increasing parallelism of the build. On machines with high numbers of cores, this can reduce build times significantly.
  • Added a new flag that disables optimizations in unit tests. This has to be enabled manually, but can result in a large build-time speed up.
• Logging Improvements
  • The system will now allow up to 256 channels in a single split file. The previous limit was 238. Error messages when the limit is exceeded have also been improved.
  • The logger will now automatically columnize data if you have more than 256 channels in a single column. This makes it easier to have large number of channels in a column without needing manual bookkeeping.
• Tooling Improvements
  • Added support to ark-spy to dump in the JSON lines format. This can make it much cheaper to run jq over large arrays or batches of messages (thanks Omar Salama).
• Stage Changes
  • The VideoCaptureStage now handles more video control types, including menu types. It provides more feedback when a type can’t be processed, or controls are disabled via configuration.
  • Made the NtpClientStage more strict on what NTP packets it considers trustworthy. This information is now exported in the stage’s statistics.
  • The HttpServerStage will now timeout lagging ’throttled websocket clients’. Previously it relied on the client to realize it wasn’t communicating (or the write buffer simply filled up). In situations where it could take a long time for the write buffer to fill up, or the client didn’t realize it was behind, it would appear as if the channel had stalled.
  • The PeriodicProducerStage pipeline testing helper will now stop running after it has gotten through its configured object list. Previously it used a periodic timer, which meant if you had a single message to send, and the timer was configured to run in 1ns, the executor would try to run it indefinitely.
  • Added a new SingleShotProducerStage pipeline testing helper for pushing out a single message on startup.
• API Changes
  • The ark::comms::HttpHelper class was extended to support non-filename multipart bodies.
  • The ark::pipeline::Subscriber class can now invoke callbacks with the following signatures: shared pointer, shared pointer with publish timestamp, const value, and const value with publish timestamp. Can reduce some boilerplate (thanks Matt Langford).
  • You can now load config packages from strings. Previously these had to be on disk. This can be useful for cases where the config package is self-contained in one file (doesn’t reference anything else).
  • Added ability to load a config package directly from local storage. This allows you to retrieve a number of configs stored entirely within one local storage key, all at once.
• Catalog Changes
  • The ark-batch-tool process now allows you to override the container image, making it easier for batch sim jobs to specify the exact container they want to run.
  • You can now download attachments with the ark-query-attachments tool.
• Supervisor Changes
  • You can now chain commands together in ark-supervisor-tool, making it easier to stop/start processes, or “deploy and then start”.
  • Added ability to reject releases if they have an incorrect category. This prevents deploying a software release as an ArkOS release, or vice-versa.
  • Releases are now rejected if they have an incorrect architecture. For example, if you tried to deploy an x86_64 image to an aarch64 machine, this will now fail.
  • You can now specify the docker library URL to pull images from (thanks Anibal Limon).
• Bug Fixes
  • Fixed a bug in the HttpServerStage that could result in some connections to a particular client being throttled when another connection to that same client was experiencing a specific class of write errors. This manifested as sporadic reconnects.
  • When using a LogAmenderStage in a real time pipeline, it was possible to run into a race condition that would cause an unhandled exception on startup. This was addressed.
  • Addressed an issue where the PcapParser could incorrectly identify a fragment as out of order if the ‘finalizing fragment’ in a fragmented UDP packet was dropped.
  • Fixed a bug with double-registering metadata with a dictionary in the config package (thanks skyler).
  • Fixed a bug with deploying releases when the underlying hardware has had an incorrect clock (specifically, far in the future). This caused the just-deployed release to be pruned from the ring buffer.
  • Fixed a performance-loss bug in the annotation plugin in the Qt GUI.
  • Fixed a bug with the grid builder scale calculation, causing it to be slightly offset.

Release 2024.04.0

• Breaking Changes
  • StageInterface is no longer passed to start() calls. This was extremely misleading, as most of the APIs exposed by StageInterface were non-functional or broken if invoked in start. The primary use case driving this was setting the preferred time of the pipeline, which is now settable in a topology change callback instead.
  • PCL was upgraded to 1.14 (thanks Cameron Kisailus).
• API Improvements
  • Added concept of ’local storage’ to Ark. This allows you to have a structured method for storing data locally on your robots (for example, calibrations, per-robot configurations, or other settings). This storage is signed and verified before loading, and can be accessed either at initialization time (when signed with RSA keys) or runtime (when signed with HMACs or RSA keys).
  • Added helper APIs for reading SMART data from NVME drives. A future release will include a stage to publish and log SMART data.
  • Added a helper API to generate unique ‘codes’ (rather than GUIDs). These look like sequences of numbers, and can be useful for when you want a unique (or mostly unique) identifier, and don’t want to use a GUID.
  • Added ability to set mutually exclusive options in the Ark command line parser, allowing you to say “option A or B must be specified (at least one, but not both at the same time).”
  • Added new APIs for taking “time of day” into account. This allows you to apply a time of day (such as 8:00 PM) to an existing system_clock timepoint in an arbitrary timezone, and get back the proper timepoint.
  • Added an API for launching processes from an in-memory buffer. This can be helpful for when you wish to transmit and launch an executable without writing to disk.
  • Remote API calls (such as those made to the Catalog or other similar services) can now be updated to use ByteBuffer as a direct return value. This can be useful for when you want to return a large binary blob without encoding it to JSON.
  • Added OpenAI APIs to support chat messages and text-to-speech requests. These are thin wrappers over their REST APIs.
• Pipeline Improvements
  • Pipeline subscriber callbacks can now accept plain types (in addition to std::shared_ptr-wrapped types) (thanks Matt Langford).
  • Improved error messages when pushing data in the SimClockExecutor fails. This typically occurred when you tried to push JSON data onto something expecting binary data, and deserialization failed.
  • Added ability for the real time executor to have a preferred start time. This means that during playback, the pipeline time will be approximately correct when using the real time executor (rather than starting at zero). Note that the clock will drift over time from log time.
• CANbus Enhancements
  • The COB extended CAN ID flag is now set properly for PDOs.
• Build System Improvements
  • Added initial Bazel support (thanks Juan Ortega). There are a number of patches that need to be applied yet, but the skeleton for Bazel support is present.
  • Added a --build-type command line flag to make.sh to specify the CMAKE_BUILD_TYPE more easily (thanks Juan Ortega).
  • Added support for multiple config packages in add_container.
• Supervisor Improvements
  • Added the ability to deploy configuration changes independently of the rest of the software. This allows you to modify your config package locally and execute your software with the modified config package.
  • Added the ability to start processes managed by the Supervisor with custom configuration options, rather than just using the ones baked into your deployable.yml. Particularly useful for testing and avoiding a deploy.
  • Added the concept of ‘production lock’ to the Ark Supervisor. This allows you to lock out deploys and restarts to start/stop, making it less likely that you’ll accidentally deploy or start/stop processes on the wrong robot.
  • Added a quiet flag to the deployment packager, to reduce the amount of chatter it prints when compressing large releases.
  • You can now deploy releases by name in the ark-supervisor-tool, making command lines more readable.
• JavaScript Improvements
  • Added ability to import rbuf files using the Ark JavaScript engine (quickjs). This requires enabling module support, but otherwise allows you to import rbuf files from JavaScript as you would using our esbuild or parcel frameworks.
• Logging Improvements
  • LogWriterPlugin was updated to allow renaming a log after you have already started it. Similarly, the LogWriter class was updated with the ability to rename a log after construction.
  • Added the ARK_LOG_OUTPUT_PATH environment variable to indicate where the log output of the LogWriterStage should go. This allows customization of log output without changing command line flags or configuration, useful for some cloud environments.
  • Improved the output of ark-offload-log to give more status/details.
  • Added support to ark-download-log to download directly from a NAS, rather than being required to go through S3 or Catalog (previously you could playback directly from a NAS, but not download logs from a NAS).
  • LogReader will now prefix channels without a namespace with a ‘/’ (so, ‘statistics’ will become ‘/statistics’).
• Perf Analysis Improvements
  • Scopes are now collected one last time when the TracerCollectorStage is shutting down. Otherwise, some scopes that may have been created between the last collection period and shutdown would be lost.
  • Added support to ark-perftool for returning traces around the midpoint of a log.
  • Added a DUR_TOTAL column to the tracer report to indicate the total amount of time spent in a scope (thanks Matt Langford).
• Catalog Improvements
  • Improved subheadings for artifacts in Catalog, including additional information (such as creation or ingest times) for all artifacts.
  • Robot Secrets are now requested more frequently in the TelemetryStage. Previously these were requested at about once every 15 minutes, we now do it at once per minute instead, allowing configuration changes to take effect much more quickly.
• ArkOS Improvements
  • debootstrap now uses apt instead of dpkg to install Debian packages, allowing for conflict resolution (thanks Anibal Limon).
  • Support for copying generated DTB files for the Nvidia portions of the ArkOS build (thanks Anibal Limon).
  • Added support for forwarding ssh credentials in the ArkOS build (thanks Anibal Limon).
  • The Nvidia portion of the ArkOS build now can use the Nvidia BSP repo (thanks Anibal Limon).
  • Added support for enabling the debug build of the Nvidia UEFI module (thanks Anibal Limon).
  • Support separate top-level repo and Ark repo paths in the Nvidia L4T steps (thanks Ganesh Subramaniam).
• Tooling Improvements
  • Added machine-readable output to ark-version, making it more useful for certain scripts.
  • Added tooling to convert Ark Logs to mcap files.
• GUI Improvements
  • The Qt image widget can now decode JPEG images directly (thanks Brandon Rioja).
  • Added support for automatic transform network reset in the GUI. This is a new toggle which will reset the transform network if it ever detects that time has gone backwards.
• Bug Fixes
  • V4L streams no longer consider the ’error bit’ a fatal error, as some devices set this flag quite frequently, and documentation indicates such frames should just be ignored (ie, the error is recoverable).
  • Fixed a bug that prevented opening of logs with URLs from the Python LogReader class.
  • Fixed a problem with deserializing groups in the JavaScript rbuf decoder. Specifically, if the final group in your schema is transmitted, but the schema on the receive side doesn’t include that group, you would often get an error indicating the data was corrupt.
  • Fixed an issue with decoding group headers with the wrong endian-ness in the JavaScript rbuf decoder, resulting in unknown groups causing deserialization errors.
  • Fixed a bug where old transforms were not pruned from the transform stats tree (thanks Brandon Rioja).
  • Fixed a bug in the Hokuyo driver that caused data to be injected twice when an error range was reported (thanks Rakshith Vishwanatha).
  • Fixed a bug with PDO inhibit configuration (the units were specified in milliseconds instead of microseconds).

Release 2024.01.0

• Breaking Changes
  • Clang was updated to 17.0.6, along with a number of third party libraries (notably, AWS, which saw a fairly large version jump). The built-in C++ SDK credentials provider had problems parsing older credentials file formats, so we now use our INI parser for this work instead.
  • aarch64 builds were previously built with -O2, while x86/64 builds were built with -O3. Now both builds use -O3 to be more consistent (thanks Dmitri Milideev).
  • The always_on deployable configuration in the Platform Supervisor was replaced with a new startup_policy (on startup, on demand) and restart_policy (never, always, or only on failure).
  • Added new third party packages for pulseaudio (and many sound formats), websocketpp, and foxglove.
  • Added support for the ONNX runtime (thanks Aníbal Limón).
  • Python is now enabled on aarch64 host builds (but not cross-builds). Previously, it was only enabled on x86/64 host builds (thanks Aníbal Limón).
• Pipeline Enhancements
  • Added support for dynamic expected runtimes. This allows you to define a callback in your subscriber configuration that can compute the expected runtime of your subscriber callback. For example, you can use this to scale your runtime up or down based on input data size. Traditional static runtimes remain the default.
  • Periodic Timers can now be configured to trigger immediately on startup. The default behavior remains to only trigger after their rate has expired.
  • Fixed a slight race condition in the real time executor that could cause ‘single shot’ timers to fire slightly after other subscriber callbacks are invoked. They will now always fire before subscriber callbacks.
  • Added a --verify-connections flag to ark::main which allows you to verify that all of your publishers/subscribers are connected (without running the pipeline). Useful for integration tests.
• API Improvements
  • Improved ROS compatibility. Added example pipelines for communicating with ROS networks natively, using ROS serialization.
  • Added support and examples for communicating directly with Foxglove, making it possible to use their web visualization against Ark pipelines.
  • Extended serialization support in Ark to now support rosmsg and protobuf formats. This allows you to use these alternate serialization formats in some contexts. Note that not all tools support using these formats yet.
  • Added a new ark::js module, which provides classes that are capable of instantiating a JavaScript runtime and executing JavaScript code from within your C++ code (effectively wrapping QuickJS).
  • Added a new ark::concurrency::CallbackThread class, which allows you to execute functions from a particular thread context. This is useful for when you are using a library that relies on always running in the same thread context.
  • Added support for Unix domain sockets to the HTTP helper APIs. This allows you to make HTTP requests to local daemons that communicate over domain sockets, rather than TCP.
  • The ark::rest::RestService web service backend now supports a configurable endpoint for emitting metrics (thanks Eddie Sasagawa).
  • The supported HTTP request types now supports the PATCH verb (thanks Eddie Sasagawa).
  • Added new helper methods to ark::core::ByteBuffer for copying and wrapping data more intuitively. These are the wrap_data (for when you wish to maintain ownership and do a zero copy wrapping of data) and the copy_data (for when you wish to copy the data and let the byte buffer manage ownership) APIs.
  • Added a new API ark::core::generate_code which allow you to generate random human readable codes with some number of digits (in a base between 2 and 62). This can be used to generate passcodes for users to enter, or any place you need human-readable unique numbers.
  • Added a helper library for subscribing to robot information RobotInformationSubscriber. This makes it easy to fetch things like your configured robot name and serial number.
  • Added improvements to ark::core::CircularBufferIterator for working with const iterators (thanks David Wheeler).
  • Added additional JSDoc types to more types in the JavaScript rbuf codegen (thanks Brandon Rioja).
• Debuglog Improvements
  • Added support for ’throttling’ your debug log statements. New _THROTTLED macros were created for each of the debuglog variants, which allow you to self-throttle statements to a given rate (for example, printing no more than once a second).
  • Added support for printing a given debug log statement once and only once. A new _ONCE macro for each of the debuglog variants was created.
  • You can disable the ‘shutdown on PCAP file complete’ functionality in the PcapPublisherStage now, allowing PCAP-driven pipelines to continue on after the PCAP is finished playing.
  • You can now set the ARK_FORCE_ANSI environment variable to force the use of ANSI color codes, even if we detect that the terminal doesn’t support them.
• Catalog Improvements
  • Added support for storing public keys with Catalog users (similar to Github). These can be made use of in downstream tools (such as ArkOS, see below).
  • You can now pass container tags directly in with image names in the add_container cmake macro (thanks Eddie Sasagawa).
  • Adjusted the number of CPUs that a batch job can request up to 32 vCPUs (thanks Eddie Sasagawa).
  • Support configurable AWS regions in the Catalog terraform (thanks Eddie Sasagawa).
  • Support configurable Google Maps API keys when generating the Catalog Frontend (thanks Eddie Sasagawa).
• ArkOS Improvements
  • Version information is now written out to /etc/arkos/version.json in a machine-readable way. This allows you to easily read the image version and report it back in your pipeline.
  • Added a new config flag enable_catalog_authorized_keys to your ArkOS build configurations. Enabling this allows users to authenticate (and gain sudo access) entirely based on their public/private keys as stored in the Catalog. This reduces the need to have a password on your robots.
• CANbus Enhancements
  • Reworked SDO to be more robust, and added in support for block transfers (thanks Jason McMullan).
• Stage Enhancements
  • The SICK interface gained the ability to request and process firmware version information.
  • The Novatel SPAN interface now does a better job of handling fragmented packets, along with adding support for communicating over serial links (in addition to TCP links).
  • Added a simple stage for talking to the local Tailscale deamon and reporting basic status information on a channel.
  • Robot serial number is now used to construct metric keynames. This is helpful for seperating out your metrics by serial number (particularly when coming from a live robot).
  • The LogWriterStage will now emit statistics even if it is busy writing logs to disk. This can be useful in cases when the log writer is badly falling behind or a disk is locked up.
  • You can now emit kmsg messages to a dedicated channel (in addition to the debuglog) through configuration (thanks Alec Beljanski).
• Python Improvements
  • Python stage subscriber callbacks can now receive the publish timestamps, just like C++ subscriber callbacks.
• Build System Improvements
  • Progress is now emitted when downloading toolchains and third party packages, unless the ARK_DOWNLOAD_PROGRESS environment variable is set to ‘hide’.
  • You can now configure the endpoint for third party package definitions in both the normal build toolchain (thanks Eddie Sasagawa) and package build (thanks Juan Ortega).
  • You can skip building third party packages now with the --no-build option (thanks Juan Ortega).
  • Added a new scripts/coverage.sh script, which allows you to generate coverage for a subset of the unit tests, useful to speed up coverage testing of a particular module (thanks Jason McMullan).
  • The add_container cmake macro now creates a <target_name>-create-repo target, which will automatically create a Catalog Docker repository for your container (thanks Eddie Sasagawa).
  • Added --clean and --expunge to the make.sh script. --clean will delete your build directory, while --expunge will remove your build and toolchain directories (thanks Juan Ortega).
  • We now use aria2 to download third party packages, allowing you to download them much faster on gigabit (or faster) speed connections.
• GUI Enhancements
  • Added a new ‘acknowledge faults’ button to the fault plugin. This will cause previously active faults to show up as ‘yellow’ in the fault plugin, until you click the ‘acknowledge’ button. This is useful for quickly noticing that a fault had been active in the past, even if it was only active briefly.
  • Improved sorting of faults in the fault plugin, making it easier to keep recently active faults at the top of the list.
• Bug Fixes
  • Config-package dependencies in docker containers now properly regenerate when the host config package changes. Previously, these only generated when the dependency tool changed (or on fresh builds).
  • Strings are now encoded/decoded as UTF-8 in the JavaScript rbuf serializer. Previously these were decoded as ASCII, which meant that strings with Unicode characters in them were decoded improperly. Further, we encoded strings as UTF-8, but wrote out the length of UTF-16 codepoints, so there was a chance we would corrupt the rbuf bitstream.
  • Reduced copies used in the ark::gui::render_to_image API, helping to reduce memory usage and improve performance (thanks Brandon Rioja).
  • Fixed a bug where ‘perspective mode’ in the Qt GL GUI had a very wide FOV set. This is now set at 90 degrees, matching the web GL GUI.
  • Reduces the chances that a corrupt rbuf bitstream could lead to an OoM error or exception.
  • Fixed a bug that prevented multiple subscribers to the same channel in the same Python stage. Previously, you could only subscribe to any particular channel once in Python stages.
  • Fixed optional directory C++ codegen in rbuf (thanks Matt Langford).
  • Fixed a bug that caused [[removed]] fields in rbuf to still be emitted (with default values) in JSON output.

Release 2023.09.0

• Breaking Changes
  • Terraform was updated to v1.5.2. There are some changes related to the AWS module that are incompatible. Please look at Terraform documentation for more details.
  • Dictionaries with enum keys in rbuf were previously serialized to JSON as pairs. Now, they are serialized as dictionaries. Old JSON messages containing dictionaries with enum keys will not be deserialized correctly.
  • Updated a number of third party libraries. This includes updating Cairo to make use of enhanced font rendering (including Pango). This does introduce a linking issue when linking against code that uses both Qt and Cairo/Pango. Please avoid linking both of those libraries into a single executable.
  • Ark web tokens previosuly identified themselves as AT. They now identify themselves as JWT, as they were effectively JWT tokens anyway. Any existing tokens will need to be regenerated.
• Stage Enhancements
  • Added a new stage for uploading metrics files (say, parquet files), to the Catalog. This allows you to periodically write metrics to tables, and then store those in S3, and finally, to query those with Athena (or equivalent).
  • Python stages now have the DISPLAY variable set, allowing you to create GUI windows from within Python stages (useful for debugging).
  • New controls were added to disable setting video controls in the VideoCaptureStage (thanks Ganesh Subramaniam).
  • The LogReader can now write data directly to the LogAmender, allowing you to ‘rename’ a channel in playback (thanks Matt Langford).
  • The UdpSocketStage (and underlying ark::comms::Socket interface) gained the ability to bind to a specific interface, making it possible to filter packets at the Linux interface level.
• Catalog Improvements
  • Implemented software token MFA support in the Catalog authentication system, allowing you to use multi-factor authentication (thanks Eddie Sasagawa).
  • Added support for catalog queries when both the version and identifier were specified in the search query string.
• API Enhancements
  • Allow for reading and decoding VLAN-tagged packets in the pcap library (thanks Ganesh Subramaniam).
  • Better handling for fragmented packets in the pcap library (thanks Ganesh Subramaniam).
  • You can now emit parquet files directly to byte buffers, saving you from a trip to disk when generating parquet files.
  • Some of the transform network APIs (get_transform, can_transform, etc) now return an expected result with an error message, making it easier to debug why a transform cannot be retrieved.
  • Added a helper class (ark::time::GlobalTimeSubscriber) for subscribing to and transforming global time within a stage.
  • Added a new API (remote_to_local) to the ark::time::TimeModel class. This allows you to transform a remote (possibly global) timestamp to a local timestamp. This has less precision than going from local to remote, but may still be useful for some use cases.
  • Added a new helper function (ark::logging::pipeline_to_global_time_from_manifest) to translate pipeline time to global time, given a log manifest.
  • The ark::main module’s LogOpenedCallback can now be called with either the manifest (old behavior) or the opened log reader, or the read config package. This reduces the amount of code some pipelines need to use to extract information about a log before setting up their pipeline.
  • Added iterators to ark::core::ByteBuffer, making it possible to use in situations where you need to use begin() or end().
  • Added a new API to the supervisor, download_release, which will simply download a release. Useful to download a release without installing it.
  • The series of REST libraries gained the ability for callbacks to be invoked with the raw bearer token, rather than the parsed Authentication result. Useful for services that need to check an arbitrary bearer token, rather than a JWT.
  • Added a helper library to format strings in any timezone (ark::time::strftime_tz). Uses the ICU library under the hood.
  • C++ GUI plugins can now create their own topology change listeners (just like pipelines) to listen for local-GUI toplogy events (publishers/subscribers created).
• Pipeline Improvements
  • The pipeline publisher/subscriber system will now allow types read from data logs or over comms that have simply had their names changed (but are otherwise equivalent) to be passed through the publisher/subscriber system. This makes it easier to refactor data without needing to rerecord old logs.
  • Added ability to remap channels in pipeline configuration. This allows you to take a channel name as a stage added it in the publisher/subscriber, and remap it to a different name. This can be useful both for debugging and as a better way to control publisher/subscriber connectivity above and beyond namespaces.
• Configuration Improvements
  • Added ability to copy a configuration from one configuration package to another.
• Serialization Improvements
  • Added count and max helper functions to rbuf-generated C++ enum code. This allows you to get a count of all enumerated values (and the largest possible enumerated value).
  • Added a new routine ark::serialization::is_equivalent that will return true if two schemas are equivalent or not. Equivalence is defiend as having the same fields (by type and in order), both in the base structure and in any groups.
  • Enhanced support for constant expressions in rbuf, allowing for the usage of multiple operations in a single declaration (previously only one operation was allowed).
  • Added support for nested arraylists and arrays in rbuf. This allows you to have arrays of arrays (of arrays), rather than needing intermediary objects.
• Tooling Improvements
  • Allow custom playback rates in the log reader stage and GUI. You can now run interactive playbacks faster (or slower) than real time (thanks Matt Langford).
  • Added a packer wrapper script, allowing you to use a ‘hermetic’ version of packer for producing updated AMI images.
  • Objects can now be used as dictionary keys in the rbuf parser. Previously, the wire format allowed this, but the code generation did not.
  • Added a script that makes uploading custom sysroot/toolchains easier.
  • Added a new ark-version-tool to report the versions of the software repositories.
• ArkOS Improvements
  • Applies user-level permissions to the default user’s home directory when building an image. This allows you to overlay /home with proper permissions (thanks Aníbal Limón).
  • Many improvements to support installing packages (via apt-get) as a second stage of the installation process (thanks Aníbal Limón).
  • Added support to build and install a kernel package, useful for when you need to build 3rd party kernel modules, such as Nvidia (thanks Aníbal Limón)
  • The example kernel was updated from 5.15 to 6.1, as 5.15 doesn’t have a very actively maintained RT patchset anymore.
• Build Improvements
  • When building on host aarch64 machines, the build will now create Python virtual environments, just like it does on host x86/64 machines. Previously it disabled Python-related stages on aarch64, assuming you were cross compiling (thanks Aníbal Limón).
  • Added new g2o and gtsam libraries (thanks Samvrutha Tumuluru and Joe Lisee).
• Bug Fixes
  • Fixed a race condition when using Python Stages. Previously, if you had multiple publishers or subscribers, and were publishing data from multiple threads at the same time, you may have run into an unhandled exception. This has been addressed.
  • Fixed a bug in module name resolution in the rbuf Python code generator, that would resolve the path to the module incorrectly depending on the module prefix.
  • Fixed a bug that prevented the use of explicitly namespace’d enumerations as keys in rbuf dictionaries. Previously, only implicitly namespace’d enumerations generated correct code.
  • Fixed a bug that caused releases to always be uploaded to the TBD catalog, due to a flag in the cmake add_release script (thanks Joe Lisee).
  • Fixed a minor race condition in computing maximum latency on a pipeline callback (the time between when a subscriber is ‘ready’ and when it actually executes). This occurred extremely rarely.

Release 2023.06.0

• Breaking Changes
  • The MultiSense SensorCalibration output was removed. The LibMultiSense interface was upgraded to the latest version, which dropped support for this message type. The message interface remains to allow use in older logs.
  • The toolchain was upgraded to clang 16.0.6, and many of our third party dependencies were updated. The two most notable updates were fmt (to v10) and PCL (to v13.1). The fmt change will likely require you to be more diligent about using fmt::underlying and fmt::streamed.
  • The pipeline system is now more restrictive when connecting data envelope publishers to subscribers, taking additional steps to ensure types match. Previously it was possible to decode messages with different types but the same schema by taking advantage of this. The best workaround is to subscribe to an AbstractEnvelope and manually deserialize.
• Third Party Libraries
  • Added new third party libraries for Thrift, Google Snappy compression, and Apache Arrow (specifically, Parquet I/O related libraries). This will allow you to read/write parquet files from C++.
• Catalog Enhancements
  • You can now update existing ingest jobs (previously you could only create new ingest jobs). The ark-catalog-ingest-tool was extended to support updating jobs through the command line.
  • You can now delete existing ingest jobs. The ark-catalog-ingest-tool was extended to support deleting ingest jobs through the command line.
  • You can now search catalog attributes as groups, with and or or keywords (such as (robot_serial:10 or robot_serial:12).
  • You can now use additional operators when searching for catalog attributes, including equals, not equals, greater than, or less than.
  • You can now revoke existing tokens with ark-auth-tool (thanks Aníbal Limón).
  • You can now request tokens in ark-auth-tool by passing in a password (via command line argument or enviornment variable) (thanks Aníbal Limón).
  • Ingest jobs can now query collections and artifacts in the Catalog.
• Core Enhancements
  • You can now use the --video-bitrate command line flag to control output bitrate for your generated video files.
• Logging Enhancements
  • You can now amend log metadata with ark-logtool. This allows you to change or add new key/value metadata pairs to a manifest.
  • You can now remove channels from logs with ark-logtool. This allows you to amend a log and completely remove a channel.
  • Performance when opening or seeking logs with many split files and channels has been greatly improved, in some cases seeing a speed up from 5 seconds to 10 milliseconds. This was caused by inefficent regex matching on queries.
  • You can now explicitly block channels from being played back in a log, even if you have requested all channels to be played back, or that channel is in the pipeline and has no publisher.
• GUI Enhancements
  • You can now plot durations and timepoints in the plotting tool.
  • You can use space to toggle between play and pause in the log reader plugin now (thanks Matt Langford).
• ArkOS Improvements
  • Nvidia (L4T) based ArkOS builds now allow you to create a arkos-persistent partition on the root MMC device, enabled via config (thanks Aníbal Limón).
  • You can now configure the environment when building ArkOS images (thanks Aníbal Limón).
  • The Ark toplevel-repository is now mapped directly into the ArkOS container, allowing you to build images when the toplevel-repo is not in your home directory (thanks Aníbal Limón).
  • Loop-device mapping in the ArkOS build script is improved and less likely to experience spurious errors (thanks Aníbal Limón).
• Serialization Enhancements
  • Added a new ParquetWriter class, which enables you to write rbuf structures (as rows) to Parquet files. This is useful for emitting large numbers of metrics that might be read in a data lake. See here for details.
  • There is a new ostream<< operator for rbuf-generated enums, allowing you to get a string version of the enum with cout or fmt (thanks Matt Langford).
  • There is a new format_as operator generated for rbuf structures and enums, allowing you to print structures and enums with fmt without the fmt::streamed markup.
  • The operator~ overload is now present for rbuf bitmask enumerations. This makes it possible to unset bits within a bitmask.
  • JSDoc @type information is now generated with rbuf for Javascript. This is specific to variants, and provides type-hints to make it clearer what the potential variant types are (thanks Brandon Rioja).
  • Booleans are now decoded as the Boolean type in rbuf-generated Javascript code (thanks Brandon Rioja).
  • Exceptions are now thrown as Error type rather than string literals in rbuf-generated Javascript code (thanks Brandon Rioja).
  • Enumerations are now sorted by name rather than value in rbuf-generated Javascript code, so they appear alphabetically (thanks Brandon Bioja).
• API Enhancements
  • Added a new CANopen segmented transfer manager for downloading large SDOs. The existing segmented transfer manager was renamed to SegmentedUploadManager.
  • Added a new remove method to the S3Client, making it possible to remove keys using our S3 wrapper.
  • A new ark::database::SqliteDatabase API was exposed, making it easier to interact with and use sqlite3 directly and safely.
• Stage Enhancements
  • The VideoCaptureStage will now timestamp images with pipeline time (previously it timestamped them with whatever the kernel returned). You can enable the old behavior by setting the preserve_original_image_timestamps flag.
  • The “fake hardware clock” path in the NtpClientStage is now adjustable. The default is set to where the Debian package expects it to be, but you can now adjust it to be at an alternate location.
  • The NtpClientStage now allows you to synchronize to the hardware (and fake) clock periodically, rather than just on large time jumps.
  • The FaultStage now allows faults to be configured with a ‘service code’, which is intended to be a unique and stable identifier for a particular fault. This allows faults to be identified and looked up, while changing other characteristics (such as description strings or channels).
  • The LogWriterStage commands and status messages were enhanced to take in an optional request identifier, allowing you to determine if the log that is being written out was started from your request to start a log (rather than someone else’s).
  • The LogWriterStage now allows configuration of per-split-file maximum items and size. This was previously able to be configured through the C++ APIs, but not exposed through the stage configuration.
  • The MultiSenseIntefaceStage now supports dynamic configuration of AutoExposureConfig.
  • The WriteImageStage now will prefix command-line argument flags with the stage name if not using the default stage name. This allows multiple WriteImageStage stages to be added to the same pipeline.
• Tooling Enhancements
  • Any of the Ark tools that make use of HTTP comms will now obey the SSL_CERT_DIR and SSL_CERT_FILE OpenSSL environment variables, allowing you to specify paths to your certificates.
• Notable Bug Fixes
  • The EDK repo for Nvidia UEFI-support in ArkOS is now pinned to the version of Jetpack that we are using.
  • CSV files with carriage returns were not properly stripped in some situations.
  • ArkOS builds no longer require tags in the parent repo to function properly (thanks Aníbal Limón).
  • The EC2 metadata server was ignored in some circumstances when it would not have been in pre-2023.02 releases. It will now only be ignored when an endpoint is manually specified (ie, tests). This means that if you do not have a region configured, and are not running on an EC2 instance, you might incur a one-two second penalty when starting an S3 client.
  • rbufc would associate errors with the wrong source files when including files. This has been addressed.

Release 2023.04.0

• Breaking Changes
  • The underlying compiler was updated to Clang 16.0.1, which brings numerous improvements in C++20 and C++23 support. Additionally, libstdc++ was updated to 12.2 from 10.2. You will most likely need to include additional headers when upgrading (as the STL headers are less likely to include each other in this new version). Compile times have generally improved (Ark builds about 20% faster on our test machines). Additional third party packages were added (lmfit), and OpenMP is enabled for PCL by default.
  • The API to support deploying software or operating system releases from your local machine changed to take advantage of new HTTP server functionality. This results in a large improvement in performance when uploading releases, but required an API change break. To deploy software, you will need to make sure your ark-platform-supervisor is upgraded to the latest version of Ark.
  • The arkos-persistent partition is no longer mounted in the initial RAM disk of ArkOS. Instead, you must provide a fstab with the partition, and its recommended that your systemd unit files require that mount to be present before starting.
  • The NmeaStage was renamed to NmeaInterfaceStage for consistency reasons.
  • The SimClockExecutor will now continue to deliver messages during shutdown and before finalize. See below for more details.
  • The default_frame_id field in ark::gui::GuiApplicationConfig was renamed to default_renderer_frame_id to better reflect intent.
  • The Fault Stage will now refuse to start if you have a fault configuration that tries to listen on a channel that does not exist (and no default value is provided).
• Build System Improvements
  • The make.sh command gained the --test-dir command line flag, which allows you to run all unit tests contained in a directory.
  • The make.sh command gained the --run command line flag, which allows you to build a target and then execute it all in one command.
  • The add_release cmake function gained the FILES parameter, which allows you to include arbitrary files in your release.
  • The -fPIC flag is enabled when building libstdc++ (including Fortran). This enables linking larger binaries that reference Fortran code (notably, anything using LAPACK/BLAS).
  • The system now uses pixz to decompress toolchain blobs, offering a substantial performance improvement when decompressing the toolchain on a fresh system start. Thanks Joe Lisee!
• Pipeline Improvements
  • The SimClockExecutor will now deliver messages during shutdown. This makes it more consistent with the RealTimeExecutor. Previously it stopped delivering messages right before shutdown is invoked. The major change from previous releases is that simulations used to end precisely when you requested (for example, if you asked for 10 seconds of simulation, you got exactly that). Now, the simulation enters shutdown phase when the duration is exceeded, but can continue for some time (until all message queues empty). The positive side of this (beyond consitency) is that you can now do “final” work during shutdown (such as finalizing video).
  • The pipeline system gained the ability for stages to be informed of a shutdown request (for example, if you receive a SIGINT). This allows you to take some action before shutdown begins in earnest.
  • If a stage changes the pipeline exit value from ‘success’ to ‘failure’ during shutdown, that is now honored. Previously, you could not change the exit value once shutdown had begun.
  • Added the concept of a “serializable subscriber”. This indicates you want to receive abstract envelopes, but you know the exact type you want those envelopes in. This enables additional error detection at the pipeline level, and improves communication of type to downstream stages.
  • Topology listeners will now receive a notification that the topology is fully established. This is useful to take final steps (or validations) before start is invoked on all stages.
  • A pipeline can now be ‘frozen’, causing all execution (including timers) to stop working. This can be used to pause simclock pipelines for further analysis. This can be acheived remotely by writing to the /executor/runtime_control channel, or locally by using the executor_runtime_control_function() method within the StageInterface.
  • Pipeline metrics now have executor statistics associated with them. This includes information on the initialize and start times, number of cycles executed, and if the pipeline is active or paused.
• Logging Enhancements
  • Added a new ‘repair log’ API and ark-logtool command, allowing you to cut out basic corruption from logs that were damaged.
  • The Ark log reader can now query for channels based on a regex. Instead of specifying each channel name manually, you can now specify a regex, and all channels that match that regex will be returned.
  • The Ark log reader can now access logs over HTTP directly.
  • The Ark log reader will now refresh presigned URLs to logs stored on S3 periodically. Previously, these were only fetched once, so if your log processing job took over an hour, and you needed to fetch additional data, you would get a security error.
  • Introduced the ARK_DEBUGLOG_VERBOSITY environment variable to allow increasing the verbosity of debuglog statements. Useful for tests or command line utilities, where configuration of verbosity is not otherwise exposed.
• Serialization Enhancements
  • Added support for constant values in rbuf schemas. Previously, these were only allowed as ‘global’ variables. Now, structures can have constant values as well.
  • Added a new values() API for fetching all potential values an rbuf-generated enum can have at runtime.
  • Enum ‘values’ are now strongly typed in the Python rbuf code generator (thanks Matt Langford).
  • All rbuf types are now brace-initialized in C++. This addresses an issue where fixed-size arrays were not zero-initialized.
• Stage Additions
  • The Velodyne stages now support decoding VLP16 packets.
  • The Velodyne stages now support PPS-based time synchronization. You can enable this in the sweep configuration.
  • The MultiSenseInterfaceStage now supports automatically combining color images (and a YUV420 output), via a new configuration option (merge_luma_chroma_channels). If a BGR output is desired instead, you can set the configuration option output_bgr_color_images as well.
  • The MessageSynchronizer stage now allows for subscription to channels to synchronize during topology change events (ie, dynamically based on pipeline configuration).
  • A new GprmcPublisherStage was added to the system. This can consume a GpsFix message and produce a valid GPRMC message. This is then transmitted to a particular port (say, on a Velodyne) or domain socket (for Chrony).
  • The FaultStage will now send out notifications for the initial state of all faults on startup, allowing downstream handlers to trigger if a fault is active initially.
• Python Improvements
  • The GUIs can now subscribe to channels that were only published by a Python stage. Previously, you needed to publish a channel in a non-Python stage as well.
  • Subscribers in Python stages now have their types checked. If there is a type mis-match between the subscriber in Python vs the rest of the Python, the pipeline now refuses to load (bringing them inline with C++).
  • Python stages can now request and utilize shutdown tokens, allowing them to request a pipeline exit.
• Communication Enhancements
  • The built-in HTTP server can now redirect requests directly to files, rather than storing them in memory. This allows the server to receive much larger payloads on machines that have limited memory.
  • Optimized some paths in the HTTP server request handling, yielding performance benefits when receiving very large messages.
  • Introduced a new ’throttled’ protocol for Websockets to both the Qt and web-based systems. This allows the pipeline to be more selective in dropping messages, reducing latency (in terms of receiving the most recenently published message) in bandwidth constrained links.
  • The GUI (both Qt and web-based) can now indicate in their subscriber configurations that they wish to receive the last published message on a channel upon connection (so-called ’latched’ messages in ROS). This can make it easier to get the data you want on reconnect, without writing a custom query-based system.
  • Websockets now support Zstd-based compression (in both Qt and web-based systems). In some cases, this offers significant bandwidth savings (at cost to additional CPU). The web-based comms system did not support LZ4 compression in the past, so the gains are more pronounced there.
  • You can now use ark-spy (or the JSON viewer) to see messages that are being published from the GUI to the pipeline. Previously, these were not available.
• GUI Enhancements
  • Improved support for RGBA-based (and BGRA-based) images in all of our renderers. These now display as intended in both Magnum and Cairo.
  • Renderer layers can now be published on any channel, as long as they are in the /gui/renderer_layers namespace. This allows you to break out renderer layers into different channels, making it easier to disable them from the GUI side (or to apply different QoS settings to different layers).
  • Introduced the concept of renderer layer sets (ark::gui::RendererLayerSet). These are published under any channel in the /gui/renderer_layers namespace. These are a collection of renderer layers, and they will all be rendered at once (or not at all). This is helpful for applying QoS settings and indicating that all layers are part of a group and must all be rendered at once (or not at all).
  • You can now set a frame for your GUI application to startup in via the startup_render_frame_id and/or startup_follow_frame_id fields in the GuiApplicationConfig structure. This will override any previously selected frame, but the user can still change away from it to a different frame if they desire.
  • You can now take ark::gui::RenderLayer objects and render them to a JPEG file (or other image) with a single call (ark::gui::render_to_jpeg). This can make it easy to debug your visualizations using standard Ark tooling, without needing a GUI.
  • The annotation plugin now displays pipeline time for all annotations, along with better supporting long or multi-line annotations.
• Tooling Additions
  • The ark-spy tool can now output the size of messages on a channel (both as serialized data and the on-wire, potentially compressed, size).
  • The deployment script can now deploy to non-host architectures (for example, aarch64), if you specify the new --target flag.
• Deployment Improvements
  • The deployment tools now have a longer timeout by default when uploading/downloading archives. Additionally, you can tweak this value on the command line. Useful for when deploying archives over very slow links.
  • Deployment of very large archives (including ArkOS and Software images) from a local machine is now considerably faster.
• ArkOS Improvements
  • Commited a default configuration for building ArkOS images for the Nvidia Orin. This allows you to build, run, and deploy ArkOS on Orin boards.
  • The sync mount option was turned off for the ArkOS boot drive. This was largely redundant with journaling all data, and had a major impact on write performance.
• Configuration Additions
  • A new API ark::config::serializable_object_to_config was added, allowing you to take a serializable (rbuf) object, and emit a block-formatted YAML object that can be parsed by the configuration system.
  • When using the pipeline APIs for adding stages with a programmatically constructed configuration, these configurations were inserted in the “flow” style of YAML (essentially, JSON). These are now inserted in the block-formatted YAML, for consistency.
  • System and Steady timestamps can now be decoded using the ‘duration’ suffixes (for example 1_h would decode as a timestamp of “one hour”). Thanks Matt Langford!
  • Added a new ark::config::ConfigPackage::set_config_option API for setting the option used by a particular configuration block, overriding what is present in the config package.
• API Additions
  • Ark gained the ability to parse URDF files and generate both transform networks and 3D scenes from them. This allows you to visualize models and to see those models move in real time as values change.
  • The ark::core::ByteBuffer API gained the ref_count() and clone() APIs, making it easier for you to detect when the buffer is simply wrapping data (these APIs mimic the OpenCV Mat equivalents). clone will always create a deep copy.
  • The ark::core::ByteBuffer API gained a convenience static method, wrap_data, which can wrap a string view in a ByteBuffer, reducing some casts that had to be done in the past.
  • The image API now has a UV8 data format, for specifying images which contain UV planes (but no luma information).
  • Ark gained a new ark::core::get_environment_variable_or_default API to allow you to fetch a default value in the event that an environment variable is not set.
• Bug Fixes
  • The env.sh script referenced ldconfig as if it was in your PATH, which is not true for some distributions (notably, Debian). This was fixed, allowing Ark to build out of the box on Debian systems again.
  • The simclock event timer (and simclock timer itself) could overflow in some situations, resulting in exceptions talking about an internal error (or hangs). This has been addressed, in the sense that all hangs were fixed, and the error messages now attribute the underlying issue appropriately.
  • Fixed a bug in the Qt GUI that would cause data to render incorrectly depending on the Z value for the camera origin.
  • Fixed a bug with registering configuration structures in pipelines if those structures included variants. These are typically used during unit tests.
  • Fixed a bug when using UVC-based cameras. Some controls (32-bit signed integers) would be presented as if their ranges were unsigned.
  • Fixed a bug with the PCAP parser that treated some packets (for example, ICMP) as UDP. This manifested as unknown UDP packets being passed back to the user. These are now no longer passed to the end user (and are reported in the statistics structures).
  • Fixed a bug with rbuf dependencies and Python virtual environments. In some cases, the necessary Python rbufs would not be regenerated when a dependent file changed, causing runtime errors.

Release 2023.03.0

• Breaking Changes
  • The env.sh script now only adjusts the LD_LIBRARY_PATH for systems that have an older libstdc++ (for example, Ubuntu 18.04). Additionally, it no longer adjusts PYTHONPATH to point at our sysroot’s python installation (which often caused confusing results on newer versions of Ubuntu). The linting script now sets PYTHONPATH explicitly for the tools that require it, and you can launch the Qt6 designer through a script in the scripts directory now. Ubuntu 18.04 machines will still require the LD_LIBRARY_PATH to be adjusted to point at our libstdc++ (or newer), just as they did in previous releases. env.sh should work identically.
• Pipeline Enhancements
  • Lowered the overhead within the SimClock executor’s scheduler. In some workloads, this overhead has been reduced by 2x to 4x. This overhead is typically overshadowed (by a wide margin) by algorithm times, but was dominant in the case of pipelines with hundreds of stages that executed extremely quickly.
  • Improved valididation of the RealTime executor’s pipeline configuration. This now catches more potential issues on startup, particularly when running with --exit-after-initialize.
  • Two new options now exist for users of ark::main, allowing you to set the “option” that a stage configuration is using, or allowing overriding a specific parameter. See the config docs for more details.
  • The class_name field in StateConfig was deprecated and should no longer be set. This parameter was not validated (or used) in any way.
  • The ark-spy tool can now display the “last received” message on any publisher. For example, if you want to see the last published config package, you can run ark-spy -c /config/package --latest.
• Serialization Enhancements
  • The rbuf format now supports optional types. This can save some space on the wire (if a field is not present), and can better communicate intent if a field is truly optional and the default value will not suffice. This extends to parsing configuration.
• Build System Improvements
  • Improved rbuf dependency generation and removed deferred evaluation in cmake. This reduces overall cmake reconfigure times and very slightly improves rebuild times.
  • Embedded files now generate a cmake target which must be included to have access to the embedded header file contents.
  • Added shellcheck to the scripts directory, allowing you to invoke it to lint your shell scripts.
• Catalog Improvements
  • All authentication now goes through the Catalog. We no longer expose AWS services to users. The ark-auth-tool now communicates with the Catalog. The previous authentication tool was renamed to ark-aws-auth-tool. This allows for the removal of the ark::aws target in many cases, lowering overall binary size for many common tools and pipelines.
  • Improved the scheduler backend, allowing for lower latency feedback on job status and scheduling much larger batch jobs.
  • Added support for batch and ingest jobs that require GPU resources. You can set the new gpu_count parameter when scheduling jobs to run on a GPU instance.
  • Internally, the catalog now no longer allows access to any S3 buckets. All batch and ingest job activity must go through the catalog authentication system.
  • Improved the ark-catalog-ingest-tool to allow you to query and create ingest jobs from the command line. This tool also allows you to reingest logs or re-run ingest jobs.
  • The Catalog gained the ability to schedule batch and ingest jobs that require a GPU.
• Logging Enhancements
  • Improved error recovery when downloading logs over HTTP (or when using presigned URLs). Also sets more specific (and generous) timeouts around data rates and overall transfer times.
  • The log writer stage and log amender stage now will cause pipeline initialization to fail if you explicitly try to log a non-serializable channel. Previously these channels were silently ignored.
  • A new start-in-paused-state flag was added to the main::MainOffboardConfiguration structure, which allows you to always start your pipeline in a paused state.
  • The ark-logtool tool gained the ability to amend in a user-specified organization id. This is helpful for logs that are missing organization identifier, which are required for uploading logs to Ark Catalog system.
• ArkOS Improvements
  • A new create-release utility was added, allowing you to upload ArkOS releases to the Catalog.
  • The ark-supervisor-tool gained the ability to deploy specific ArkOS releases from the Catalog to a machine running ArkOS. The machine will pull down and install the release as instructed.
  • The platform supervisor can now deploy releases belonging to a private organization (both software and ArkOS releases).
  • Improved cross-build support for ArkOS, allowing construction of aarch64 disk images. These still require UEFI to boot.
  • Added experimental support for ‘Linux for Tegra’ (for Nvidia Orin boards, for example). This allows you to build ArkOS images for an L4T board, although it does require UEFI to boot.
• ROS Integration
  • You can now generate rosmsg schemas from rbuf schemas, making it possible for ROS to interpret a subset of rbuf messages natively. Does not support conversion of rbufs that make use of features that rosmsg does not support.
• GUI Enhancements
  • You can now publish (and subscribe) to non-serializable data between GUI plugins. Previously, all GUI plugins could only publish or subscribe to serializable data.
  • Added support for “local debug rendering” (an equivalent to OpenCV imshow). This allows you to display either images or renderer layers in a popup, from anywhere in your stage. Useful for quick visual debugging.
• Core Enhancements
  • Added a new Private flag to core::write_string_to_file that allows you to adjust permissions on a written file such that you are the only user that can access that file.
  • The core::Guid class now provides a std::hash method, allowing you to use it as a key within C++ std::unordered_map.
• Fault Enhancements
  • Added a new NOT_EQUAL and STRING_NOT_EQUAL fault operators. This allows fault configuration to be set for not equal evaluations.
• Bug Fixes
  • Fixed a potential deadlock in the lockfree thread pool. This could occur in some rare situations when a work item was submitted to the locklfree queue, all threads were about to enter into an idle state, and you never submitted any additional work. It only affected thread pools that had the lockfree queue enabled, and the threads unblocked if you submitted new work.
  • The config parser will now properly parse fixed array types from YAML files. Previously only dynamic arrays (arraylists) were supported.
  • The log reader will now give a bit more time once the last message is played in a log for the simulation to process said message before exiting.
  • Hide clang-tidy warnings around implicit boolean conversion in the autogenerated rbuf code when using the spaceship operator and types with bools (thanks Howard Smith).
  • The --dot-output now correctly quotes node names, allowing you to use some additional special characters in your stage names.
  • Fixed a bug in lint.sh --fast that would print spurious warnings about missing files when you had deleted or renamed files in your repository (and hadn’t pushed them).

Release 2023.02.0

• Breaking Changes
  • The comms::PollSet and comms::EventCondition were moved into the core namespace. No other changes were made to these classes.
  • Third party “install path” variables used to be prefixed with the cmake namespace. They are now prefixed with the package name instead. For example, this means that OpenSSL_INSTALL_PATH is now openssl_INSTALL_PATH. This addresses a bug where multiple packages in the same cmake namespace had install paths that squashed each other, but means you will need to update any reference to those install paths in your local code.
  • All unit tests now have a 10 minute timeout. If this is exceeded, the test is considered failed. Previously, we suggested relying on the CI system to timeout the jobs, but this would cause information about what test was timing out to be lost.
  • The gamepad module has been moved into the ark::input namespace to improve consistency.
  • ark-crypt-tool was removed from the system. It’s functionality was merged into ark-signature-tool, which was overhauled to support new catalog-managed keys (see below).
  • The ARK_SIGNATURE signing mechanism was removed entirely, in favor of using standard signature formats. Previously, large files were signed using a Ark-specific format, and required Ark-specific tools to verify.
  • The ark::logging::ReadCursor::required_urls() API call will now return a structure containing paths to the manifest and split files, organized by identifier. Previously it just returned a list of split files.
  • The ark::catalog::UploadArtifactStage request message’s type field was renamed to content_type. Further, you can now leave it empty and the type will be inferred from the file extension using the MIME database.
• Notable Changes
  • The core pipeline system is now entirely lockless. All publishers can transmit messages without taking out any locks (and all subscribers can receive those the same way). The core thread pool was also adjusted to allow submission of jobs locklessly and without allocations.
  • Added the ability to stream and log kernel performance counter information, making it easy to capture high resolution performance data in real time and log it alongside the rest of your data. This is useful for debugging timing and performance issues.
  • The backend of the catalog was updated such that you no longer use AWS credentials for common operations, including downloading/offloading logs, running simulations, uploading artifacts, or creating or deploying releases.
  • Added a new --test-regex command line option to make.sh. This will build and execute all unit tests that match the given regular expression. Use this to easily execute subsets of tests, rather than the full suite with make check.
  • Generated rbuf code now provides a std::hash routine. This allows you to compute a hash for your rbuf objects (or to use them as keys in an unordered map). Currently the hash is based on FNV1A, but this is expected to change in the future. Do not rely on the hash to be deterministic between pipeline executions.
  • The offloader now supports offloading data logs to plain HTTP servers (such as nginx). This allows you to offload logs to many common NAS appliances. Additionally, you can now define multiple potential offload targets (including to custom instances of the Catalog or various HTTP servers).
  • Ark API documentation now supports references, enums, macros, typedefs, and associating code samples.
  • Updated third party dependencies (ECR, SQS, SNS, and KMS were added to the AWS SDK), added Boost::SML, and updated other third party versions (mostly security-update related).
  • Updated the packaged AWS CLI and Node to latest versions.
• Config Changes
  • You can now specify durations in config files with suffixes, such as _ms for milliseconds or _s for seconds. Previously you had to specify them in nanoseconds.
• Deployment Changes
  • The deployment system has changed its default packaging system from .zip to .tar.zst. Using Zstd for the packages results in much faster compression/decompression sizes and generally improved storage.
• Catalog Additions
  • The catalog can now manage your private keys for you. It uses the AWS KMS system as a backend, and allows you to create key pairs and sign with them, securely. This is now used to sign software and operating system releases.
  • Catalog-based tools can now reauthenticate by setting additional environment variables (such as ARK_JWT_REFRESH_TOKEN) rather than relying on a config file to be set.
  • Improved terraform build files to make it easier to deploy the catalog onto your own AWS infrastructure. Note that this is a work in progress – more changes are necessary before this is considered complete.
• Build System Additions
  • You can now set the ARK_ENABLE_EXAMPLES flag to OFF. In addition to disabling unit tests and Qt applications, this can reduce build times by turning off compilation of examples.
  • The method for computing rbuf dependencies has been adjusted to use build-time depfiles rather than explicit dependencies that were computed at cmake configure time.
  • Ark now provides a .clangd sample file which will enable unused-include file detection. If using clangd, you will now see include files that are not used highlighted. Note that this feature does have false positives.
• Python Changes
  • The logging bindings were enhanced signficantly, allowing you to read logs in Python without using the LogReaderStage.
  • You can now add bindings to your Python virutal environments with the BINDINGS keyword.
• API Changes
  • You can now get progress back when doing long HTTP requests using the comms namespace HTTP helpers APIs.
  • You can now set low-speed timeouts using the comms namespace HTTP helpers APIs.
  • A new ark::core::poll API was added, which is the equivalent of the syscall poll, but can take in a chrono duration (and honors sub-millisecond timeouts).
  • The ark::crypt APIs now expose additional signature formats (specifically, sha256-based PKCS1 V1.5), along with allowing for singing with precomputed digests.
  • Support for the Bayer RB12 image format (thanks Matt Langford).
  • Support for format conversion in the VideoScaleStage (thanks Matt Langford).
  • The Archive API now supports .tar.zst (for Zstd) and .tar.lz4 (for LZ4) based archives.
• Pipeline Changes
  • The real time executor’s timer pump is now run on a separate thread (and named). This allows you to more easily identify it in top/ps output.
• Stage Changes
  • Added a PerfEventCollectorStage that allows you to collect low-level kernel perf counters. This can be very expensive, but can also give you very low level information on performance on your robots.
  • Runtimes and Wait times are now gathered for all of the threads in your pipeline. Additionally, both this information and various thread priority and utilization numbers are visible in the hardware health plugin.
  • Added support for writing latched messages out in the LogAmenderStage.
  • Added a new --start-in-paused-state flag to the LogReaderStage (and any pipelines using the main system). This allows you to start a log playback ‘paused’ instead of ‘playing’, allowing you time to connect a GUI and hit the play button when you are ready.
  • The LogWriterStage now works properly under the SimClockExecutor. Previously, if logging the config package of your pipeline, it would generate corrupt logs (with the config package appearing at the end of the log).
• Bug Fixes
  • A minor bug in timer execution determinism was fixed, ensuring timers are always executed with a deterministic timestamp.
  • Fixed a bug that prevented you from being able to read fixed-size arrays with rbuf in Python.
  • Fixed a bug that prevented you from using older MultiSense firmware with our stage.
  • Fixed a bug that prevented you from using multiple add_container targets in the same CMakeLists.txt file.

Release 2022.12.1

This release fixes a few critical bugs in the 2022.12.0 release along with adding some additional third party packages and profiling capabilities.

Specifically:

• Fixed a bug in MultisenseInterfaceStage that prevented images from being published in some non-PTP configurations.
• Added a ark::backtrace_testing_main library as a replacement for GTest::gtest_main. This is identical except will enable stack trace display on segfaults/crashes by default. Most Ark unit tests were converted to use this library.
• The SocketCAN stage now handles running out of kernel-level buffers more gracefully. Previously, a reconnect was forced; now, packets are dropped and recorded in the output statistics. APIs were adjusted to support reporting when writes failed due to buffer exhaustion.
• The amount of time that the primary timer thread will sleep for in the real time executor is now bounded to no more than 50ms.
• The timer thread in the real time executor is now instrumented with profiler traces, allowing you to detect if that thread is blocking (which in turn would cause timers to fail to fire).

Release 2022.12.0

• Breaking Changes
  • The web communications manager now supports dynamically unsubscribing from channels that you have previously subscribed to based on a subscription token, rather than channel name. This removes the issue where multiple plugins that were subscribed to the same source could unsubscribe all other plugins. The API has changed to only accept tokens, however.
  • The ark::time::TimeClientStage was renamed to ark::time::NtpClientStage. Additionally, the stage’s default name changed from TimeClientStage to NtpClientStage, and the configuration structure was renamed from time_config to ntp_client_stage_config.
  • The ark::core::get_process_name() API was renamed to ark::core::get_this_process_name() to better indicate intent.
  • The PythonStage (and Python support in general) now defaults to using the system Python rather than the hermetically included Python. You can get the old functionality back by setting the ARK_USE_HERMETIC_PYTHON option to ON. This was done to simplify some use cases with PIP, virtualenvs, and cross-builds, but we will review in the future. Make sure you have python3-venv installed in your system packages.
  • The ark::comms::NetworkAddress method address() now returns an optional (previously it returned a uint32_t). This optional indicates if the network address has an address component or not (it may just have a port component).
• Notable Changes
  • Established a new awlog file format that provides a mechanism for storing log data in a single file, suitable for being downloaded and streamed to web clients. See documentation for more details.
  • Improved parallelization of the simclock runner significantly. Previously, if one stage was executing for a long (wall) time, and many other stages only needed to execute for a short period of (wall) time, then said stages would not be parallelized as well as they could be. This did not affect determinism, only real-time performance. This can improve overall runtimes by 20-30% in some workloads.
  • Introduced an ark::pipeline::MessageSynchronizerStage for synchronizing messages together into a common set (by default, based on pipeline time, but you can customize this with your own lambda).
  • Introduced the concept of a .packages.conf file which allows you to customize the toolchain, sysroot, and third party package bundles that you wish to use on your project. Further, you can now customize the URL that said packages are downloaded from.
  • Reduced the chances of the Qt user interface locking up due to a subscriber falling behind in its queue due to long processing times.
  • The supervisor now verifies manifest signatures before extracting a release. Previously, it extracted a release, verified the signatures, and if that verification failed, removed the extracted files. This prevents some forms of attacks in maliciously crafted archives. The supervisor continues to verify signatures and hashes before launching software.
  • Added support to rbuf for basic constant expressions. You can now assign constants to other constants (or to enum values), along with executing basic operations such as bitmasks or addition.
  • The process library now uses vfork by default, and Ark itself does better management of file descriptors across process boundaries. This includes setting O_CLOEXEC on most file descriptors that we open, along with force-closing all open descriptors before processes are launched.
  • Upgraded clang to 15.0.6 (fixing a few stability issues and other minor improvements), and updated a few third party libraries (no major changes, mostly focused around security updates). Additionally, the openssh and dbus libraries were added.
• Web Changes
  • Refreshed overall styling of the webviz system, modernizing and cleaning up all of the various plugins and layout systems. A number of bug fixes were made, including layout storage and default layout application. A version of the webviz system is now periodically deployed to the http://catalog.tbdrobotics.com/webviz location.
  • The web communications manager now supports subscription to JSON channels.
  • Extracted the ark-web-renderer package for embedding the three.js-based data visualizer in standalone web applications.
  • Introduced the ark-web-awlog package for reading in awlog files from web applications.
  • Added support for mouse clicking / mouse movement updates to ark-web-renderer, returning coordinates in world positions.
  • Fixed a memory leak in the ark-web-renderer component revolving around rapidly changing text objects.
  • The webviz system is now built by esbuild rather than parcel2.
  • Added a plugin to the webviz system for displaying messages as JSON (like the Qt-equivalent).
  • Added a plugin to the webviz system for log playback (play, pause, seeking, etc).
• Python Improvements
  • The PythonStage will now have its start, shutdown, and finalize methods invoked when the rest of the pipeline is starting up/shutting down. Previously, only initialize worked.
  • The PythonStage now can fetch configuration from the pipeline’s configuration package.
  • The PythonStage now takes a virtualenv_name configuration parameter, which allows it to resolve the virtualenv dynamically, rather than you needing to bake the path into your stage config.
  • Added support to add_container and add_release for automatically copying the virtualenv into an appropriate location, simplifying deployment of Python artifacts.
  • The IPC system for communicating with Python stages has been reworked; performance is greatly improved when sending large amounts of data (such as images). For example, sending 6x 720p images previously took 90ms on common hardware; it now takes 700us.
• Visualization Improvements
  • You can now specify if a mesh is 3D or 2D. In many cases, meshes were conceptually 2D, with a zero’d Z parameter. This resulted in a lot of wasted space going over the wire and into logs (which compression helped with somewhat). Consider reformatting your meshes as 2D if you can.
  • You can now specify a single color for your meshes. Previously, in order to work with all three renderers (Magnum, Cairo, and THREE.js), you needed to specify one color per point, which is fairly wasteful when all of your data is the same color. You can now specify a single color and it will apply to every point in your mesh.
  • The Cairo renderer now renders images (in the HUD and in the 3D view) with nearest filtering, which reduces blurriness on images and improves performance.
• Stage Changes
  • The Hokuyo stage will now periodically request and publish version information.
  • Both the Hokuyo and SICK interface stages publish renderer data in 2D (and single color) to reduce bandwidth on the wire for these stages.
  • The Swift Duro stage will now properly handle IMU messages that have timestamps based on local system time, rather than only supporting global (GNSS) time.
  • Added a use_original_image_timestamps configuration parameter to the MultiSenseInterfaceStage, which forwards the original image timestamps along (rather than converting them to pipeline time).
  • Added PTP and triggering configuration to the MultiSenseInterfaceStage. This allows you to configure the MultiSense to synchronize to PTP (and trigger based on multiples of PTP), provided you have an independent PTP system established.
  • Added exposure target intensity and gamma configuration to the MultiSenseInterfaceStage.
  • Added a new Cia315ControllerStage that implements a CAN/UDP bridge (using the standard CIA315 protocol). You can use this to bridge CANbuses over wired or wireless links.
  • Added a LocalTimeClientStage suitable for publishing global time to the rest of the stack, with time based off the systems local time. You can use this if you have an external tool, such as chrony or ntpd, sychronizing your system clock, but you want global time published through the pipeline as if we were running an NTP client locally.
  • When building with Ark as a submodule, the VersionInformationStage now publishes version information for both the Ark repository and the top-level (“project”) repository (typically, your repository). Previously it would only report the Ark version number and hash.
  • The debuglog stage was refactored to be deterministic. Previously it was non-deterministic, which meant that logged debuglog messages may not have the same ‘publish’ timestamps between runs.
  • Added support for using JSON pointers in the fault stage.
  • The testing ark::pipeline::CallbackStage stage was enhanced to allow you to have a callback invoked with no publisher. Simply omit the PublisherType template argument.
• API Updates
  • The ark::core::ScopeExit class gained the ability to cancel – causing it to no longer fire your lambda when leaving a scope. Useful for tighter control on error cleanup.
  • Added a new ark::core::get_this_process_invocation_name() API to return the name that the process was invoked with (ie, argv[0]).
  • The ark::core::CircularBuffer class now supports overwriting old objects (previously it would error when it was filled).
  • The ark::config::ConfigPackage::from_file method can now take a list of search paths. This makes it easier to test loading config packages from locations that are not the root of your repository.
  • The ConfigPackage API now allows for search paths when discovering included files. This makes it easier to use the API when running in unit tests.
  • You can now set IDLE, BATCH, and FIFO priority levels in thread pools (including pipeline thread pools).
• Notable Bug Fixes
  • Improved the strictness of memory ordering in the ark::core::ByteBuffer copy-on-write implementation. This fixes some (potential) issues when copying byte buffers to many different locations extremely quickly.
  • Fixed a number of bugs in handling of disparity cost images coming from the MultiSense.
  • The Qt user interface no longer crashes when it receives invalid/corrupt images.
  • The Qt user interface is now more resilient to subscriber callbacks starving out the UI thread. This could manifest as the UI locking up. With these fixes, the user interface should be more responsive even under heavy load.
  • Fix a bug in ark-playback that prevented it from starting correctly when data logs contained SICK LIDAR data.

Release 2022.11.0.1

This is a minor bugfix release to 2022.11. A few notable fixes:

• Const-correctness fixes to the profiler and generated rbuf code to satisfy new clang-tidy requirements.
• Fixes to the Hokuyo stage that could lead to a segfault in the event of certain malformed packets being received.
• Fixed a bug with applying queries to log manifests that did not copy the log metadata over.
• Minor ASAN issue fix in the ark::core::ByteBuffer class addressed.

Release 2022.11.0

• Breaking Changes
  • The main_onboard and main_offboard routines now return a value that you should return from your main. They are marked as [[nodiscard]] to help spot instances where you aren’t doing this. This value will help indicate to the operating system if your pipeline exited successfully or in failure (ie, a shutdown token exited in a failure state).
  • As mentioned below, the fmt library was upgraded to the latest version, which brings in compile-time format string checks, along with a change in our it outputs streamable objects. Previously, if an object had ostream operators, you could output it with fmt as-is, but you now need to wrap it in fmt::streamed. This affects things like timestamps and paths, where you now need to resolve them down to primitives or strings. Similarly, enums now need to be wrapped with fmt::underlying in order to be emitted as their underlying type.
  • The HTTP helper APIs no longer throw on non-200-success-codes (for example, 202 or 204). It treats these as successful instead.
  • The default fill color for primitives changed from being full-white to being not set. See below for more details.
• Notable Bug Fixes
  • Fixed a bug in the SimClock executor that would cause the clock to fail to advance if you had multiple event timers (or single shot timers) in the same stage that all wanted to execute simultaneously.
  • Fixed a bug that could cause a log writer to report errors writing that would occur when writing compressed data out that both grew in size (versus its uncompressed size) and exceeded the configured split file size limits.
  • Fixed a bug that prevented compilation of rbuf files with comments before the namespace directive, or comments that ended with a EOF marker instead of a newline marker.
  • Fixed a bug with filtering time ranges when creating log cursors. This would manifest in some situations where you had amendments that did not intend to affect time ranges.
  • Fixed a bug that prevented compilation of rbuf files with DOS-style line endings.
  • Support additional image formats in the Qt camera image display. All image formats that Ark supports should now be supported in Qt.
  • Patched a regression in Qt 6.3.0 that prevented zooming in on the X-axis of line plots.
• Notable Changes
  • Updated the compiler to clang 15.0.3, along with updating numerous third party dependencies. Of particular note is the update to fmt 9.1.0, which brings compile-time format string verification, and OpenCV also now builds the ml and highgui modules, although linking to highgui requires linking against system dependencies to function. You may have issues with linking large executables; if so, consider specifying -no-pie as a linker option.
  • Added support for reading ROS bag files programmatically (without any ROS dependencies). This allows you to read bag files and pull out messages.
  • Added support for compiling rosmsg message definitions. This adds a new front-end to the rbuf serialization engine, allowing it to process .msg files (or definitions) and emit code (or use reflection) that can deserialize existing ROS messages.
  • The config parser can now process variant field types, allowing you to use variants within the config system.
  • The main_offboard system now will automatically add the new pipeline::ExeuctionTimeoutStage, which provides a --execution-timeout command line argument. This allows your pipeline to exit gracefully, but in failure, if the timeout (measured in real time, not simulated time) expires. For example, you can use this to say “if my sim hasn’t finished in 5 minutes, exit in failure, but still record off results of the simulation”.
• Rbuf Improvements
  • Added fixed-size array types (array<Type, Size>) to the rbuf serialization system. This allows you to add arrays with a (known, unchanging) size to your buffers. This can save some space on the wire when serializing, and might better communicate intent in some situations.
  • Added support for final schemas – these schemas do not include any versioning information, and cannot have groups added/removed. The benefit is that you can save some bytes in the on-wire representation.
  • The JSON deserialization codegen for rbuf will now ignore any key set to a NULL value. Previously, it would exception out if this occurred. NULL values were not generated by rbuf JSON serialization, but other JSON tools could generate them.
• Renderer Improvements
  • The Cario renderer would render HUD text anchored in the wrong location. This has been addressed.
  • The Cairo renderer gained the ability to properly render line strips, line loops, and circles. Additionally, limited support was added for rendering texture-mapped quads.
  • All renderers will skip filling in primitives if the ‘fill’ color is not set, or outlining primitives if the ’edge’ color is not set.
• Stage Updates
  • The path_prefix configuration in the LogWriterStage can now accept HOUR/MINUTE/SECOND and DAY_OF_YEAR variables, allowing you additional granularity on path generation.
• Compression Additions
  • Added BZ2 to the Ark compression helpers. This enables you to use BZ2 compression in places like the log writer. Note that due to the high CPU cost of BZ2 (and relatively low compression), it’s recommended that you use Zstd or LZ4 instead.
  • Added LZMA to the Ark compression helpers. This enables you to use LZMA compression in places like the log writer. This might be a good choice if you prioritize storage at all costs, but it’s recommended to consider Zstd instead. It is a better choice than BZ2, however.
• API Updates
  • The URL class gained a filepath() method that will return the URL as a path to a local file, assuming the scheme is either unset or file:. Returns an empty optional otherwise.
  • The ‘convert Ark to OpenCV image’ APIs gained the ability to treat channels as signed rather than unsigned, which is useful for using in some OpenCV APIs, particularly when working with depth images.
  • Add a set of helper classes for getting Ark-specific JWTs (HMAC-SHA256), useful for signing content and verifying it on a different machine (when both machines have access to the same shared secret).
• Build Improvements
  • You can now disable building of all Qt-related applications and plugins in Ark by setting the cmake variable ARK_ENABLE_QT_APPLICATIONS to OFF.
  • You can now disable building and running of all Ark-specific unit tests by setting the cmake variable ARK_ENABLE_UNIT_TESTS to OFF. This can reduce Ark build times by as much as 30%.
  • You can now disable building of Python-based stages by setting the cmake variable ARK_ENABLE_PYTHON_STAGES to OFF. This can be useful in environments without access to the PIP repositories.
• Tooling Updates
  • The ark-logtool tool gained the ability to generate amendments that support renaming channels from one string to another. This allows you to ‘repair’ a log that used the wrong channel names, or had channel names change over time.
  • ark-logtool can now set the name of amendments as you write them, making it a bit easier to organize them.
  • ark-logtool gained the new --amend command line option, which is a short hand for --save-new-manifest <path to parent of the loaded manifest>.
  • ark-logtool can now generate an amendment with a channel marked as latched, even if it was originally recorded as non-latched.
  • Added --cmake-arg to make.sh to allow direct configuration of cmake from the make infrastructure.
  • Added a new ark-logcopy tool that allows you to make copies of logs with channels removed or recompressed. Might be useful for reducing the space consumption of snippets or creating an independent version of an amendment/snippet.

Release 2022.09.0

• Breaking Changes
  • Package manifests/signatures are now included in the deployment package archives, rather than being standalone files or stored independently in the catalog. This makes it easier to distribute a single file that contains everything necessary for a release (and improves verification of the software launch parameters).
  • The supervisor stage now operates in a different namespace. Supervisor Tools from older versions of Ark will not communicate with a newer Supervisor and vice-versa.
  • The ark-supervisor-utility tool was named to ark-supervisor-tool for consistency.
  • The add_container build system function now places all binaries in the /workspace directory. Previously it placed them in the root (/), but this caused issues when users had their own lib and bin directories.
• Stage Updates
  • The HardwareHealthStage now monitors thermal and CPU metrics, such as frequencies, governors, and device (hard disks, motherboard, etc) temperatures. Metrics are captured on a 5-second interval.
    The metrics collected by the HardwareHealthStage are configurable and may be disabled, as the time to collect the data varies on each platform. See the HardwareHealthStageConfig for configurable options.
  • Added the concept of a ‘robot time zone’. This is a generic interface that allows you to publish what time zone a robot is currently in. This can be useful for transforming timestamps into human-readable time. A stage was added (ark::time::LocalTimeZoneStage) to publish the local machine’s timezone as a default.
  • All stages that took in a “priority” configuration parameter now also allow configuration of “processor affinity”, which allows you to pin their threads on certain cores. This includes the UDP, TCP, CAN, USB, RTSP, serial, and video encoding stages.
  • Added a stage and assorted utility routines for posting messages to Slack.
  • Added AutoExposure ROI support to the MultisenseStage.
  • The OffloadStage can now retrieve offload credentials from per-robot secrets flowing from the Catalog telemetry, removing the need for individual users to authenticate or storing offload credentials on the robot itself. This requires robot registration.
• Catalog Updates
  • Migrated the underlying database and REST service code out of the Catalog’s API gateway and into toplevel Ark libraries in ark::rest and ark::database. This allows creation of C++ REST interfaces.
  • The catalog can now post incoming key/value metrics pairs to CloudWatch, allowing you to add alerts based on metrics coming back from your robots.
  • Added per-robot authentication (via oauth2), including improvements to the test harness to validate such authentication.
  • Added a new ‘registration interface’ to the Catalog, allowing a robot to establish a trust relationship with the Catalog. This essentially will pull down a newly-generated client ID/secret pair once a human has authorized the registration, and the robot can use that pair to gain access to Catalog resources and per-robot configuration.
  • Added a new GeoIP public API, allowing authenticated clients to get an approximate location and timezone based on their public IP address. This API also allows for public IP address discovery.
• ArkOS Updates
  • You can now deploy new ArkOS images through the ark-platform-supervisor using the standard ark-supervisor-tool tooling. This removes the need to manually cleanup old ArkOS releases, enables signature checks when ArkOS is updated and activated, and makes it easier to do updates.
  • The ark-platform-supervisor now has a release ringbuffer, and will only maintain the last N releases (as configured). Previously it let releases accumulate indefinitely. It will prioritize erasing oldest releases first, and always save the active and failsafe releases.
  • ArkOS will now verify that the rootfs.squashfs is the original rootfs.squashfs associated with the kernel that is being booted, improving the secure start chain.
  • The ark-platform-supervisor can now update software and operating system images on partitions that are normally mounted read-only. It will handle re-mounting as read/write automatically.
• Pipeline Updates
  • The MainOffboardConfig structure now allows you to configure the maximum runtime of your offboard pipeline.
• Build System Updates
  • A new experimental flag was added (disabled by default) which produces shared-object builds of the core Ark runtime (pipelines, serialization, config, time) and main runtime. This allows you to dynamically link in Ark and use your own build system (and third party dependencies) independent of the rest of the Ark build ecosystem. Set ARK_ENABLE_SHARED_ARK_RUNTIME to ON to test.
  • The add_container command gained the ARK_CONFIG_PACKAGES parameter, allowing you to automatically pull in all of your dependent YAML files into your container automatically.
  • Added the libgpiod third party library to our dependencies.
  • Some APIs (CRC16/32 and Base64) were broken out of ark::crypt and into ark::crypt_lightweight_hash, to avoid bringing in all of the crypto code when you just want to do simple CRCs/base64 encoding. ark::crypt will still automatically bring in the ark::crypt_lightweight_hash, so you don’t need to update any existing code.
• Web Enhancements
  • The web-based communications library now relies on the browser to handle timeouts (with keepalives), rather than using our custom timer. This reduces websockets from closing when they are just simply idle, making it less likely that you ‘miss’ an object during a reconnect cycle. Further, console log statements around comms are now at debug verbosity level.
  • We now provide a esbuild transformer for rbuf files in JavaScript – this allows you to transparent import rbuf files in your JavaScript code, and esbuild will automatically convert them into JavaScript for you, removing the need for a pre-processing step, and handling dependencies properly.
• Notable Enhancements
  • The Archive class can now read and extract purely in-memory archives, rather than needing the archive to exist on disk. Further, you can now write content into archives from in-memory sources, rather than requiring it to exist on disk.
  • Added a new tool to generate key pairs, sign files, and verify those signatures. Useful if you want to use the Ark signature verification code from the command line. You can also use this to download public keys from the AWS key management system.
  • Added new image transformation routines, allowing you to rotate or transpose images quickly using the ark::image::transform_image API.
  • Added an API for converting greyscale images into ‘mono-bit’ images (packing the image into single-bit values for if a pixel should be on or off).
  • Added a ark::serialization::write_serializable_type_to_buffer API, allowing you to serialize a type and its schema information to a buffer. This is identical to serializing to a file, allowing you to deserialize it via reflection.
  • Added a checked_bulk_write API to the ark::usb::UsbDevice class (intended for raw communication with USB devices). Allows you to double-check that the number of bytes you expected to write were written, or a ‘short write’ exception is thrown.
  • Added a new ark::crypt::HashContext class to allow you to update a hash (such as SHA256 or MD5) multiple times, rather than requiring a one-shot API call.
  • The HTTP helper classes have been extended to support additional authentication mechanisms, URL encoding, and more request types.
  • Improved string decoding in the generated JavaScript rbuf code, resulting in the ability to decode larger strings (224KB+) with only a minor performance hit.
• Notable Bug Fixes
  • The RealTimeExecutor previously set its own thread priority before invokine start on all stages. This resulted in stages that started their own threads inheriting the thread priority of the RealTimeExecutor, rather than the default thread priority of the process. As an example, if the executor thread’s priority was set to real time, all threads started with start would inherit that and need to downgrade manually. start is now invoked after the executor thread priority is set.
  • Reading dynamic files (such as those in /proc) were previously limited to reading a single page size at once. The read_string_from_dynamic_file API will now attempt to read the entire file.
  • Configuration files that used non-string field types (such as integers) for dictionary keys now are properly parsed into structures.
  • The JSON str() method took an unsigned integer for indention, but should have taken a signed integer, allowing you to disable formatting altogether by providing a negative number.
  • Fixed a bug in the Python and JavaScript rbuf generated code revolving around durations. The code was assuming durations were unsigned, making negative durations not work properly.
  • Fixed a bug in the Python and JavaScript rbuf generated code that prevented the REMOVED attribute from being applied.
  • Import statements in Python/Javascript rbuf generated code will no longer bring in the same class from two different locations into the same scope.

Release 2022.06.0.1

This is patch release and contains a number of bug fixes (already included in newer releases):

• Fixed a bug with filtering time ranges when creating log cursors. This would manifest in some situations where you had amendments that did not intend to affect time ranges.
• Fixed a bug that prevented compilation of rbuf files with comments before the namespace directive, or comments that ended with a EOF marker instead of a newline marker.
• Fixed a bug that prevented compilation of rbuf files with DOS-style line endings.
• The RealTimeExecutor previously set its own thread priority before invokine start on all stages. This resulted in stages that started their own threads inheriting the thread priority of the RealTimeExecutor, rather than the default thread priority of the process. As an example, if the executor thread’s priority was set to real time, all threads started with start would inherit that and need to downgrade manually. start is now invoked after the executor thread priority is set.
• Configuration files that used non-string field types (such as integers) for dictionary keys now are properly parsed into structures.
• Fixed a bug in the Python and JavaScript rbuf generated code revolving around durations. The code was assuming durations were unsigned, making negative durations not work properly.

It also adds the following features (already included in newer releases):

• The ark-logtool tool gained the ability to generate amendments that support renaming channels from one string to another. This allows you to ‘repair’ a log that used the wrong channel names, or had channel names change over time.
• ark-logtool can now set the name of amendments as you write them, making it a bit easier to organize them.
• ark-logtool gained the new --amend command line option, which is a short hand for --save-new-manifest <path to parent of the loaded manifest>.

Release 2022.06.0

• Breaking Changes
  • The supervisor code was removed from the ark::process module and placed into its own ark::supervisor module, reducing the dependencies when using ark::process::Process considerably.
  • The archive code was removed from the ark::process to the ark::archive module, reducing dependencies if you just with to extract archives.
  • The ARK_ARCHITECTURE cmake variable was removed and replaced with ARK_TARGET_ARCHITECTURE to better reflect intent (the architecture of the target machine). Ideally you would use the cmake equivalent variables CMAKE_SYSTEM_PROCESSOR (for target architecture, ie, the architecture you are building for) or CMAKE_HOST_SYSTEM_PROCESSOR (for the host architecture, ie, the architecture you are building on).
  • The env.sh script now defines the CC and CXX environment variables to point to our versions of clang/clang++. Previously we did not set these, which led some tools/builds that looked at those variables (notably in NPM) to use the local g++ (if any) rather than our compiler.
• Notable Changes
  • You can now build natively on aarch64 architectures. Previously, you needed to cross-compile on x86/64. Additionally, you can cross-compile for x86/64 on aarch64 architectures.
  • Qt GUIs now work on aarch64 architectures. Previously, they would not compile when building for aarch64, due to missing libraries. We now provide aarch64 Qt libraries.
  • Addition of a native-Python rbuf code generator, allowing you to serialize/deserialize rbuf objects in Python without the use of bindings.
  • The JavaScript Web libraries (for comms and serialization) are now packaged as NPM modules, making it possible to include in your repositories without copying/pasting code or playing other linking tricks.
  • Ark Telemetry gained the ability to store bags of objects, where objects consist of a string key and a JSON payload. This allows you to build things like reports or status objects that consist of many objects coming from different stages in your pipeline.
  • Third party libraries were updated, updating the version of AWS and cpython, and now including Kinesis, CUPS, and ncurses third party libraries.
  • You can disable tcmalloc linking (and status reporting) by setting ARK_ENABLE_TCMALLOC to FALSE. You typically need to manually link against tcmalloc to enable it, but if you included the perf package, you automatically got tcmalloc. You can now disable this.
• Stage Additions
  • The fault stage can now trigger on nested structures. Previously it only supported triggering off flat structures (or fields that were not nested within other objects).
  • The gps_fix channel was previously assumed to always be in root namespace. It now can exist in any namespace, and tooling/plugins will react accordingly. This allows multiple GPS devices to be present in a pipeline. The Swift and NMEA stages were updated accordingly.
  • The log writer stage and log amender stage gained the ability to log DataEnvelopes. Previously, these were ignored, as they were assumed to only come from a comms stage or log reader.
  • FaultConfiguration now includes a STRING_EQUALS operation to allow for exact comparison with string enumeration values. This accommodates for enumeration values that are similar, such as “IN_USE” and “NOT_IN_USE” to be compared.
  • The TcpClientStage gained the ability to use Unix domain sockets.
  • Added a CyberPower UPS stage, which can communicate with an appropriate UPS and log UPS status information (such as time remaining, voltage, etc).
• API Additions
  • The ark::process::Process class gained an exited() API to let you know if the process has exited without invoking the wait system call. It also gained the ability to assign search paths for the binary in the ProcessConfiguration structure.
  • Added a new parse_config_with_all_options API to parse a single YAML file configuration and return a mapping containing all of the options in said configuration file, pre-processed into concrete structures.
  • The Ark websocket code gained support for bi-directional communications (both in the ark::comms::HttpServer and in the ark::comms::WebSocketClient classes). Added support for a few additional pieces of the websocket spec, making it easier to connect with and communicate with other languages.
  • The Object class field_as_string and field_as_double methods gained the ability to introspect into nested structures using the dot notation (ie, parent_field.child_field) rather than needing to manually recurse.
  • When using data envelopes, you can now go from ‘raw bytes’ to JSON, without needing to know any type information. Previously, you’d need to know the type information yourself to do this transformation.
• Tooling Improvements
  • The ark-spy tool will no longer exit immediately if your pipeline isn’t running. It will loop and continue trying to connect until successful.
  • A new ark-supervisor-gui tool was created to offer a GUI visualization of the platform supervisor, allowing you live monitoring of running processes (and live stdout view).
  • rbufc can now read rbuf files from stdin and write generated code to stdout, making it easier to use in some build environments that lack files.
  • We now provide a parcel transformer for rbuf files in JavaScript – this allows you to transparent import rbuf files in your JavaScript code, and Parcel will automatically convert them into JavaScript for you, removing the need for a pre-processing step, and handling dependencies properly.
• Notable Bugfixes
  • The JavaScript rbuf generator and serialization/deserialization code had a number of bugs fixed which prevented you from fully using all rbuf objects in JavaScript.
  • The system fails more gracefully if libEGL cannot initialize due to missing hardware. Previously a std::abort was issued; now an exception is thrown.
  • The pcap stage previously returned packets that all fell on the same time in a somewhat random (but deterministic) order. It now returns those packets in the order in which they were read from the file.
  • The core::ArgumentParser class can now accept single dashes as arguments to both positional and option flags. Previously these were interpreted as options themselves.
  • Fixed a bug with reporting overall pipeline success/failure if you had multiple stages with soft shutdown tokens. In the event that one stage reporting shutdown (successfully) and other stages didn’t report anything, we would report failure instead of success.
  • The Ark CI ‘Doxyfile’ was placed in the root of the ‘ark’ directory, which meant that if you were using Ark as a subrepo and Doxygen on your repo, it would end up seemingly skipping ‘ark’ content, as our ‘Doxyfile’ is setup for XML output only. The file was moved into our docs directory to avoid this.
• Preview Changes
  • Addition of the ability to create Python virtual environments through cmake, and then add them as dependencies to other targets. This allows building a virtual environment containing rbuf messages, in-Ark libraries, and PIP packages (requirements.txt style). Note that presently the venv is built with the Ark version of cpython (3.10) which might be too old for your system python. If that is the case, make sure to add the Ark cpython into your PATH. See the docs for more details.
  • Addition of a simple “sidecar” interface, with Python as the first implementation. This allows you to create basic Python stages and run them deterministically (or in real time) as part of your pipeline. At the moment, only subscriber callbacks, publishing, and periodic timers are supported. Notably, performance, particularly around communications, is expected to improve, and configuration is not yet piped into the Python StageInterface in a meaningful way.

Release 2022.05.0

• Breaking Changes
  • Qt was upgraded from v5.15.2 to v6.3.0. This comes with a number of minor path and cmake changes. Notably, any use of the cmake qt5_X functions should be replaced with qt_X equivalents. Other scripts that were looking for Qt5 specifically will need to be updated to look at Qt6. Finally, the new version of Qt has additional dependencies that Qt5 did not have, make sure you check the README for additional packages that may need installed on your system.
  • The offload messages changed formats. The new version of Ark’s offloader and tools can not talk to older versions, and vice-versa.
  • The supervisor/deploy tool messages changed formats. The new version of Ark’s supervisor and deploy tools can not talk to older versions, and vice-versa.
• Notable Changes
  • Support in rbuf for the new variant type, allowing you to serialize objects that may be exactly one of a pre-defined list of objects.
  • Support in rbuf for associating comments with their schemas, fields, and enumerations, and then regenerating those in the generated C++ and JavaScript code, making the comments visible in tools such as VSCode or clangd.
  • The core write_string_to_file gained a new write options, Atomic, allowing you to write out your string content atomically (by writing to a backup file and renaming over the target file).
  • The serialization write_serializable_type_to_file and read_serializable_type_from_file have been enhanced. You can now pass along write options, allowing you to write them out atomically. They also include schema information (in a backwards/forwards compatible manner), allowing you to read a file written with this technique and get back an ark::serialization::Object which you can further process with reflection APIs.
  • clang was updated to v14.0.3, along with several third party packages. This brings in a few bug fixes, notably for the aarch64 architecture. The third party package updates were mostly for security fixes, along with a bump in boost to the latest (1.79).
• Stage Additions
  • Additional stages, API, and pcap support for communicating with SocketCAN devices. This allows you to connect to the device and send/receive packets, following the model of the TCP/UDP/USB/serial stages. It additionally reports CAN-specific status (such as BUSOFF).
  • A CAN packet splitter/filter stage, this will split up packets coming in over the CANbus into different channels, making it possible to filter packets before they get to their destination.
  • The PCAP stage has a few enhancements. It now emits statistics periodically, including processed packets, filtered packets (by address), and dropped packets. It also allows you to define a custom start time. Finally, you can play/pause it from the standard LogReader interface.
  • The FaultStage now allows for configuration to apply to a “family” of channels. The configuration allows for specifying a channel name via regex matching. This will allow a single fault configuration on “/lidar/.*/sweep_stats” to cover channels “/lidar/front/sweep_stats” and “/lidar/rear/sweep_stats”. This is especially useful for namespaced stages.
  • The offload stage can now read credentials from a file on disk, supporting offloading without a user needing to provide their own credentials.
  • The offload stage gained the ability to automatically offload logs as they complete. For robots that are constantly connected to the Internet, this allows for rapid and hands-off offloading of logs to the cloud.
  • The is_log_actively_used API now checks in /tmp/ark-active-logs for actively used logs, rather than just checkingin /tmp. This better separates out our files. Further, it now obeys the ARK_ACTIVE_LOG_RUNFILE_PATH environment variable, so you can output these files into a different location (such as /var/run) if desired.
• C++ GUI Improvements
  • Improved renderer layer management. Added ability to clear all graphical layers manually (previously this only occurred when reconnecting) and fixed a bug that caused non-namedspaced graphical layers to appear as if they were actually in a namespace.
  • Added an optional ability to track frame heading when following a frame, rather than just the X/Y position.
  • The hardware health plugin now displays host uptime, and the pipeline metrics plugin now displays pipeline uptime. Reformatted both to make better use of screen real-estate.
  • A new command line option was added, --settings-path, which allows you to adjust the path that settings are read from and written to. Another option, --disable-settings-output, prevents updating the settings file on exit. These can be used together to create pre-made settings files in your releases.
  • A bug was fixed in the perspective display that made it difficult to measure points or translate the camera (the orthographic display was unaffected).
  • The Magnum renderer gained the ability to display 3D models.
  • As mentioned in breaking changes, Qt was upgraded from Qt5 to Qt6.
• Command Line Tool Improvements
  • Various tools that interact with the pipelines (ark-spy, ark-put, the GUI system) all obey a new ARK_PIPELINE_HOST environment variable now. This allows you to set a variable in your current shell so all of the tools will talk to that robot, without needing to manually specify --host, simplifying command line interaction with the robot.
  • Added a new ark-cat-rbuf-file command line tool, which allows you to load rbuf-serialized files from disk and display them as JSON files.
  • Added a new --exit-after-initialize command line flag to all pipelines using the main system. This allows you to verify that your pipeline will parse configuration and initialize properly without starting the pipeline.
• Notable Bugfixes
  • Fixed a bug that could cause silent corruption in logged split files. This would occur if you added a large number of channels to a single column, causing indexes to be corrupted and some data to be dropped from the log.
  • The log offloader previously only received offload credentials once, on startup. This prevented it from working properly when the credentials were invalid or expired, requiring you to restart the pipeline. Credentials are now forwarded withe very offload request.
  • Fixed a bug with parsing rbuf files without a namespace. This could cause unqualified fields to be prefixed with ‘::’, which would cause lookup failures.
  • The log plotting tool can now open logs directly from the cloud, rather than requiring you to download them first. This was due to incorrectly parsing the command line arguments.
  • The JSON decoder in the GUI used to reset the scroll position on update, making it hard to watch large messages. This has been fixed.
  • The log reader is more efficient with memory when reading logs with many thousands of split files stored in a single column.

Release 2022.04.0

• Major Changes
  • Updated to the latest version of clang (14.0.0). This brings in a handful of new C++20 features, see more here. This also upgrades cmake to the latest version, along with many third party dependencies (including python, zlib, OpenSSL, absl, backtrace, and ceres).
  • Addition of single-shot timers to the system. This allows you to queue work in your initialize function that will execute as soon as the pipeline starts.
  • Addition of the [[removed]] attribute to fields in rbuf. This allows you to prevent fields from showing up in codegen, while still maintaining backwards/forwards compatibility.
  • Removal of the mapping module from the repository. This was intended for demonstration purposes. Some of the existing APIs (specifically, visualization of pose axes) was migrated into ark::gui.
• Build Additions
  • The build system can now output to custom build directories, rather than the hard-coded build used previously.
  • The build system is now more CUDA-aware. A number of changes were made that make it much easier for you to integrate with a local CUDA build and use that to compile and use .cu files. A future release will include bundled CUDA support.
  • The make script can now easily disable various sanitizers and configuration options, instead of just being able to enable them.
• Stage Additions
  • Added a new RTSP stage for communicating with RTSP-based cameras (video only). This currently only supports H264-based cameras, not JPEG cameras.
  • Added a new “TCP Server” stage which allows you to make a stage that listens for connections and forwards the resulting messages on to the rest of the pipeline.
  • The TCP and UDP stages gained the ability to configure kernel buffer sizes, reducing the liklihood of dropped packets on particularly overloaded machines.
  • The PCAP stage now tracks and publishes statistics on what channels it is operating on and what channels it is filtering.
  • The log reader stage now supports storage of logs at different levels then a top-level S3 bucket. Useful for if you are storing data logs in your own bucket.
• Notable Bufixes
  • Fixed a bug where event timers would not trigger in the real time executor if no other messages were being received and no other timer was executing. This only affected the real time executor, not the simclock executor.
  • Fixed a bug in the log reader/log writer where the global stop time may not be present if a log was terminated via a software or hardware crash. Additionally, this removes some ambiguity when the log start and log end time models differed dramatically (they are now combined into one model).
  • Fixed a crash in the log plotting tool in some situations when you have exceeded the eight plot limit. This would typically occur if you selected and deselected a number of variables after the limit had been reached.
  • The camera plugin now plays back H264 data that is not entirely made up of keyframes far more reliably on lossy links.