Pipeline Configuration

Every pipeline contains a config::ConfigPackage that is considered to be the main source of configuration for the entire pipeline.

This ConfigPackage can contain configuration for each stage. This works by generating a ‘config’ with a name that matches your stage’s name. For example, let’s say you have two stages, both UdpSocketStages:

package:
  configs:
    # This is the configuration for the first stage,
    # which we named arbitrarily.
    FirstUdpStage:
      path: "ark/lidar/velodyne/stages/velodyne_data_socket_config.yml"

      # Override the defaults and set the address you want
      # to listen to
      overrides:
        address: "10.0.0.1:2368"

      # Configure the namespace you wish the socket stage to listen
      # on here.

      metadata:
        stage_config:
          comms_namespace: "/udp1"

    # This is the configuration for the second stage.
    SecondUdpStage:
      path: "ark/lidar/velodyne/stages/velodyne_data_socket_config.yml"

      overrides:
        address: "10.0.0.2:2368"

      metadata:
        stage_config:
          comms_namespace: "/udp2"

There is a lot going on here. First of all, this configures two separate stages, FirstUdpStage, and SecondUdpStage. Both of those point to the same base configuration, a Velodyne data socket configuration.

We then override just the address for both of them – FirstUdpStage now has an address of 10.0.0.1:2368, and SecondUdpStage now has an address of 10.0.0.2:2368.

Finally, we configure metadata for both. We set the comms_namespace with in the stage_config block to be /udp1 and /udp2, for the first and second stage respectively.

Now, when you assemble the pipeline, you can do this:

pipeline::Pipeline pipeline;

auto config = config::ConfigPackage::from_file("/path/to/your.yaml");
pipeline.set_config_package(config);

pipeline.add_stage<comms:UdpSocketStage>("FirstUdpStage");
pipeline.add_stage<comms:UdpSocketStage>("SecondUdpStage");

Configuration is applied to the stages by doing name-matching.

Within the stage itself, the configuration is passed in with the StageInterface, in the initialize call:

#include "ark/pipeline/config.hh"

void MyStage::initialize(const pipeline::SocketInterface &interface)
{
    auto config = pipeline::get_stage_config<UdpSocketStageConfig>(interface);
}

You can also get the config package directly to fetch configurations from other stages (or even configurations not related to any particular stage, like calibration), by using the StageInterface::config_package() API. This returns a shared pointer to the package, and you can use the normal get_config and get_metadata APIs on that to fetch concrete configuration.

Finally, if you don’t care if configuration is defined in your pipeline or not, and are OK using the defaults, you can fetch your configuration with:

#include "ark/pipeline/config.hh"

void MyStage::initialize(const pipeline::SocketInterface &interface)
{
    auto config = pipeline::get_stage_config_or_default<UdpSocketStageConfig>(interface);
}

If configuration is present, it will return that. If not, a default config of the given type will be constructed and returned.

Pipeline Config

The entire pipeline may be configured as well. The executors, in particular, use metadata attached to the pipeline configuration.

The SimClockExecutorConfig provides the ability to throttle the executor to the system clock. As an example:

package:
  configs:
    Pipeline:
      metadata:
        simclock_config:
          throttle_to_system_time: 1.0

If this value is set to 1.0, it will throttle to match the speed of the system clock. A value of 0.5 would run twice as fast, while a value of 2.0 would run twice as slow.

Note that if you are using main, you can enable this at runtime with the command line option --throttle-to-system-time 1.0.

See all executor configuration in ark/pipeline/executor_config.rbuf.

Config Stage

A configuration stage (ConfigPackageStage) is provided to publish the configuration package to anyone that is interested (typically the GUI, for visualization of the configuration).

This is published over the /config/package channel.

Logging Config

The LogWriterStage will automatically log the config package when a new log is started. It is logged into the /config/logged/package channel.

Programmatically adding Stage Config

It can be useful to create pipelines without needing to use YAML files, particularly for unit testing.

Helper APIs are present to allow you to create a config and add a stage with that config to a pipeline programmatically.

Example:

#include "ark/pipeline/config.hh"

pipeline::Pipeline pipeline;

comms::UdpSocketStageConfig config;

config.address = "10.0.0.1:2368";
config.reuse_address = true;
config.group_size = 30;

pipeline::add_namespaced_stage_with_config_to_pipeline<comms::UdpSocketStage>(pipeline, config, "/ns1");

The pipeline::add_namespaced_stage_with_config_to_pipeline API will add your stage to the pipeline, set the necessary metadata for the namespace you wish it to be in, and assign the configuration to it. Configuration is serialized into YAML for storage, so the ConfigPackage is identical between using YAML or this API.

If you want to just use the default namespace, you can save yourself some typing with the pipeline::add_stage_with_config_to_pipeline API. Similarly, if you don’t have any config, but want to namespace your stage, you can use pipeline::add_namespaced_stage_to_pipeline API.

This routine works identically to Pipeline::add_stage, in that it forwards the remaining arguments on to your stage’s constructor, and returns a shared pointer to the newly created stage.

Programmatically adding Pipeline Config

This is almost identical to stage configuration. It enables you to programmatically configure executor or pipeline config:

#include "ark/pipeline/config.hh"

pipeline::Pipeline pipeline;

pipeline::SimClockExecutorConfig config;
config.throttle_to_system_time = 1.0;

pipeline::add_pipeline_metadata_to_pipeline(pipeline, "simclock_config", config);