Config Parsing

Config Files

Each configuration is defined through an rbuf file. The configuration system will then load content in a YAML file into the concrete C++ structure defined by the rbuf.

For a simple example:

---
config:
  default:
    my_parameter: 10

If you had an rbuf that looked like:

schema MyConfig
{
    uint32 my_parameter;
}

Then you could load the above YAML into that rbuf, and my_parameter would equal 10.

Durations have the ability to have a ‘suffix’ applied to them. While they are stored as nanoseconds in the rbuf/generated code, they can be written in seconds, minutes, or milliseconds by applying the appropriate suffix. For example, 1_s means “one second” and 50_ms means “50 milliseconds”.

Config Options

Configuration files can also have ‘options’ which allow you to specify different parameter sets that users may be interested in switching between.

For example:

---
config:
  default: &params
    paramater_a: 10
    paramater_b: 20
  extra:
    <<: *params
    parameter_b: 15

In this example, if you used the default option, you would get 10 and 20 back, for parameters a and b respectively. If you asked for the extra option, you’d get back 10 and 15. This is because extra has overridden parameter_b with the value 15.

This makes use of the YAML anchoring syntax.

Config Metadata

Configuration files can have the concept of ‘metadata’ which is not normally parsed. These metadata objects are typically full structures of their own, and contain data not directly associated with the configuration, but related to it.

For example:

---
config:
  metadata:
    stage_config:
      comms_namespace: "/imu"
      class_name: "ark::comms::UdpSocketStage"

  default:
    parameter_a: 10

In this example, there is a metadata block known as stage_config. This can be processed and fetched into the ark::pipeline::StageConfig structure. This particular metadata is used by the pipeline to configure namespacing, and to help downstream customers understand what type of stage this configuration is.

Config Package

Conceptually, a ConfigPackage consists of many different individual configurations, which can then be fetched at execution time.

For each configuration, you define the path to the YAML file to load, the option to make use of from that file, and any “overrides”.

As a quick example:

---
package:
  configs:
    MyConfig:
      path: "/path/to/my/config.yml"
      option: "extra"
      overrides:
        parameter_a: 30
      metadata:
        human_readable:
          name: "Special Configuration"

Building on the options example, this would load the config YAML you specified, choose the extra option for you, then change parameter_a to 30, giving you back 30 and 15 for parameter_a and parameter_b, respectively.

You can fetch this configuration by using the following API:

#include "ark/config/config_package.hh"

auto package = config::ConfigPackage::from_file("/path/to/your/package.yml");

auto config = package.get_config<MyConfig>("MyConfig");

Note that you can optionally provide a list of search paths to from_file – that will allow the config package loader to search in alternate locations for included files. It will still search the current working directory if the file cannot be found otherwise.

You can further fetch metadata from it by using this API:

auto metadata = package.get_metadata<MyHumanReadableMetadata>("MyConfig", "human_readable");

Note that if the metadata is not present, this will return a default-initialized structure.

A ConfigPackage is assumed to contain all of the configuration for a pipeline, so, all of your stages will have configuration stored in this package.

Note that for testing, it is a lot of boilerplate to make a ConfigPackage for every test that you are writing. You can add a stage and a paired configuration to a pipeline with this helper:

#include "ark/pipeline/config.hh"

MyConfig config_struct;
config_struct.parameter_a = 10;

pipeline::add_stage_with_config_to_pipeline<MyStageClass>(pipeline, config_struct, ...);

This works similarly to add_stage() - all arguments after the configuration struct are just forwarded to the stage constructor, and the API returns a pointer to the created stage.

Using with Pipelines

Pipelines can have a configuration package set, and these are provided to each of the stages through the StageInterface system.

Set the config package on a pipeline like this:

pipeline.set_config_package(package);

The package is copied and stored in the pipeline as a shared pointer. You can retrieve this from your stages through the StageInterface passed into your stage in initialize():

#include "ark/pipeline/config.hh"

void initialize(ark::pipeline::StageInterface &interface)
{
    auto config = pipeline::get_stage_config<MyConfig>(interface);
}

The above code fetches the config from the package that was registered with the name of the stage. You can fetch other configurations from the package if you like as well, using something like this:

auto config = interface.config_package()->get_config<MyConfig>("Name of Config to Use");

This will return the configuration under ‘Name of Config to Use’, deserialized into the MyConfig structure.

See the pipeline section on configuration for more details.

Config Package Configuration

You can further adjust what configuration your system gets by configuring how the config package loads. You can do this via either command line interface or C++ APIs.

If you are using ark::main routines for your pipeline, you will have two options:

  • --config-option - Allows selection of an ‘option’ for an individual stage/config block
  • --config-override - Allows you to override a specific parameter within a stage/config block

As an example, say you have a sensor in a stage named SensorStage and it has two options (default and debug). For example:

---
config:
  default: &params
    paramater: 10
  debug:
    <<: *params
    parameter: 15

You can swap from the default option to debug option with:

./my-pipeline --config-option SensorStage=debug

If you wanted to just override the parameter itself, you could do:

./my-pipeline --config-override SensorStage/parameter=20

You can specify overrides and options multiple times.

These can be configured programmatically when loading config packages via the ConfigPackage::from_file method by populating the ark::config::ConfigPackageParseConfig structure. It allows you to adjust the same options, with the same syntax.

Serializing

You can serialize and deserialize a full ConfigPackage, allowing you to log your configuration (or restore it later for viewing).

The APIs from_wire() and to_wire() will pass you back a serializable ConfigPackageWireData structure. This is an rbuf that can be written to logs or sent over the wire, containing all of your configuration data (in YAML form).

Parsing Individual Configurations

If you simply want to parse a YAML file into an rbuf, you can use the parse_config API on its own.

Aside from being YAML, the only real benefit over just parsing JSON is that the type checking and error messages are a bit more sensical.

As an example, say you have this YAML file:

---
config:
  default:
    camera_path: "/dev/video0"
    image_format: "z16"
    desired_frame_rate: 15

You can parse that into a structure with this C++ code:

#include "ark/config/config_parser.hh"

auto config = config::parse_config<video::VideoCaptureStageConfiguration>("config.yml");

Note that you can also request a specific option in your config file, as a second argument to parse_config:

auto config = config::parse_config<video::VideoCaptureStageConfiguration>("config.yml", "second_option");

By default, the default configuration is returned, just as with the package.

If you wish to return all of the configurations (options) at once (for example, to enumerate all of the names, or just because you want to load multiple configurations from the same file), you can do so with the parse_config_with_all_options API:

---
config:
  default:
    camera_path: "/dev/video0"
    image_format: "z16"
    desired_frame_rate: 15
  second:
    camera_path: "/dev/video1"
    image_format: "rgb"
    desired_frame_rate: 15

Can be parsed with:

#include "ark/config/config_parser.hh"

auto configs = config::parse_config_with_all_options<video::VideoCaptureStageConfiguration>("config.yml");

The configs object (in this case) is a std::map<std::string, video::VideoCaptureStageConfiguration> object, with the keys being default and second.

Using Variants

You can use variants within the YAML files. For example, if you have a configuration that looks something like:

schema MyTypeA
{
    uint64 member_var_a;
    uint64 member_var_b;
}

schema MyTypeB
{
    string member_var;
}

schema MyConfig
{
    variant<MyTypeA = 1, MyTypeB = 2> my_variant;
}

You can configure it in YAML as such:

config:
  default:
    my_variant:
      type: "MyTypeA"
      value:
        member_var_a: 10
        member_var_b: 20

Or alternatively,

config:
  default:
    my_variant:
      type: "MyTypeB"
      value:
        member_var: "This is now type B."

The variant will be set properly in your C++ code based on the type field, as the nested object will be set from the value field.

Optional Values

Any fields that are marked as optional in your rbuf will be left unset if they are not defined in your configuration file. If they are not marked as optional, they will be initialized to their default values (typically empty or 0).