Local Robot Storage

Concept

The LocalStorageIoStage can be added to your pipelines to provide a mechanism to store key/values persistently across reboots on your robot. This stage also provides mechanisms to subscribe to updates on these keys, as well as fetching those keys on startup.

This makes it ideal to store things like per-robot configuration values or other settings.

Stage

You can configure your stage in your config package like so:

    ConfigLocalStorage:
      metadata:
        stage_config:
          comms_namespace: "/local_storage"
      overrides:
        storage_namespaces:
          - name: "config"
            hmac_signature_key_value: "ASecretSigningKey"
        local_storage_base_path: "/mnt/my/robot/storage"
        max_value_size_b: 262144

This assumes you previously added the local storage stage to your pipeline with:

#include "ark/local_storage/stages/local_storage_io_stage.hh"

pipeline.add_stage<ark::local_storage::LocalStorageIoStage>("ConfigLocalStorage");

A brief explanation of parameters:

  • storage_namespaces - You might have multiple storage stages using the same base path. This allows you to separate keys and configuration.
  • local_storage_base_path - The base path local files are written out to.
  • max_value_size_b - The max size, in bytes, of any value.

For each storage namespace, there are a number of parameters you can configure:

  • name - The name of the namesapce.
  • hmac_signature_key_value - HMAC key (hardcoded into the config file, not ideal).
  • hmac_signature_key_secret_name - Name of the Catalog Secret that contains an HMAC key.
  • rsa_public_key_path - Path to an RSA public key to verify signatures.

Note that on product robots, you would ideally get the signature key from some other source, such as Catalog. For that, do not provide a hmac_signature_key_value, and instead, use a hmac_signature_key_secret_name parameter. This will cause the stage to retrieve the signature key from the catalog secrets set.

For additional security, or in case you wish to load local storage at init time, when an HMAC key is not available, you can sign your data with an RSA key pair. In that case, the rsa_public_key_path points to the path of the RSA public key that can be used to verify the data.

Both an HMAC and RSA are optional (or you can use both).

Pipeline Interaction

The stage provides the following channels:

  • notification - A message is emitted containing key/value pairs as a keys are updated or loaded.
  • request - A channel to send requests for writing, reading, or listing keys.
  • response - A channel that receives responses to the requests.

Each request/response is tagged with a reference ID, which you can use to filter responses to your particular requests.

On startup, the LocalStorageIoStage will read all existing keys and publish them as notifications.

A class exists to simplify the process of listening for keys, you can use the LocalStorageSubscriber class in your stage, like so:

#include "ark/local_storage/local_storage_subscriber.hh"

LocalStorageSubscriber subscriber_host;

subscriber_host.add_subscriber(stage_interface, "ConfigLocalStorage", "StorageNamespace", "KeyOfInterest", [](const auto &update) {
    std::cout << update.key << " was updated to have the value: " << update.value << "\n";
});

At this point, you will see ‘KeyOfInterest was updated to have the value…’ whenever that particular key was updated. You can listen for any number of keys, across any number of stages, with this subscriber.

Tools

If you’ve added the stage above to your pipeline, you can use the ark-local-storage-tool to interact with local storage:

./build/ark-local-storage-tool --host localhost:8080 --namespace config --set-key 'MyConfig=5'
Key written successfully.

./build/ark-local-storage-tool --host localhost:8080 --namespace config --get-key MyConfig
5

./build/ark-local-storage-tool --host localhost:8080 --stage-namespace /local_storage/config --list-keys
DATE                     HASH             SIZE     SIG       NAMESPACE  KEY
2023/11/20 15:01:58 EST  4b6e8286fd734bf  23696    RSA       config     MyStageConfig
2023/11/20 15:04:38 EST  83527ded182c33a  7        HMAC/RSA  config     Key1
2024/03/14 23:58:05 EDT  5491324de89c0ff  3110     none      config     MyConfig

This does require you to know the stage namespace ahead of time, so you may want to wrap storage updates with your own tooling.

If you wish to sign data (for a namespace that is configured with RSA verification), then you would typically use catalog. For example:

./build/ark-local-storage-tool \
  --host localhost:8080 \
  --namespace config  \
  --sign-with-key config/release \
  --set-key-from-path MyStageConfig \
  --key-path ./project/pipelines/my_stage_config.yml

This will set the ‘MyStageConfig’ key equal to the content of the YAML file, signed with the key config/release via the Catalog.