Log Writer Stage

The log writer stage allows you to record data from channels into the Ark logging format.

Configuration

You need to configure which channels will be listened to, and how that data will be stored. This can all be done through the LogWriterStageConfiguration structure, which can be populated with YAML:

---
config:
  columns:
    - column_name: "data"
      included_channels:
        - name: "/gps"
        - name: "/imu"
        - name: "/config"
          latched: true

    - column_name: "image"
      compression_type: "lz4"
      included_channels:
        - name: "/image"

Pass this into the LogWriterStage when you construct it:

auto config = config::parse_config<logging::LogWriterStageConfiguration>("config.yml");

pipeline.add_stage<logging::LogWriterStage>(config);

This configures two columns, one listening for data on “/gps”, “imu” and “/config” channels, and the second on “/image”. The “/image” channel is LZ4 compressed, and the “/config” channel is latched.

Within the included_channels list, you can specify either a name or a pattern to be logged. If you choose to use name, the channel must be an exact match. This is the default and preferred, due to its explictness. If you wish, you can use pattern, which will include all channels that match the pattern (as a regular expression).

Per-Directory Logs

You can emit per-directory logs (ie, each log exists in its own directory) with the configuration option path_prefix. This path is also a template string that allows you to use a handful of variables to generate dynamic directories.

For example: /tmp/logs/${YEAR}-${MONTH}/${YEAR}-${MONTH}-${DAY_OF_MONTH}/${LOG_IDENTIFIER}

Becomes: /tmp/logs/2021-11/2021-11-18/89120123-6781-3456-123491235721

Variables:

• YEAR (numeric year, ie, 2022)
• MONTH (numeric month, where ‘1’ is January)
• DAY_OF_MONTH (numeric day, dated from the 1st of the month)
• DAY_OF_YEAR (numeric day, dated from the 1st of the year)
• HOUR (hours, 00-23)
• MINUTE (minutes, 00-59)
• SECOND (seconds, 00-59)
• LOG_IDENTIFIER (log identifier GUID)

Interacting

You need to send messages to the logger so that it begins logging, it does not log by default.

Look at log_writer_controls.rbuf for the schema of the control message – it allows you to specify if you want the logger to begin or end logging, and allows you to specify a log name.

Commands arrive in via the control channel.

If you wish to start a new log, but avoid dropping any existing data, you can use the ‘RestartLog’ command. This will stop the current log and start a new log, all in one step. It will also ensure that any data that was buffered but unwritten for the old log will be pushed to the new log, so you will not “lose” any messages that were going to disk.

Metrics

The log writer stage produces a message at 1Hz that indicates the state of the logger – including data rate, objects written, flush intervals, and more. This is published on the statistics channel.