Logging Overview

The logger allows you to efficiently record data coming from many different sources in such a way that it can be accurately played back again for later analysis and debugging.

Conceptually, the log files are divided into two different files:

  • splits - Which contain all of the raw data you have recorded
  • manifests - Which describe how to assemble the splits into a log

When accessing a log, you will always do so through manifests.

The log writer makes a best effort at writing logs out in a power-safe fashion. In other words, if your software crashes, or your host logger board loses power, the logs should not become corrupted (this does depend on underlying hardware, operating system, and your chosen filesystem).

Organization

Physically, log data is organized into a series of columns and split files. Each column can contain one or more channels. Split files contain sequential data over the channels that are configured for that column.

Dividing data into “columns” allows for more efficient data reading. If you know that data is big, or that a single channel will typically be accessed, you should make it into its own column.

In other words, if you want to access “imu” data frequently by itself, place it in its own column. Tools will then only need to access the split files in that column to read through the IMU data. This can save a lot of time (and downloads!).

The only downside to using as many columns as possible is that it can make parallel fetching of data from remote sources slower. For example, if you have 10 columns, and want to access data from all of those columns, all of them will need to be on disk.

Snippets

You can carve up a log into “snippets” to make it easier to send to other people. For example, if you have a very long log, let’s say an hour, you can create a 1 minute snippet from that log by using ark-logtool:

./build/ark-logtool 32bf130f-a4b5-4f68-94a8-3d8802300262 -n 630 -x 690 -s my_snippet

This will take all of the data between timestamps 630 and 690 in the log 32bf130f-a4b5-4f68-94a8-3d8802300262 and save it to the my_snippet directory. You can then upload it to the logs bucket with:

./build/ark-offload-log ./my_snippet/manifests/<created snippet GUID>

Snippets do not duplicate any data – they simply update manifests. They are nearly free from a data perspective, and are a convenient way to exchange events of interest.

Amendments

Amendments are a mechanism to replace some data within a data log (or to otherwise augment a log with additional data). This is typically used to either “patch” a log (replace data with better data), or during simulation, to log algorithm outputs efficiently.

The advantage to amendments is that the original log data is not duplicated. So, if you want to write out new algorithm data, but do a log playback as if that log was recorded on the robot, you can generate an amendment – the only new data consumed will be for the data generated during the simulation, not any of the previously existing data.

Latched Messages

You can define messages as ’latched’ in the log writer stage configuration. See that page for more details on how to configure latched settings.

Whenever you make a query, the most recently received latched message of each type prior to the start of the log or query will be played.

In other words, if you have two latched messages, one being a calibration, and one being a configuration, and those are sent once on startup with time T=0, and you log begins at time T=10, you will receive the calibration and configuration before you receive any messages at time T=10.

This can be used to avoid republishing data periodically to ensure it is logged. There is some cost to disk space – the split file that contains the latched message will need to be downloaded to replay it (or made part of any amendment or snippet).

Event Buffers

Message channels can be configured to queue up data and only log to disk when an event occurs. This can be helpful if there are large messages that you only want to actively record around some event occurance. For example, you may want to queue up camera images, but only write them to disk when an event occurs, such as an obstacle being detected. You can use the LogWriterEventBufferConfig to configure certain messages to queue and only be written after an event. Please note that the LogWriter must be actively logging to these events.

The event_buffer queue depth is configured in terms of time, rather than max capacity. Here is a sample event buffer configuration for the LogWriterStageConfiguration:

 default:
   ...

   event_buffers:
     # Define a 15 second buffer of Camera Images before an event
     - name: "CameraBlackBox"
       storage_time_ms: 15000  # 15 second buffer window

   columns:
     ...
     - column_name: "camera_left"
       compression_type: "none"
       included_channels:
         - name: "/camera/left/compressed_image"
           event_buffer_name: "CameraBlackBox"

     - column_name: "camera_right"
       compression_type: "none"
       included_channels:
         - name: "/camera/right/compressed_image"
           event_buffer_name: "CameraBlackBox"
   ...

The storage_time_ms defintes the time window to buffer any messages registered to the defined event_buffer.

To trigger an event, a LogWriterCommand can be published with the event_buffer fields set to activate the event_buffer to log the queued data. You can also define an amount of time after the event to also be recorded. This gives you the option to record messages before and after the defined event. The LogWriterCommand.event_buffer_duration defines a time duration after the event.

    // User-defined event has been detected.  
    // Inform the LogWriter to activate the event_buffer and record an additional 5 seconds of data.
    ark::logging::LogWriterCommand cmd;
    cmd.command = ark::logging::LogWriterCommandType::EnableEventBufferForDuration;
    cmd.event_buffer_name = "CameraBlackBox";
    cmd.event_buffer_duration = std::chrono::seconds{5};
    log_cmd_publisher_->push(cmd);

This example will tell the LogWriterStage to record 15 seconds before an event and 5 seconds after.

Please note that the data channels registered to the event_buffers will only be recorded during defined event_buffer time windows. The registered messages will not be logged in any other log time window.