Tools

Log Tool

The ark-logtool provides a few functions to help make sense of a log. You can point the tool towards the path of any log, and then query it for its time range, the objects being logged, its manifest, or even print out and deserialize various channels contained within the log.

It has a few major pieces of functionality:

  • dumping the contents of the log (or channel) as JSON
  • displaying index entry metadata (timestamps, compressed sizes, etc)
  • printing the manifest out as a JSON structure
  • printing the time span and duration of the log out
  • listing all of the channels (and their sizes/types)
  • Creating amendments with a variety of options

For commands that dump the contents of logs, you can restrict what you display by channel name or time range.

Here’s an example of dumping channels in a test log:

dtascione@desktop:~/ark$ ./build/ark-logtool ~/test_log/manifests/7c3b1ef4-5575-4491-919d-5b05d0f15ee6 -l
MIN            MAX            COUNT      RATE       SIZE(MB)   TYPE                                   NAME
17446.862      17463.864      18         1.1        0.00       gps::GpsFix                            /gps_fix
17447.030      17464.528      18         1.0        0.00       logging::LogWriterStageStatistics      /logger/statistics
17446.340      17464.729      180        9.8        316.41     video::CameraImage                     /realsense/color/image
17446.645      17464.110      18         1.0        0.00       video::VideoDeviceStatistics           /realsense/color/statistics
17446.391      17464.722      235        12.8       413.09     video::CameraImage                     /realsense/depth/image
17447.257      17464.122      17         1.0        0.00       video::VideoDeviceStatistics           /realsense/depth/statistics
17446.337      17464.777      9347       506.9      0.51       imu::RawImuData                        /realsense/imu/raw_imu_data
17447.156      17464.176      18         1.1        0.00       imu::ImuDeviceStatistics               /realsense/imu/statistics

Dumping a log as JSON

The ark-logtool can dump any channel as a JSON message, which is formatted in such a way that you can post-process it with jq or other tools:

./build/ark-logtool ~/logs/a49aa8a4-0e58-4bb7-8101-e67ef2e54fc2 -dc /robot/wheel_encoder_data
[
{
  "capture_time_ns": 1730823676000,
  "ticks": [
    22758,
    64267
  ]
},
{
  "capture_time_ns": 1730833680000,
  "ticks": [
    22758,
    64267
  ]
},
{
  "capture_time_ns": 1730843678000,
  "ticks": [
    22758,
    64267
  ]
}
]

Piping into jq to show the two ticks next to each other:

./build/ark-logtool ~/logs/a49aa8a4-0e58-4bb7-8101-e67ef2e54fc2 -dc /robot/wheel_encoder_data | jq '.[] | "\(.ticks[0]), \(.ticks[1])"'
"22758, 64267"
"22758, 64267"
"22758, 64267"

The JSON Lines (jsonl) format is also supported by adding the option --format-json-lines. This is strongly recommended for any streaming or parallel processing use cases. This format is functionally equivalent to using the standard JSON format and piping it to jq '.[]

./build/ark-logtool ~/logs/a49aa8a4-0e58-4bb7-8101-e67ef2e54fc2 -dc /robot/wheel_encoder_data --format-json-lines
{"capture_time_ns":1730823676000,"ticks":[22758,64267]}
{"capture_time_ns":1730833680000,"ticks":[22758,64267]}
{"capture_time_ns":1730843678000,"ticks":[22758,64267]}

Note that using the option --include-timestamps will inject timestamp messages on standard output, and the data can no longer be processed by jq. This applies even when using --format-json-lines.

CSV Extraction

You can dump individual channels to CSV files with the ark-extract-csv tool. This tool can only dump “primitive” fields from a message (not nested objects or lists). It constructs one column per field in your message.

For example:

$ ./build/ark-extract-csv ~/outside_logs/manifests/9aef0dd9-3852-4568-bb52-6cd56239b4a9 -c /gps_fix
indexed_time,fix_time_utc_ns,latitude,longitude,altitude,satellites_tracked
1464.676630,1604867005000000000,40.397813,-79.769104,232.900000,9
1465.680934,1604867006000000000,40.397813,-79.769103,233.000000,9
1466.676053,1604867007000000000,40.397814,-79.769103,233.000000,9
1467.677721,1604867008000000000,40.397814,-79.769103,233.200000,9
1468.673769,1604867009000000000,40.397814,-79.769103,233.500000,9

Amendments

The ark-logtool allows for easy generation of amendments for common scenarios.

  • Create a new manifest obeying filtered time and channel inputs.
  • Renaming channels from one string to another. This allows you to ‘repair’ a log that used the wrong channel name or had channel names change over time.
  • Set the name of amendments, making it a bit easier to organize log data.
  • Update channel to be marked as latched, even if it was originally recorded as non-latched
  • Update the organization identifier, which is a required field for uploading logs to Ark Catalog

Plots

Built-in tools allow automatic plotting of fields within your messages. The ark-log-plotter-gui tool allows you to visualize messages from the entire log. Data is plotted according to the logged timestamp. Users can zoom in to inspect details as needed.

Log Plotter Gui Tool