Web Log (awlog)

Ark now provides the awlog format (for Ark Web Log). This is primarily intended to create optimized single-file logs that are suitable for playback in our webviz suite of tooling, but they do contain much of the same core functionality as ark::logging logs (the primary features lost are zero-copy creation of snippets and any type of amendment support).

Presently, all data in an awlog is compressed with zstd, but that is expected to be made configurable at some point.

Tooling

There are two primary awlog-related tools right now:

  • ark-awlog-tool - A tool to inspect awlog files
  • ark-awlog-convert - Converts an ark::logging log into an awlog losslessly

Many of the command line options for ark-awlog-tool are identical to ark-logtool. As a quick example of inspecting a log:

./build/ark-awlog-tool ./391d3934-1d10-4000-a4d3-98c2e90d553e.awlog -l
MIN            MAX            COUNT      RATE       SIZE(MB)   TYPE                                          NAME
236.747        238.542        10         5.6        0.00       ark::debuglog::DebugLogOutputGroup            /debuglog
233.170        242.101        9          1.0        0.00       ark::logging::LogWriterStageStatistics        /logger/statistics
233.588        242.989        142        15.1       6.27       ark::image::Image                             /multisense/forward/left/compressed_image
233.747        242.757        10         1.1        0.00       ark::time::GlobalTimeEstimate                 /time/global_time
233.007        242.997        1372       137.3      0.16       ark::PushTransformMessage3dStamped            /transform_network/transform
233.453        238.474        2          0.4        0.00       ark::version::VersionInformation              /version_information

You can also print information out:

/build/ark-awlog-tool ./391d3934-1d10-4000-a4d3-98c2e90d553e.awlog -i
Log Id:    391d3934-1d10-4000-a4d3-98c2e90d553e
Common Id: aee21401-b25f-4bbb-9d97-c51e8d5bf5f2
Name:      TestPlanRetail

Start:    0.000000
End:      242.996527
Duration: 242.996527

Any metadata that is associated with the log will also be printed in this case.

Finally, you can dump index entries out:

./build/ark-awlog-tool ./391d3934-1d10-4000-a4d3-98c2e90d553e.awlog -e
[242.654744] /multisense/forward/left/compressed_image (46337 bytes)
[242.696959] /transform_network/transform (106 bytes)
[242.706583] /transform_network/transform (106 bytes)
[242.714025] /transform_network/transform (150 bytes)
[242.721551] /multisense/forward/left/compressed_image (46367 bytes)
[242.746373] /transform_network/transform (106 bytes)
[242.756619] /transform_network/transform (106 bytes)
[242.756850] /time/global_time (21 bytes)

Converting Ark Logs

You can use the ark-awlog-convert tool to convert an ark::logging log into an awlog. It takes parameters for the input log and output log, and then you can further filter things down by time range and allowed/blocked messages.

This allows you to losslessly translate logs, or to take snippets of logs (keeping only certain channels or time ranges). Since awlog files are typically used for the web, it can be a nice way to reduce the size of your logs to make them download faster.

Logging from the Pipeline

You can also use the ark::awlog::AwlogWriterStage to write out awlog files directly from your simulation. This can be useful for running batch simulations or other translation jobs, where you can generate an awlog from simulated data, upload it to AWS, and then let others view the results (without having your exact build of software).

See the stage documentation for more details.