Reading Logs

You can read through logs with the LogReader class. This class follows some database conventions – you construct a query, which specifies time ranges and channels you are interested in, and then step through the results.

It is best to only ask for the data and time you need – this allows the log reader to efficiently request only the data you asked for on disk, rather than loading everything.

For example, if you have a large log with many image channels, and you only want to access one channel, just request that channel. It will be far faster then requesting all channels and doing the filtering yourself.

API Example

You can read a log by using this sample code:

ark::logging::LogReader reader("/path/to/manifest");

// Now instantiate a read cursor -- this reads over all data in the
// log, sorted by time.
auto cursor = reader.create_read_cursor();

while (auto object = cursor.step(object))
{
    // ... your work here ...
}

It is advisable to use the query interface to filter the log down to the channel you wish to read. This can speedup playback considerably.

As an example:

ark::logging::Query query;

query.minimum_time = std::chrono::steady_clock::time_point{5000ns};
query.maximum_time = std::chrono::steady_clock::time_point{10000ns};
query.message_allow_list = { "/pose" };

auto cursor = reader.create_read_cursor(query);

That will only read the /pose channel, and only between 5 and 10 microseconds.

Convenience methods exist to read only a handful of channels:

auto cursor = reader.create_read_cursor({"/pose"});

Only reads the /pose channel (it automatically creates a query for you).

You can either specify full channel names in the ark::logging::Query, or you can specify a regex:

ark::logging::Query query{.message_allow_list={"/cameras[0-2]/image"}};

In this case, this will return the channels /cameras0/image, /cameras1/image, and /cameras2/image (assuming they are in the log).

Accessing Logs via URLs

You can read logs directly via HTTP, S3, and from the Catalog. If you wish to access logs that are stored in the Catalog, simply provide the GUID. The software will automatically take care of determining where to download the files from.

A few examples:

./build/ark-logtool 4f8f8814-0579-48cb-b826-53efdff7d274 -l
./build/ark-logtool https://my-custom-server.com/manifests/4f8f8814-0579-48cb-b826-53efdff7d274
./build/ark-logtool s3://my-bucket/manifests/4f8f8814-0579-48cb-b826-53efdff7d274
./build/ark-logtool /mnt/logs/manifests/4f8f8814-0579-48cb-b826-53efdff7d274

The ark::logging::LogReader constructor accepts a URL and will dispatch to the appropriate backend as necessary.

If you wish to download logs directly from S3 in your custom tool or pipeline, you do need to link against the ark::s3_log_registry. This is kept as a separate ‘plugin’ as it does bring in the AWS SDK into your binaries, which can inflate their sizes.