Distill Stage

This stage is used to distill object data into a summarized format. The goal of these stages are to create summaries about the data, which can be consolidated into a report. An example of a DistillStage is for reporting on HardwareHealth messages, which should generate a report with information on the overall statistics for System Memory usage observed in all of the messages.

These types of stages are used primarily in the distill_pipeline utility, which will generate a summarized report of log data to associate with the log in Catalog.


Use the DistillMsgStageConfig to configure each stage:

  • title : for the report section
  • subscribe_channel : configures which comms channel to subscribe to for the input data type


Each distill stage reports on summarized information for a given recorded type in the log.

DistillStage Classes

The DistillStage class itself a base class, which holds a DistilledReport that contains summarized data in json format. This stage has a public report() function, which can be used to generate the summarized report from all the of the data.

There are two derived classes for common data distilling usecases: DistillMsgStage<Type,DistilledType> and SummaryMsgStage<Type>.

      DistillStage <|-- DistillMsgStage
      DistillMsgStage <|-- SummaryMsgStage
      class DistillStage  
          +report() DistilledReport
      class DistillMsgStage~T~ {
          #distill(const Type &msg) void
          #summarize() DistilledType
      class SummaryMsgStage{

DistillMsgStage<MsgType, DistilledType>

This type of distilled stage subscribes to the MsgType and should generate a DistillMsgType, which summarizes the data seen that is useful for reporting. Here’s an example

    double mem_used;

    double max_memory_used;

The distill and summarize functions must be overriden in order to distill information from the message and to generate a summary of the data seen.

Continuing with the example:

class DistillMsgStage<SystemStatistics, SystemStatisticsDistilled>
    void distill(const SystemStatistics& msg)
        max_memory_ = std::max(msg.mem_used, max_memory_);

    SystemStatisticsDistilled summarize()
        SystemStatisticsDistilled summary;
        summary.max_memory_used = max_memory_;
        return summary;

    double max_memory_;



In other cases, it’s useful to collect the information regarding the timing of a given message. The MsgType is assumed to be timestmaped and the report will , received and the spacing of the messages. Using a SummaryMsgStage class will generate a summarized view of the message timing data. This is useful for characterizing sensor data being received. To use this stage, there must be a way to extract the timestamp from the MsgType as a steady_clock timestamp.

std::optional<std::chrono::steady_clock::time_point> extract_sensor_time(const SensorData &msg)
    return msg.capture_time;

To create a SummaryMsgStage, the function to extract the timestamp must be provided to the class.

                    std::bind(&video::extract_sensor_time, std::placeholders::_1));

The resulting report for SummaryMsgStage will yield the timing information for the data type.


The summarized report can be generated by running report() on each of the DistillStages.