Distill Stage

This stage is used to distill object data into a summarized format. The goal of these stages are to create summaries about the data, which can be consolidated into a report. An example of a DistillStage is for reporting on HardwareHealth messages, which should generate a report with information on the overall statistics for System Memory usage observed in all of the messages.

These types of stages are used primarily in the distill_pipeline utility, which will generate a summarized report of log data to associate with the log in Catalog.

Configuration

Use the DistillMsgStageConfig to configure each stage:

• title : for the report section
• subscribe_channel : configures which comms channel to subscribe to for the input data type

Interacting

Each distill stage reports on summarized information for a given recorded type in the log.

DistillStage Classes

The DistillStage class itself a base class, which holds a DistilledReport that contains summarized data in json format. This stage has a public report() function, which can be used to generate the summarized report from all the of the data.

There are two derived classes for common data distilling usecases: DistillMsgStage<Type,DistilledType> and SummaryMsgStage<Type>.

 classDiagram
      DistillStage <|-- DistillMsgStage
      DistillMsgStage <|-- SummaryMsgStage
      class DistillStage  
      {
          +report() DistilledReport
      }
      class DistillMsgStage~T~ {
          #distill(const Type &msg) void
          #summarize() DistilledType
      }
      class SummaryMsgStage{
      }

DistillMsgStage<MsgType, DistilledType>

This type of distilled stage subscribes to the MsgType and should generate a DistillMsgType, which summarizes the data seen that is useful for reporting. Here’s an example

SystemStatistics
{
    double mem_used;
}

SystemStatisticsDistilled
{
    double max_memory_used;
}

The distill and summarize functions must be overriden in order to distill information from the message and to generate a summary of the data seen.

Continuing with the example:

class DistillMsgStage<SystemStatistics, SystemStatisticsDistilled>
{
    void distill(const SystemStatistics& msg)
    {
        max_memory_ = std::max(msg.mem_used, max_memory_);
    }

    SystemStatisticsDistilled summarize()
    {
        SystemStatisticsDistilled summary;
        summary.max_memory_used = max_memory_;
        return summary;
    }

private:
    double max_memory_;

}

SummaryMsgStage

In other cases, it’s useful to collect the information regarding the timing of a given message. The MsgType is assumed to be timestmaped and the report will , received and the spacing of the messages. Using a SummaryMsgStage class will generate a summarized view of the message timing data. This is useful for characterizing sensor data being received. To use this stage, there must be a way to extract the timestamp from the MsgType as a steady_clock timestamp.

std::optional<std::chrono::steady_clock::time_point> extract_sensor_time(const SensorData &msg)
{
    return msg.capture_time;
}

To create a SummaryMsgStage, the function to extract the timestamp must be provided to the class.

distill::SummaryMsgStage<SensorData>>(
                    config, 
                    std::bind(&video::extract_sensor_time, std::placeholders::_1));

The resulting report for SummaryMsgStage will yield the timing information for the data type.

Metrics

The summarized report can be generated by running report() on each of the DistillStages.