Part 12 (Playback Log Data)
In this section, we’ll build a simple pipeline to playback a data log. We’ll set up the pipeline to run two SecondStages, but there will not be a FirstStage. Instead, we will simulate FirstStage output by using the logged data. The log will be the source of MyString data, which will trigger the subscriber callbacks in SecondStage. We call this concept “LogSim”, details here
Let’s start by creating a new pipeline ${APP_ROOT}/tutorial/playback/playback_pipeline.cc:
#include "ark/comms/stages/http_server_stage.hh"
#include "ark/main/main_offboard.hh"
#include "ark/pipeline/config.hh"
#include "ark/pipeline/pipeline.hh"
#include "tutorial/second_stage/second_stage.hh"
#include <iostream>
int main(int argc, const char **argv)
{
try
{
//
// Setup for playback - The LogPlaybackConfiguration will add a LogReaderStage to our pipeline
//
auto config = ark::main::MainOffboardConfiguration::LogPlaybackConfiguration();
//
// Allow for user interaction with this pipeline
//
config.interactive_playback = true;
//
// Construct pipeline
//
ark::pipeline::Pipeline pipeline;
pipeline.add_stage<ark::comms::HttpServerStage>();
//
// Only add in SecondStages
//
pipeline.add_stage<SecondStage>("SecondStage01");
pipeline.add_stage<SecondStage>("SecondStage02");
//
// Execute in realtime, to allow user interaction to pause the pipeline via LogReaderPlugin control
//
return ark::main::execute_realtime_pipeline(argc, argv, std::move(pipeline), config);
}
catch (const std::exception &exception)
{
std::cerr << "FATAL: " << exception.what() << std::endl;
return 1;
}
}
In this implementation, we are setting up the pipeline to use the real time executor, which will execute the data in real-time and allow us to interact with the pipeline.
Now let’s create a ${APP_ROOT}/tutorial/playback/CMakeLists.txt to set up this pipeline. Note that we are not linking in FirstStage at all.
add_executable(playback_pipeline playback_pipeline.cc)
target_link_libraries(
playback_pipeline
PRIVATE second_stage
ark::main_offboard
ark::http_server_stage
)
Then add this to the end of ${APP_ROOT}/CMakeLists.txt:
add_subdirectory(tutorial/playback)
Now let’s invoke ‘make.sh’ to build:
./make.sh
Running Playback
Now that we have compiled our playback pipeline, we can execute it:
./build/playback_pipeline
Note that there is an ERROR reported stating that required arguments are missing:
~/ark$ ./build/playback_pipeline
ERROR: The positional 'log-url' must be specified at least once.
Usage:
./build/playback_pipeline [OPTIONS] <log-url>
Options:
--version Print version information about this binary out
-v, --verbose Increase verbosity level (can be set multiple times)
--dot-output Emit a dot-file illustrating the pipeline layout
Video:
--video PATH Write renderable output to the given video file
--video-backend Backend to use for video generation [cairo, magnum].
Catalog:
--register-catalog URL Registers your artifacts with the catalog at the given URL.
Reads from environment 'CATALOG_API_URL' when available
--artifact-id GUID When registering catalog artifacts, use this identifier.
Reads from environment 'ARK_ARTIFACT_ID' when available
Pipeline:
-c, --pipeline-config Path to the pipeline configuration file to load
--throttle-to-system-time ARG The simclock runner will run no faster than system time scaled by this amount
--collect-runtimes OUTPUT_PATH Add stage to calculate the expected runtimes for this pipeline
Log Playback:
-n, --minimum-time Time to start log playback at (in seconds)
-x, --maximum-time Time to stop log playback at (in seconds)
--force-playback-all-channels Forces playback of all channels, not just the subscribed channels
HTTP Server:
--http-server-address ADDRESS The network address to use for the http server stage.
(Default: 0.0.0.0:8080)
--disable-http-server Prevents the HTTP server from initializing.
Positional:
log-url URL or path to the log manifest for playback
The playback_pipeline requires input to the LogReaderStage to point to the path where the log exists.
Let’s run the binary again and point to one of the logs created. The logs will be located on your local machine at `/tmp/ark_logs’. Set the path to one of the manifest
~/ark$ ./build/playback_pipeline /tmp/ark_logs/manifests/615b0adc-3e18-4d9d-87bf-0dc991cca2c8
(2022-03-23 10:32:52.833) [info] (version.cc:16) Starting execution with ARK_V2.0.0-0-g18b30772/x86_64 (18b3077258b9002f8233a410f5e357f72a14c9fc)
(2022-03-23 10:32:52.834) [info] (http_server_stage.cc:874) HTTP Server is listening on "0.0.0.0:8080".
(2022-03-23 10:32:52.854) [info] (log_reader_stage.cc:1116) Playing back log, at time 0s, 0.0% complete.
(2022-03-23 10:32:52.854) [info] (second_stage.cc:40) SecondStage01 stage received message: 'At 0.504266066, my message is HELLO!.'
(2022-03-23 10:32:52.854) [info] (second_stage.cc:40) SecondStage02 stage received message: 'At 0.504266066, my message is HELLO!.'
(2022-03-23 10:32:53.334) [info] (second_stage.cc:40) SecondStage01 stage received message: 'At 1.004266066, my message is HELLO!.'
(2022-03-23 10:32:53.335) [info] (second_stage.cc:40) SecondStage02 stage received message: 'At 1.004266066, my message is HELLO!.'
(2022-03-23 10:32:53.834) [info] (second_stage.cc:40) SecondStage01 stage received message: 'At 1.504266066, my message is HELLO!.'
(2022-03-23 10:32:53.834) [info] (second_stage.cc:40) SecondStage02 stage received message: 'At 1.504266066, my message is HELLO!.'
(2022-03-23 10:32:54.334) [info] (second_stage.cc:40) SecondStage02 stage received message: 'At 2.004266066, my message is HELLO!.'
(2022-03-23 10:32:54.334) [info] (second_stage.cc:40) SecondStage01 stage received message: 'At 2.004266066, my message is HELLO!.'
(2022-03-23 10:32:54.834) [info] (second_stage.cc:40) SecondStage01 stage received message: 'At 2.504266066, my message is HELLO!.'
(2022-03-23 10:32:54.834) [info] (second_stage.cc:40) SecondStage02 stage received message: 'At 2.504266066, my message is HELLO!.'
(2022-03-23 10:32:55.334) [info] (second_stage.cc:40) SecondStage01 stage received message: 'At 3.004266066, my message is HELLO!.'
(2022-03-23 10:32:55.334) [info] (second_stage.cc:40) SecondStage02 stage received message: 'At 3.004266066, my message is HELLO!.'
(2022-03-23 10:32:55.834) [info] (second_stage.cc:40) SecondStage01 stage received message: 'At 3.504266066, my message is HELLO!.'
(2022-03-23 10:32:55.834) [info] (second_stage.cc:40) SecondStage02 stage received message: 'At 3.504266066, my message is HELLO!.'
(2022-03-23 10:32:56.335) [info] (second_stage.cc:40) SecondStage01 stage received message: 'At 4.004266066, my message is HELLO!.'
(2022-03-23 10:32:56.335) [info] (second_stage.cc:40) SecondStage02 stage received message: 'At 4.004266066, my message is HELLO!.'
(2022-03-23 10:32:56.834) [info] (second_stage.cc:40) SecondStage01 stage received message: 'At 4.504266066, my message is HELLO!.'
(2022-03-23 10:32:56.834) [info] (second_stage.cc:40) SecondStage02 stage received message: 'At 4.504266066, my message is HELLO!.'
(2022-03-23 10:32:57.334) [info] (second_stage.cc:40) SecondStage02 stage received message: 'At 5.004266066, my message is HELLO!.'
(2022-03-23 10:32:57.334) [info] (second_stage.cc:40) SecondStage01 stage received message: 'At 5.004266066, my message is HELLO!.'
(2022-03-23 10:32:57.334) [info] (log_reader_stage.cc:838) Log playback complete.
Note that the two SecondStages execute their callbacks with the logged MyString messages from the log.
Due to setting interactive_playback on the pipeline configuration, the pipeline will run indefinitely, until the user exits the process.
User Interaction
Let’s add the LogReaderPlugin to the GUI so we can interact with the LogReaderStage pipeline.
Edit the ${APP_ROOT}/tutorial/gui/tutorial_gui.cc file to add the LogReaderPlugin:
#include "ark/gui/gui_application.hh"
#include "ark/config/gui/config_package_plugin.hh"
#include "ark/debuglog/gui/debug_logger_plugin.hh"
#include "ark/logging/gui/log_writer_plugin.hh"
#include "ark/logging/gui/log_reader_plugin.hh"
#include "ark/perf/gui/tracer_plugin.hh"
#include "ark/pipeline/gui/pipeline_metrics_plugin.hh"
#include "ark/plotting/gui/json_decoder_plugin.hh"
#include "ark/plotting/gui/plotting_host_plugin.hh"
int main(int argc, const char **argv)
{
ark::gui::GuiApplicationConfig app_config;
app_config.name = "Favorite GUI";
ark::gui::GuiApplication application(std::move(app_config), argc, argv);
// Central
application.register_central_plugin<ark::config::ConfigPackagePlugin>();
application.register_central_plugin<ark::perf::TracerPlugin>();
application.register_central_plugin<ark::pipeline::PipelineMetricsPlugin>();
application.register_central_plugin<ark::plotting::PlottingHostPlugin>();
// debug messagse
application.register_bottom_plugin<ark::debuglog::DebugLoggerPlugin>();
// Log Writer
application.register_right_plugin<ark::logging::LogWriterPlugin>();
application.register_right_plugin<ark::plotting::JsonDecoderPlugin>();
// Log Reader
application.register_right_plugin<ark::logging::LogReaderPlugin>();
return application.execute();
}
Next, update the ${APP_ROOT}/tutorial/gui/CMakeLists.txt file to link in the plugin:
qt_add_resources(resources_source ${QDARKSTYLE_RESOURCE})
add_library(tutorial_resources ${resources_source})
add_executable(tutorial_gui tutorial_gui.cc)
target_link_libraries(
tutorial_gui
PRIVATE ark::gui
ark::config_package_plugin
ark::debug_logger_plugin
ark::tracer_plugin
ark::log_writer_plugin
ark::log_reader_plugin
ark::pipeline_metrics_plugin
ark::plotting_host_plugin
ark::json_decoder_plugin
-Wl,--whole-archive
tutorial_resources
-Wl,--no-whole-archive)
Let’s build the gui and run it to see the LogPlayer plugin
./make.sh
Running playback and gui
Execute both the gui and the playback_pipeline to interact with the LogReader control.
In one terminal, start the gui:
./build/tutorial_gui
In another terminal, start the playback pipeline:
~/ark$ ./build/playback_pipeline /tmp/ark_logs/manifests/615b0adc-3e18-4d9d-87bf-0dc991cca2c8
In the gui window, click on the the Log Player tab along the right pane. Your window should look something like this.
Note that you can see the current playback time, information about the log (such as name, guid, etc). You can also try pressing
Pause and Play to start/stop playback.
This is an example of how to simulate stage executing in a pipeline using logged data.
SimClock playback
Let’s modify our pipeline to run using the SimClock. More details on logsim
Let’s start by creating a new pipeline ${APP_ROOT}/tutorial/playback/playback_pipeline.cc:
#include "ark/comms/stages/http_server_stage.hh"
#include "ark/main/main_offboard.hh"
#include "ark/pipeline/config.hh"
#include "ark/pipeline/pipeline.hh"
#include "tutorial/second_stage/second_stage.hh"
#include <iostream>
int main(int argc, const char **argv)
{
try
{
//
// Setup for playback - The LogPlaybackConfiguration will add a LogReaderStage to our pipeline
//
auto config = ark::main::MainOffboardConfiguration::LogPlaybackConfiguration();
//
// Disable interaction
//
config.interactive_playback = false;
ark::pipeline::Pipeline pipeline;
pipeline.add_stage<ark::comms::HttpServerStage>();
pipeline.add_stage<SecondStage>("SecondStage01");
pipeline.add_stage<SecondStage>("SecondStage02");
//
// Execute via simclock.
//
return ark::main::execute_simclock_pipeline(argc, argv, std::move(pipeline), config);
}
catch (const std::exception &exception)
{
std::cerr << "FATAL: " << exception.what() << std::endl;
return 1;
}
}
Recompile playback_pipeline and execute it to see the difference in behavior.
$ ./build/playback_pipeline /tmp/ark_logs/manifests/615b0adc-3e18-4d9d-87bf-0dc991cca2c8
(2022-03-23 11:01:40.437) [info] (version.cc:16) Starting execution with ARK_V2.0.0-0-g18b30772/x86_64 (18b3077258b9002f8233a410f5e357f72a14c9fc)
(2022-03-23 11:01:40.438) [info] (simclock_executor.cc:370) The SimClockExecutor is initializing the start time to 0.504644301s.
(2022-03-23 11:01:40.438) [info] (log_reader_stage.cc:1116) Playing back log, at time 0s, 0.0% complete.
(2022-03-23 11:01:40.438) [info] (second_stage.cc:40) SecondStage01 stage received message: 'At 0.504266066, my message is HELLO!.'
(2022-03-23 11:01:40.438) [info] (second_stage.cc:40) SecondStage02 stage received message: 'At 0.504266066, my message is HELLO!.'
(2022-03-23 11:01:40.438) [info] (http_server_stage.cc:874) HTTP Server is listening on "0.0.0.0:8080".
(2022-03-23 11:01:40.438) [info] (second_stage.cc:40) SecondStage01 stage received message: 'At 1.004266066, my message is HELLO!.'
(2022-03-23 11:01:40.438) [info] (second_stage.cc:40) SecondStage02 stage received message: 'At 1.004266066, my message is HELLO!.'
(2022-03-23 11:01:40.439) [info] (second_stage.cc:40) SecondStage01 stage received message: 'At 1.504266066, my message is HELLO!.'
(2022-03-23 11:01:40.439) [info] (second_stage.cc:40) SecondStage02 stage received message: 'At 1.504266066, my message is HELLO!.'
(2022-03-23 11:01:40.439) [info] (second_stage.cc:40) SecondStage02 stage received message: 'At 2.004266066, my message is HELLO!.'
(2022-03-23 11:01:40.439) [info] (second_stage.cc:40) SecondStage01 stage received message: 'At 2.004266066, my message is HELLO!.'
(2022-03-23 11:01:40.440) [info] (second_stage.cc:40) SecondStage01 stage received message: 'At 2.504266066, my message is HELLO!.'
(2022-03-23 11:01:40.440) [info] (second_stage.cc:40) SecondStage02 stage received message: 'At 2.504266066, my message is HELLO!.'
(2022-03-23 11:01:40.440) [info] (second_stage.cc:40) SecondStage01 stage received message: 'At 3.004266066, my message is HELLO!.'
(2022-03-23 11:01:40.440) [info] (second_stage.cc:40) SecondStage02 stage received message: 'At 3.004266066, my message is HELLO!.'
(2022-03-23 11:01:40.441) [info] (second_stage.cc:40) SecondStage01 stage received message: 'At 3.504266066, my message is HELLO!.'
(2022-03-23 11:01:40.441) [info] (second_stage.cc:40) SecondStage02 stage received message: 'At 3.504266066, my message is HELLO!.'
(2022-03-23 11:01:40.442) [info] (second_stage.cc:40) SecondStage02 stage received message: 'At 4.004266066, my message is HELLO!.'
(2022-03-23 11:01:40.442) [info] (second_stage.cc:40) SecondStage01 stage received message: 'At 4.004266066, my message is HELLO!.'
(2022-03-23 11:01:40.442) [info] (second_stage.cc:40) SecondStage01 stage received message: 'At 4.504266066, my message is HELLO!.'
(2022-03-23 11:01:40.442) [info] (second_stage.cc:40) SecondStage02 stage received message: 'At 4.504266066, my message is HELLO!.'
(2022-03-23 11:01:40.442) [info] (log_reader_stage.cc:838) Log playback complete.
(2022-03-23 11:01:40.442) [info] (context.cc:850) 'LogReaderStage' has requested a shutdown: Success.
(2022-03-23 11:01:40.442) [info] (context.cc:884) All tokens have requested a shutdown; the pipeline will exit.
(2022-03-23 11:01:40.442) [info] (second_stage.cc:40) SecondStage01 stage received message: 'At 5.004266066, my message is HELLO!.'
(2022-03-23 11:01:40.442) [info] (second_stage.cc:40) SecondStage02 stage received message: 'At 5.004266066, my message is HELLO!.'
(2022-03-23 11:01:40.442) [info] (simclock_executor.cc:503) Simulation complete (simulated 4.50s in 0.00s for 512 executions, 952.83x sim:wall time).
The pipeline completes and exits on its own when the log data has been fully read.
Note that the SecondStages still execute their subscriber callbacks as they did with the realtime executor, but the time in which they executor is much faster. The execution is run 952x faster than wall time, in this case.
LogSim is a benefical concept to test system behavior using recorded real-world data.
This concludes our tutorial. Please see the other bits of documentation to learn more about using individual Ark features!