ark::logging

Enums

• OffloadLogFileStatus (int32_t)
  An enumeration indicating how the offload proceeded for a single file.

Typedefs

Defined in “ark/logging/clock_definition.hh”:

• using LoggingClock = std::chrono::steady_clock
  A helper for the clock we use to log all of our data.

Defined in “ark/logging/column_reader.hh”:

• using ColumnReaderPtr = std::shared_ptr< ColumnReader >

Defined in “ark/logging/column_writer.hh”:

• using ColumnWriterPtr = std::shared_ptr< ColumnWriter >

Defined in “ark/logging/forward.hh”:

• using LogRegistryPtr = std::shared_ptr< LogRegistry >

• using LogFileHandlePtr = std::shared_ptr< LogFileHandle >

• using ObjectDeclarationPtr = std::shared_ptr< ObjectDeclaration >
  Object declarations can be a bit large, this saves on copying times.

• using ObjectDeclarationCPtr = std::shared_ptr< const ObjectDeclaration >
  The same as the above, but pointing to constant data.

• using ColumnWriterUPtr = std::unique_ptr< ColumnWriter >

• using SplitWriterPtr = std::shared_ptr< SplitWriter >

• using SplitWriterUPtr = std::unique_ptr< SplitWriter >

Defined in “ark/logging/log_reader.hh”:

• using LogReaderPtr = std::shared_ptr< LogReader >

Defined in “ark/logging/log_writer.hh”:

• using LogWriterPtr = std::shared_ptr< LogWriter >

Defined in “ark/logging/split_reader.hh”:

• using SplitReaderPtr = std::shared_ptr< SplitReader >

Defined in “ark/logging/stages/log_writer_message_queue.hh”:

• using QueuedObjectPtr = std::shared_ptr< QueuedObject >

• using LogWriterEventBufferPtr = std::shared_ptr< LogWriterEventBuffer >

Classes

• ark::logging::ChannelRegistry
  Channel registry to keep track of ids and sequence numbers.

• ark::logging::ColumnReader
  A column reader is spawned from the read cursor, and is intended to sequentially access logged data from a set of split files, based on the passed in configuration and query.

• ark::logging::ColumnWriter
  A column writer is in charge of writing out data that belongs to a particular column type, tracking the individual splits and any metadata related to this column.

• ark::logging::DistillLogWriterStatistics
  Class to distill the LogWriterStageStatistics down to a report set of data.

• ark::logging::HttpLogFileHandle
  A handle to a log file this can be used to open (and hold open) a reference to a file object. You can then use this to get mappings so you can read data from the object.

• ark::logging::HttpLogRegistry
  A HTTP-based log registry. Reads files from HTTP directly, assuming that the key name is appended to some base URL.

• ark::logging::LocalLogFileHandle
  A handle to a log file this can be used to open (and hold open) a reference to a file object. You can then use this to get mappings so you can read data from the object.

• ark::logging::LocalLogRegistry
  A file-based log registry. Expects logs to be on-disk. Logs are simply directly read.

• ark::logging::LogAmenderStage
  This stage will look at the various publishers in the system and construct a log amendment against the configured log. It assumes that a log is being played back from the log reader that log will be amended.

• ark::logging::LogCleaner
  The logging cleanup engine allows you to analyze a directory tree for manifests and split files; it provides an interface to safely remove old manifests, and erase any split files that depended on them.

  This class is intended to be used from either an external tool or as live stage onboard.

• ark::logging::LogCleanerStage
  This stage will periodically run a routine that analyzes the amount of disk space consumed by logs and execute a “free” routine to clean up the files. It can be used to ensure there is always disk space free on your robot to take logs.

• ark::logging::LogFileHandle
  A handle to a log file this can be used to open (and hold open) a reference to a file object. You can then use this to get mappings so you can read data from the object.

• ark::logging::LogReader
  A log reader allows you to open a previously saved log from disk and access all of the saved data. Further, you can apply “queries” to this data, allowing you to filter the results down to what you want.

• ark::logging::LogReaderPlugin
  A simple plugin that can display log reader statistics, and allow the user to play/pause/seek.

• ark::logging::LogReaderStage
  This stage will open the selected log, construct publishers for every channel in that log, and then play data back approximating real time.

• ark::logging::LogRegistry
  A log registry provides an abstraction over data storage for the log reader, allowing the log reader to access data from multiple sources transparently.

• ark::logging::LogWriter
  A log writer allows you to write a log out to disk. A log consists of indexed columns of data, organized to be read efficiently. You can store any number of message types within a log.

• ark::logging::LogWriterMessageQueue
  This class provides access to a message queue, allowing you to place incoming objects into the queue and taking care of managing them. This further allows us to manage things like “event based logging”, where we buffer some objects into standalone queues that get flushed on demand.

• ark::logging::LogWriterPlugin
  A simple plugin that can display log writer statistics, and command the logger to be active or not.

• ark::logging::LogWriterStage
  This stage will look at all of the publishers in the current system and connect them up to a log writer. All data that is published is redirected to a log.

• ark::logging::OffloadPlugin
  A plugin to interface with the offload system.

• ark::logging::OffloadStage
  This stage offloads logs.

• ark::logging::OffloadTableModel
  This classes visualizes offload information as a table model.

• ark::logging::PresignedUrlLogFileHandle
  A handle to a log file this can be used to open (and hold open) a reference to a file object. You can then use this to get mappings so you can read data from the object.

• ark::logging::PresignedUrlLogRegistry
  A HTTP-based log registry. Reads files from HTTP directly, using the pre-supplied URLs and key mappings.

• ark::logging::ReadCursor
  A read cursor is spawned from log readers, and provides a mechanism to read over the data stored within a log.

• ark::logging::S3LogFileHandle
  A handle to a log file this can be used to open (and hold open) a reference to a file object. You can then use this to get mappings so you can read data from the object.

• ark::logging::S3LogRegistry
  A S3-based log registry. Reads files from S3 directly, using AWS APIs.

• ark::logging::SplitReader
  A split reader is used by the column readers to read data out from split files in an organized fashion.

• ark::logging::SplitWriter
  A split writer is used by the column writer to write data out to files in an organized fashion.

• ark::logging::ColumnConfiguration
  This represents a column configuration. Each column of data is written out sequentially. This allows you to help configure how your data is written out. All logs with data have at least one column.

• ark::logging::ColumnReaderConfiguration
  This represents the configuration for reading a particular column. Specifically, it takes in a column specification from a manifest, and a source path, and uses that to open up and read split files.

• ark::logging::ColumnWriterConfiguration
  This represents configuration for the column writer itself. It is a superset of the configuration for a column, and stores a bit more information. It is intended primarily to be used internally.

• ark::logging::ColumnWriterStatistics
  Used to return statistics related to this column writer.

• ark::logging::CreateTestLogConfig

• ark::logging::FilteredColumnSplitDeclarations
  A pairing of data returned containing split declarations in a column filtered into “latched” and “core” splits. See below for more details.

• ark::logging::HttpLogRegistryConfiguration
  Configuration for the log registry, if loading logs from the AWS Http.

• ark::logging::LocalLogRegistryConfiguration
  Configuration for the log registry, if loading logs from the local disk.

• ark::logging::LogCleanerFilter
  The filter for cleaning logs. This allows you to setup the rules by which files are removed.

• ark::logging::LogReaderConfiguration
  Configuration for the log reader. This allows you to specify the location of the log to open.

• ark::logging::LogWriterConfiguration
  The configuration for a log writer. This allows you to specify things like identifiers, names, and other metadata/configuration early.

• ark::logging::LogWriterEventBuffer
  A event buffer this stores some number of messages, only placing them (or allowing them) on the primary queue if the buffer is “enabled”.

• ark::logging::LogWriterMessageQueueConfig
  Configuration for the message queue.

• ark::logging::LogWriterMessageQueueStatistics
  Statistics related to this message queue.

• ark::logging::LoggedObject
  A simple object that is populated from the read cursor as you step through a log. It contains the raw data contained within the log.

  Note that a logged object becomes invalidated each time you step through a cursor.

• ark::logging::ManifestCleanupMetadata
  A simple structure representing manifest metadata that can be used to decide which manifests to remove.

• ark::logging::ObjectDeclarationMappingData
  Internal configuration for each object type. Stores mapping information for writing out object types to disk.

• ark::logging::OffloadLogTargetUrlPair
  A pair of URLs used for offloading a source file (contains both a PUT URL for uploading to, and a HEAD URL for querying).

• ark::logging::OffloadLogTargetUrls
  This structure contains the list of URLs that can be used to offload a log to some web server.

• ark::logging::PresignedUrlLogRegistryConfiguration
  Configuration for the log registry, if loading logs from the remote HTTP server.

• ark::logging::QueuedObject
  A queued object, an object that contains both an envelope to be written, and the declaration (source) of that object.

• ark::logging::ReadCursorConfiguration
  Configuration for the read cursor. Provides location information to read from, as well as the full manifest to read.

• ark::logging::RequiredLogUrlList
  A list of files that would be necessary to fully read a given log. This is typically returned from a read cursor.

• ark::logging::S3LogRegistryConfiguration
  Configuration for the log registry, if loading logs from the AWS S3.

• ark::logging::SplitIndexEntry
  Represents a packed structure that is used for indexing. These are written out to the split file as-is.

• ark::logging::SplitReaderConfiguration
  Configuration for a split reader. This reads the split from the given config, typically provided by the manifest. This is normally initialized for you through the column reader.

• ark::logging::SplitWriterConfiguration
  Configuration for a split writer. This is typically populated by a column writer, as it’s about to instantiate a new split.

• ark::logging::SplitWriterFileConfiguration
  Configuration for the file layout for a particular split writer.

• ark::logging::SplitWriterStatistics
  Used to return statistics related to this split writer.

Functions

Declared in “ark/logging/active_log_file.hh”:

• void set_log_actively_used(const core::Guid & manifest_identifier, bool used)
  This helps indicate if a log file is “active”. For example, a log writer can call this function to indicate it is busy using this log, and downstream code (cleaning, etc), even in another process, can use this to confirm that a log is “active” or “done”.

• bool is_log_actively_used(const core::Guid & manifest_identifier)
  Returns true if the given manifest is actively used, or false otherwise.

Declared in “ark/logging/distill_log.hh”:

• DistilledLoggedObjectStatsList distill_logged_objects(const Manifest & manifest)
  Distill information from all of the logged objects.

• distill::DistilledReport distill(const Manifest & manifest)
  Creates a DistilledReport on the logged info.

Declared in “ark/logging/filter.hh”:

• bool is_timestamp_filtered(const Query & query, const LoggingClock::time_point & timestamp)
  Returns true if the query is filtering out the specified timestamp, or false otherwise.

• bool is_time_range_filtered(const Query & query, const LoggingClock::time_point & start_time, const LoggingClock::time_point & end_time)
  Returns true if the given time range is completely outside of the query’s range (ie, is filtered out because no data within the range provided overlaps with the query’s range).

• std::set< core::Guid > resolve_object_identifiers_from_query(const Query & query, const ColumnDeclaration & column)
  Resolves all of the object identifiers that are in the specified query, against the given type information. The final result of this set will be filtered by the query’s allow list.

• std::set< core::Guid > resolve_latched_object_identifiers(const ColumnDeclaration & column)
  Resolves all of the object identifiers in the given column that have latched flags set.

• std::pair< LoggingClock::time_point, LoggingClock::time_point > restrict_time_by_query(const Query & query, const LoggingClock::time_point & minimum_time, const LoggingClock::time_point & maximum_time)
  Restricts the given minimum/maximum time by the given query. This will apply the intersection of the two time ranges.

• std::pair< LoggingClock::time_point, LoggingClock::time_point > restrict_time_by_query(const Query & lhs, const Query & rhs)
  Restricts the given minimum/maximum time by the given query. This will apply the intersection of the two time ranges.

• FilteredColumnSplitDeclarations filter_column_split_declarations(const Query & query, const ColumnDeclaration & column)
  Filters the split declarations in the given column based on the specified query. Filters them into two bins; “core” splits, which are splits that are necessary to read the data in the query, and “latched” splits, which are splits that are only necessary for reading latched messages.

• Manifest apply_query_to_manifest(const Query & query, const Manifest & manifest)
  Takes the given manifest and applies the specified query to it. This will remove any columns/splits that are completely out of bounds thanks to the given query.

• Query apply_query_to_query(const Query & source, const Query & target)
  This will apply the given query ‘source’ to the given query ’target’. The ’target’ query will be restricted based on the values in ‘source’ (in other words, time ranges will be the minimum of the two, and message lists will be the intersection between the two).

Declared in “ark/logging/helpers.hh”:

• std::vector< ObjectDeclaration > collect_object_declarations_from_manifest(const Manifest & manifest)
  Retrieves all of the object declarations stored in the given manifest, across all of the columns. Internal filters are obeyed; things like amendments will have the amended declarations returned, not the originals.

• std::pair< LoggingClock::time_point, LoggingClock::time_point > get_manifest_time_range(const Manifest & manifest)
  Returns the log starting and ending time, as a pair of timestamps.

• uint64_t get_manifest_referenced_data_size(const Manifest & manifest)
  Returns the amount of data (size, in bytes) that is referenced by the given manifest.

• uint64_t get_log_ondisk_size(const std::filesystem::path & manifest_path)
  Returns the amount of data consumed by the given log.

• std::map< logging::ObjectDeclaration, logging::ObjectStatistics > get_summarized_object_statistics(const Manifest & manifest)
  Returns overall statistics about a given log, summarizing the individual object statistics and returning a single map that goes from object identifier to statistics.

• std::optional< std::chrono::system_clock::time_point > pipeline_to_global_time_from_manifest(const Manifest & manifest, const LoggingClock::time_point & at_time)
  Translates the given pipeline time (such as indexed time) into a global time point through the use of the global information in the time model. Returns an empty optional if the model is not available.

• std::optional< std::chrono::system_clock::time_point > get_manifest_global_start_time(const Manifest & manifest)
  Returns the global time that a log was captured at, if a time model was logged. Otherwise, returns an empty optional.

• std::optional< std::chrono::system_clock::time_point > get_manifest_global_stop_time(const Manifest & manifest)
  Returns the global time that a log was finished at, if a time model was logged. Otherwise, returns an empty optional.

• core::Guid get_original_log_identifier_from_split_identifier(const Manifest & manifest, const core::Guid & split_identifier)
  Returns the original log identifier the given split file identifier first appeared in. For example, in an amendment, this will return the log identifier of the base log for splits generated originally, or the log identifier of the amendment for splits generated during amending.

• std::vector< core::Guid > get_manifest_referenced_split_file_identifiers(const Manifest & manifest)
  Returns all split file identifiers referenced by this manifest.

• serialization::SchemaRegistry get_manifest_schema_registry(const Manifest & manifest)
  Returns a serialization registry that covers every type in the manifest.

• std::optional< core::Guid > get_manifest_organization_identifier(const Manifest & manifest)
  Extracts the organization metadata from the given manifest. Returns an empty optional in the event that the metadata is not present (or is malformed in some way).

• std::string get_compression_level_for_channel_from_manifest(const Manifest & manifest, const std::string & channel_name)
  Returns the compression level for the given channel name.

Declared in “ark/logging/log_reader.hh”:

• LogReaderConfiguration create_reader_config_from_path(const std::filesystem::path & path)
  Creates a log reader configuration from the given path. Assumes the path points directly to a manifest file, and that the directory layout follows normal conventions (manifest is in a manifest directory, splits are in a splits directory).

• LogReaderConfiguration create_reader_config_from_guid(const core::Guid & guid, const core::Url & catalog_url)
  Creates a log reader configuration from the given GUID. The catalog will be queried for presigned URLs necessary to access this GUID, and an appropriate log registry will be constructed to read data from it.

  If the catalog URL is left empty, the system environment variable ‘CATALOG_API_URL’ will be used to determine where to access to get further log information. If that is also empty, a well-known default will be used (https://catalog.tbdrobotics.com/).

• LogReaderConfiguration create_reader_config_from_url(const core::Url & url)
  Creates a log reader configuration from the given URL, creating an appropriate log registry based on the URL scheme. It assumes the URL points directoy to a manifest file, and the directory layout follows normal conventions.

Declared in “ark/logging/log_writer.hh”:

• LogWriterConfiguration initialize_new_common_log(std::string name)
  Initializes a new log writer configuration, suitable for a new set of independent log data. This assumes that this log will represent new data that is being collected, and is not a snippet or amendment.

• LogWriterConfiguration initialize_amended_log(const Manifest & original_manifest, const Query & restrictions)
  Initializes a new log writer configuration, suitable for amending an existing log. This takes in both the manifest of the original log, and the query that you wish to apply to the existing data (for example, to restrict time range or objects).

Declared in “ark/logging/mcap/convert.hh”:

• std::vector< mcap::Metadata > to_mcap_metadata(const Manifest & manifest)
  Construct the mcap metadata from the ark logging manifest.

• mcap::Message to_mcap_message(const LoggedObject & obj, const mcap::ChannelId & id)
  Create the mcap Message for the ark logged object.

• void convert(mcap::McapWriter & writer, const Manifest & manifest, ReadCursor & cursor)
  Write the ark log contents as mcap.

• void convert(mcap::McapWriter & writer, LogReader & reader)
  Convience function to convert the entire log.

Declared in “ark/logging/object_declaration.hh”:

• ObjectDeclaration make_object_declaration(std::string name)
  Creates an object type suitable for use within the logger. This will generate a new unique identifier, and extract an appropriate type identifier.

• ObjectDeclaration make_latched_object_declaration(std::string name)
  Returns a latched object declaration. Latched objects will be replayed at the beginning of any query; they allow you to do things like write configuration out once, but come at a space cost.

Declared in “ark/logging/offload.hh”:

• catalog::PresignUploadLogRequest to_presigned_upload_request(const logging::Manifest & manifest)
  Takes the given manifest and transforms it into a presigned-upload URL request for the catalog.

• OffloadLogTargetUrls get_offload_log_target_urls(const catalog::PresignUploadLogResponse & upload_response)
  Takes the given presigned upload log response and produces a list of target URLs that can be used to offload a log.

• OffloadLogTargetUrls get_offload_log_target_urls(const logging::Manifest & manifest, const core::Url & base_url)
  Takes the given manifest and produces a set of URLs that can be used to offload the manifest to the given HTTPS server.

• bool is_log_file_already_offloaded(const OffloadLogTargetUrlPair & url_pair)
  Returns true if the given log file exists on a remote server, or false otherwise.

• OffloadLogFileStatus offload_log_file(const std::filesystem::path & source_path, const OffloadLogTargetUrlPair & url_pair, comms::HttpProgressCallback progress_callback)
  Offloads the given log file (a manifest or split file) to the destination URL. This will only offload if the file is not already there or not the same size.

  If the file already exists, the progress callback will be invoked with the file size. A status will be returned to indicate if a file actually was offloaded.

Declared in “ark/logging/paths.hh”:

• std::filesystem::path resolve_variables_in_logging_path(const std::string & path, const core::Guid & log_identifier, const std::chrono::system_clock::time_point & now)
  A logging path may contain several variables, such as identifiers or dates that are not known at configuration time. This will resolve those, for example: /tmp/logs/${YEAR}-${MONTH}/${YEAR}-${MONTH}-${DAY_OF_MONTH}/${LOG_IDENTIFIER}.

  Becomes: /tmp/logs/2021-11/2021-11-18/89120123-6781-3456-123491235721 Variables:

• bool is_likely_manifest(const std::filesystem::path & base_path, const std::filesystem::path & manifest_path)
  Returns true if the given path is likely to be a manifest file.

Declared in “ark/logging/repair.hh”:

• LogRepairState repair_log(const core::Url & log_url, Manifest & manifest, const std::function< void(const std::string &)> & error_callback)
  Makes a best effort to repair the given log. Walks through the data in the log and tries to cut away any regions that appear to be corrupted. Modifies the manifest in such a way that you can read data again, and returns the state of the repair for debugging.

Declared in “ark/logging/test/utility.hh”:

• LogReaderConfiguration get_reader_config_from_writer(const LogWriterConfiguration & writer_config, const Manifest & manifest)
  Takes the given log writer configuration and manifest, and returns a log reader configuration that can be used to access the log.

Declared in “ark/logging/testing/testlog_helper.hh”:

• Manifest create_test_log(CreateTestLogConfig & config)
  Helper to create a test log.

Declared in “ark/logging/validation.hh”:

• void validate_query(const Query & query)
  Throws if the given query is invalid (ie, contains conflicting information or information that is obviously bad).

• void validate_manifest(const Manifest & manifest)
  Throws if the given manifest is invalid (ie, contains conflicting information or information is obviously bad). It is probably not safe to read an invalid manifest.

Declared in “ark/logging/wire_structure.hh”:

• struct ark::logging::SplitIndexEntry __attribute__()
  The header of a Swift message.

  A system-heartbeat message, sent to indicate the overall health of the system. The offset to translate from local time to GPS time for messages tagged as such. The baseline between 2 Swift devices. This contains the relative vector distance from the base station to the rover receiver with the NED coordinate system definde by the local WGS84 tangent plane centerd at the base station. Absolute geodetic coordinates and status (best available) with covariances See fix mode for the values. The orientation of the device, in vehicle body frame. The rotations should be applied intrinsically in the order yaw, pitch, and roll in order to rotate the from a frame aligned with the local-level NED frame to the vehicle body frame. The velocity in vehicle body frame. The x-axis points at the nose of the vehicle, y-axis points to the right-hand side, and z-axis points downward. The velocity in local NED (north/east/down) coordinate. This frame is defined as the local WGS84 tangent plane, centered on the current location. The velocity in ECEF coordinate frame. A log message (ASCII text) coming from the device. Absolute geodetic coordinates and status (best available). See fix mode for the values. This structure contains configuration information about the IMU. It is necessary to fully decode SwiftMsgImuRaw packets. Raw data from the IMU, containing accelerometer and gyroscope readings. The sense of measurements are aligned with the indications on the device itself. Reports UTC time. The flags indicate the source of the offset value and fix. A decoded GPS time message, from the Swift. Represents the time since the GPS epoch began; the week begins at the Saturday/Sunday transition. Measurement units come from the MSG_IMU_AUX.