• OffloadLogFileStatus (int32_t)
    An enumeration indicating how the offload proceeded for a single file.


Defined in “ark/logging/clock_definition.hh”:

  • using LoggingClock = std::chrono::steady_clock
    A helper for the clock we use to log all of our data.

Defined in “ark/logging/column_reader.hh”:

Defined in “ark/logging/column_writer.hh”:

Defined in “ark/logging/forward.hh”:

  • using LogRegistryPtr = std::shared_ptr< LogRegistry >

  • using LogFileHandlePtr = std::shared_ptr< LogFileHandle >

  • using ObjectDeclarationPtr = std::shared_ptr< ObjectDeclaration >
    Object declarations can be a bit large, this saves on copying times.

  • using ObjectDeclarationCPtr = std::shared_ptr< const ObjectDeclaration >
    The same as the above, but pointing to constant data.

  • using ColumnWriterUPtr = std::unique_ptr< ColumnWriter >

  • using SplitWriterPtr = std::shared_ptr< SplitWriter >

  • using SplitWriterUPtr = std::unique_ptr< SplitWriter >

Defined in “ark/logging/log_reader.hh”:

  • using LogReaderPtr = std::shared_ptr< LogReader >

Defined in “ark/logging/log_writer.hh”:

  • using LogWriterPtr = std::shared_ptr< LogWriter >

Defined in “ark/logging/split_reader.hh”:

  • using SplitReaderPtr = std::shared_ptr< SplitReader >

Defined in “ark/logging/stages/log_writer_message_queue.hh”:



Declared in “ark/logging/active_log_file.hh”:

  • void set_log_actively_used(const core::Guid & manifest_identifier, bool used)
    This helps indicate if a log file is “active”. For example, a log writer can call this function to indicate it is busy using this log, and downstream code (cleaning, etc), even in another process, can use this to confirm that a log is “active” or “done”.

  • bool is_log_actively_used(const core::Guid & manifest_identifier)
    Returns true if the given manifest is actively used, or false otherwise.

Declared in “ark/logging/distill_log.hh”:

  • DistilledLoggedObjectStatsList distill_logged_objects(const Manifest & manifest)
    Distill information from all of the logged objects.

  • distill::DistilledReport distill(const Manifest & manifest)
    Creates a DistilledReport on the logged info.

Declared in “ark/logging/filter.hh”:

  • bool is_timestamp_filtered(const Query & query, const LoggingClock::time_point & timestamp)
    Returns true if the query is filtering out the specified timestamp, or false otherwise.

  • bool is_time_range_filtered(const Query & query, const LoggingClock::time_point & start_time, const LoggingClock::time_point & end_time)
    Returns true if the given time range is completely outside of the query’s range (ie, is filtered out because no data within the range provided overlaps with the query’s range).

  • std::set< core::Guid > resolve_object_identifiers_from_query(const Query & query, const ColumnDeclaration & column)
    Resolves all of the object identifiers that are in the specified query, against the given type information. The final result of this set will be filtered by the query’s allow list.

  • std::set< core::Guid > resolve_latched_object_identifiers(const ColumnDeclaration & column)
    Resolves all of the object identifiers in the given column that have latched flags set.

  • std::pair< LoggingClock::time_point, LoggingClock::time_point > restrict_time_by_query(const Query & query, const LoggingClock::time_point & minimum_time, const LoggingClock::time_point & maximum_time)
    Restricts the given minimum/maximum time by the given query. This will apply the intersection of the two time ranges.

  • std::pair< LoggingClock::time_point, LoggingClock::time_point > restrict_time_by_query(const Query & lhs, const Query & rhs)
    Restricts the given minimum/maximum time by the given query. This will apply the intersection of the two time ranges.

  • FilteredColumnSplitDeclarations filter_column_split_declarations(const Query & query, const ColumnDeclaration & column)
    Filters the split declarations in the given column based on the specified query. Filters them into two bins; “core” splits, which are splits that are necessary to read the data in the query, and “latched” splits, which are splits that are only necessary for reading latched messages.

  • Manifest apply_query_to_manifest(const Query & query, const Manifest & manifest)
    Takes the given manifest and applies the specified query to it. This will remove any columns/splits that are completely out of bounds thanks to the given query.

  • Query apply_query_to_query(const Query & source, const Query & target)
    This will apply the given query ‘source’ to the given query ’target’. The ’target’ query will be restricted based on the values in ‘source’ (in other words, time ranges will be the minimum of the two, and message lists will be the intersection between the two).

Declared in “ark/logging/helpers.hh”:

  • std::vector< ObjectDeclaration > collect_object_declarations_from_manifest(const Manifest & manifest)
    Retrieves all of the object declarations stored in the given manifest, across all of the columns. Internal filters are obeyed; things like amendments will have the amended declarations returned, not the originals.

  • std::pair< LoggingClock::time_point, LoggingClock::time_point > get_manifest_time_range(const Manifest & manifest)
    Returns the log starting and ending time, as a pair of timestamps.

  • uint64_t get_manifest_referenced_data_size(const Manifest & manifest)
    Returns the amount of data (size, in bytes) that is referenced by the given manifest.

  • uint64_t get_log_ondisk_size(const std::filesystem::path & manifest_path)
    Returns the amount of data consumed by the given log.

  • std::map< logging::ObjectDeclaration, logging::ObjectStatistics > get_summarized_object_statistics(const Manifest & manifest)
    Returns overall statistics about a given log, summarizing the individual object statistics and returning a single map that goes from object identifier to statistics.

  • std::optional< std::chrono::system_clock::time_point > pipeline_to_global_time_from_manifest(const Manifest & manifest, const LoggingClock::time_point & at_time)
    Translates the given pipeline time (such as indexed time) into a global time point through the use of the global information in the time model. Returns an empty optional if the model is not available.

  • std::optional< std::chrono::system_clock::time_point > get_manifest_global_start_time(const Manifest & manifest)
    Returns the global time that a log was captured at, if a time model was logged. Otherwise, returns an empty optional.

  • std::optional< std::chrono::system_clock::time_point > get_manifest_global_stop_time(const Manifest & manifest)
    Returns the global time that a log was finished at, if a time model was logged. Otherwise, returns an empty optional.

  • core::Guid get_original_log_identifier_from_split_identifier(const Manifest & manifest, const core::Guid & split_identifier)
    Returns the original log identifier the given split file identifier first appeared in. For example, in an amendment, this will return the log identifier of the base log for splits generated originally, or the log identifier of the amendment for splits generated during amending.

  • std::vector< core::Guid > get_manifest_referenced_split_file_identifiers(const Manifest & manifest)
    Returns all split file identifiers referenced by this manifest.

  • serialization::SchemaRegistry get_manifest_schema_registry(const Manifest & manifest)
    Returns a serialization registry that covers every type in the manifest.

  • std::optional< core::Guid > get_manifest_organization_identifier(const Manifest & manifest)
    Extracts the organization metadata from the given manifest. Returns an empty optional in the event that the metadata is not present (or is malformed in some way).

  • std::string get_compression_level_for_channel_from_manifest(const Manifest & manifest, const std::string & channel_name)
    Returns the compression level for the given channel name.

Declared in “ark/logging/log_reader.hh”:

  • LogReaderConfiguration create_reader_config_from_path(const std::filesystem::path & path)
    Creates a log reader configuration from the given path. Assumes the path points directly to a manifest file, and that the directory layout follows normal conventions (manifest is in a manifest directory, splits are in a splits directory).

  • LogReaderConfiguration create_reader_config_from_guid(const core::Guid & guid, const core::Url & catalog_url)
    Creates a log reader configuration from the given GUID. The catalog will be queried for presigned URLs necessary to access this GUID, and an appropriate log registry will be constructed to read data from it.

    If the catalog URL is left empty, the system environment variable ‘CATALOG_API_URL’ will be used to determine where to access to get further log information. If that is also empty, a well-known default will be used (

  • LogReaderConfiguration create_reader_config_from_url(const core::Url & url)
    Creates a log reader configuration from the given URL, creating an appropriate log registry based on the URL scheme. It assumes the URL points directoy to a manifest file, and the directory layout follows normal conventions.

Declared in “ark/logging/log_writer.hh”:

  • LogWriterConfiguration initialize_new_common_log(std::string name)
    Initializes a new log writer configuration, suitable for a new set of independent log data. This assumes that this log will represent new data that is being collected, and is not a snippet or amendment.

  • LogWriterConfiguration initialize_amended_log(const Manifest & original_manifest, const Query & restrictions)
    Initializes a new log writer configuration, suitable for amending an existing log. This takes in both the manifest of the original log, and the query that you wish to apply to the existing data (for example, to restrict time range or objects).

Declared in “ark/logging/object_declaration.hh”:

  • ObjectDeclaration make_object_declaration(std::string name)
    Creates an object type suitable for use within the logger. This will generate a new unique identifier, and extract an appropriate type identifier.

  • ObjectDeclaration make_latched_object_declaration(std::string name)
    Returns a latched object declaration. Latched objects will be replayed at the beginning of any query; they allow you to do things like write configuration out once, but come at a space cost.

Declared in “ark/logging/offload.hh”:

  • catalog::PresignUploadLogRequest to_presigned_upload_request(const logging::Manifest & manifest)
    Takes the given manifest and transforms it into a presigned-upload URL request for the catalog.

  • OffloadLogTargetUrls get_offload_log_target_urls(const catalog::PresignUploadLogResponse & upload_response)
    Takes the given presigned upload log response and produces a list of target URLs that can be used to offload a log.

  • OffloadLogTargetUrls get_offload_log_target_urls(const logging::Manifest & manifest, const core::Url & base_url)
    Takes the given manifest and produces a set of URLs that can be used to offload the manifest to the given HTTPS server.

  • bool is_log_file_already_offloaded(const OffloadLogTargetUrlPair & url_pair)
    Returns true if the given log file exists on a remote server, or false otherwise.

  • OffloadLogFileStatus offload_log_file(const std::filesystem::path & source_path, const OffloadLogTargetUrlPair & url_pair, comms::HttpProgressCallback progress_callback)
    Offloads the given log file (a manifest or split file) to the destination URL. This will only offload if the file is not already there or not the same size.

    If the file already exists, the progress callback will be invoked with the file size. A status will be returned to indicate if a file actually was offloaded.

Declared in “ark/logging/paths.hh”:

  • std::filesystem::path resolve_variables_in_logging_path(const std::string & path, const core::Guid & log_identifier, const std::chrono::system_clock::time_point & now)
    A logging path may contain several variables, such as identifiers or dates that are not known at configuration time. This will resolve those, for example: /tmp/logs/${YEAR}-${MONTH}/${YEAR}-${MONTH}-${DAY_OF_MONTH}/${LOG_IDENTIFIER}.

    Becomes: /tmp/logs/2021-11/2021-11-18/89120123-6781-3456-123491235721 Variables:

  • bool is_likely_manifest(const std::filesystem::path & base_path, const std::filesystem::path & manifest_path)
    Returns true if the given path is likely to be a manifest file.

Declared in “ark/logging/repair.hh”:

  • LogRepairState repair_log(const core::Url & log_url, Manifest & manifest, const std::function< void(const std::string &)> & error_callback)
    Makes a best effort to repair the given log. Walks through the data in the log and tries to cut away any regions that appear to be corrupted. Modifies the manifest in such a way that you can read data again, and returns the state of the repair for debugging.

Declared in “ark/logging/test/utility.hh”:

  • LogReaderConfiguration get_reader_config_from_writer(const LogWriterConfiguration & writer_config, const Manifest & manifest)
    Takes the given log writer configuration and manifest, and returns a log reader configuration that can be used to access the log.

Declared in “ark/logging/testing/testlog_helper.hh”:

Declared in “ark/logging/validation.hh”:

  • void validate_query(const Query & query)
    Throws if the given query is invalid (ie, contains conflicting information or information that is obviously bad).

  • void validate_manifest(const Manifest & manifest)
    Throws if the given manifest is invalid (ie, contains conflicting information or information is obviously bad). It is probably not safe to read an invalid manifest.

Declared in “ark/logging/wire_structure.hh”:

  • struct ark::logging::SplitIndexEntry __attribute__()
    The header of a Swift message.

    A system-heartbeat message, sent to indicate the overall health of the system. The offset to translate from local time to GPS time for messages tagged as such. The baseline between 2 Swift devices. This contains the relative vector distance from the base station to the rover receiver with the NED coordinate system definde by the local WGS84 tangent plane centerd at the base station. Absolute geodetic coordinates and status (best available) with covariances See fix mode for the values. The orientation of the device, in vehicle body frame. The rotations should be applied intrinsically in the order yaw, pitch, and roll in order to rotate the from a frame aligned with the local-level NED frame to the vehicle body frame. The velocity in vehicle body frame. The x-axis points at the nose of the vehicle, y-axis points to the right-hand side, and z-axis points downward. The velocity in local NED (north/east/down) coordinate. This frame is defined as the local WGS84 tangent plane, centered on the current location. The velocity in ECEF coordinate frame. A log message (ASCII text) coming from the device. Absolute geodetic coordinates and status (best available). See fix mode for the values. This structure contains configuration information about the IMU. It is necessary to fully decode SwiftMsgImuRaw packets. Raw data from the IMU, containing accelerometer and gyroscope readings. The sense of measurements are aligned with the indications on the device itself. Reports UTC time. The flags indicate the source of the offset value and fix. A decoded GPS time message, from the Swift. Represents the time since the GPS epoch began; the week begins at the Saturday/Sunday transition. Measurement units come from the MSG_IMU_AUX.