ark::catalog

Enums

Typedefs

Defined in “ark/catalog/api_gateway/service.hh”:

  • using CreateDatabaseConnectionFn = std::function< database::AbstractDatabaseConnectionPtr(const std::string &url)>
    A function pointer for producing connections.

Variables

Defined in “ark/catalog/organizations.hh”:

  • constexpr static core::Guid CATALOG_ORGANIZATION_TBD{ {0xfb, 0x42, 0xa4, 0x6a, 0x35, 0x50, 0xa2, 0x2e, 0x72, 0xb9, 0xa1, 0x8e, 0x28, 0x92, 0x55, 0x99}}
    A well-known identifier for the TBD organization. The human-readable GUID is “2ea25035-6aa4-42fb-9955-92288ea1b972”.

  • constexpr static core::Guid CATALOG_ORGANIZATION_GLOBAL{ {0x77, 0x47, 0x77, 0x77, 0x77, 0x77, 0x77, 0x77, 0x77, 0x77, 0x77, 0x77, 0x77, 0x77, 0x77, 0x77}}
    A well-known identifier to match ‘all’ organizations. This is used for ingest jobs to run against any log. The human-readable GUID is “77777777-7777-4777-7777-777777777777”.

Classes

Functions

Declared in “ark/catalog/api_gateway/auth.hh”:

  • void append_authorization_to_where(std::stringstream & where_statement, const char * where_prefix, const std::string & organization_source_table, const rest::Authorization & authorization, std::vector< std::string > & parameters, MembershipPermission permission)
    Appends a where query to the given where statement that includes authorization searches based on the authorization given. Parameters will be updated with the necessary queries.

Declared in “ark/catalog/api_gateway/membership.hh”:

  • constexpr MembershipPermission operator|(MembershipPermission lhs, MembershipPermission rhs)
    Operator for adding an option to the list.

  • constexpr MembershipPermission operator&(MembershipPermission lhs, MembershipPermission rhs)
    Operator for adding an option to the list.

  • MembershipPermission & operator|=(MembershipPermission & lhs, MembershipPermission rhs)
    Operator for adding an option to the list.

  • MembershipPermission & operator&=(MembershipPermission & lhs, MembershipPermission rhs)
    Operator for adding an option to the list.

Declared in “ark/catalog/api_gateway/organizations.hh”:

  • bool is_authorization_for_organization(const rest::Authorization & authorization, database::AbstractDatabaseConnectionPtr & connection, const core::Guid & organization_identifier, MembershipPermission permission)
    Returns true if the given authorization statement is in the specified organization. If the authorization is for a user, checks against the user database, and if its a ‘client ID’, then checks against the robots.

Declared in “ark/catalog/api_gateway/robots.hh”:

  • void validate_update_robot_telemetry_request(const UpdateRobotTelemetryRequest & request)
    Checks the given request to determine if it is valid; throws if its invalid with appropriate details.

Declared in “ark/catalog/api_gateway/testing/helpers.hh”:

  • std::filesystem::path create_catalog_test_database(const std::filesystem::path & base_path)
    This routine will spawn a database, populate it with test data, and then return a reference to the path it was created in.

Declared in “ark/catalog/attachments.hh”:

  • UploadAttachmentConfig initialize_attachment_config(const logging::Manifest & manifest, const std::string & remote_filename)
    Popuate an upload attachment config (artifact identifier, organization) from a manifest.

  • void upload_attachment(const UploadAttachmentConfig & config, CatalogApi & api, const core::ByteBuffer & data)
    Uploads an attachment to the catalog, using the given configuration and API client. The data in the byte buffer ‘data’ will be uploaded to the catalog, after receiving a presigned URL location for said data.

    If attachment_artifact_identifier is set, an attachment will be registered (pointing at the uploaded data).

  • void upload_attachment(const UploadAttachmentConfig & config, CatalogApi & api, const std::filesystem::path & local_path)
    Uploads an attachment to the catalog, using the given configuration and API client. The data file ’local_path’ will be uploaded to the catalog, after receiving a presigned URL location for said data.

    If attachment_artifact_identifier is set, an attachment will be registered (pointing at the uploaded data).

  • comms::HttpRequestOptions to_request_options(const UploadAttachmentConfig & config, const PresignUploadAttachmentResponse & presign_response)
    Converts the given config and response into a set of HTTP request options.

Declared in “ark/catalog/config.hh”:

  • catalog::ConfigSchemaFieldType to_schema_field_type(const std::string & field_name, serialization::FieldType type)
    Converts an rbuf field type into a schema field type (which is mostly just value, aside from a few of the containers and object references).

  • RegisterConfigSchemaRequest rbuf_to_config_schema(const serialization::SchemaRegistry & registry, const serialization::Schema & schema, const core::Guid & organization_id)
    Converts an rbuf schema into a config schema, attached to the local organization ID.

  • std::string diff_schema(const std::vector< ConfigSchemaFieldDefinition > & lhs, const std::vector< ConfigSchemaFieldDefinition > & rhs)
    Returns a string containing a diff between two sets of schema fields, suitable for printing out.

  • int64_t apply_units_to_signed_integer(const std::string & value)
    Takes the given string value and returns a signed integer with units applied. Currently supports time-based suffixes (_s, _ms, etc).

Declared in “ark/catalog/database_url.hh”:

  • std::string get_catalog_url(const std::string & username, const std::string & password, const std::string & host)
    Returns a formatted postgres URL for connecting to the catalog database, based on the input parameters.

Declared in “ark/catalog/files.hh”:

  • std::string key_name_for_metrics(const std::string & table_name, const std::string & extension, const std::string & robot_serial, const std::chrono::system_clock::time_point & first_timestamp)
    Returns a key name that is appropriate for use for storage of metrics on a remote server. Essentially forms a partionable stream based on the date, such that the format looks something like:

    <table_name>/2023-06-27/<robot_serial>/2023-06-27_06-10-00.parquet Note to ensure data is largely idempotent, you’ll want to bucket onto your side into deterministic timestamps.

  • TimeType bucket_name_for_metrics(const TimeType & metrics_timestamp, const DurationType & output_interval)
    Returns a good ’time bucket’ to use given the provided timestamp and the interval that you are writing files out with.

Declared in “ark/catalog/ingest.hh”:

  • core::Guid get_manifest_organization_identifier(const logging::Manifest & manifest)
    Returns the given manifest’s organization identifier, based on what is available in said manifest.

  • ArtifactRegistrationRequest make_registration_request(const logging::Manifest & manifest)
    Takes the given manifest and creates a registry request out of it. This is used to “register” logs with the catalog.

  • void validate_create_attachment_request(const CreateInternalAttachmentRequest & request)
    Validate a CreateInternalAttachmentRequest.

  • std::string format_annotations_results(const GetInternalAttachmentResponse & response)
    Output a string that creates a human-readable version of the response for Logged_Annotations.

  • std::string format_distilled_report(const GetInternalAttachmentResponse & response)
    Output a string that creates a human-readable version of the response for LogDistilledReport.

  • std::vector< CreateExternalAttachmentRequest > create_amendment_history_links(const logging::Manifest & manifest)
    Takes the manifest and processes the possible log family chain to create links.

Declared in “ark/catalog/jobs.hh”:

  • std::string serialize_command_line(const std::vector< std::string > & command_line)
    The command line for a job is stored in the database as a JSON array, for convenience. This serializes it.

  • std::vector< std::string > deserialize_command_line(const std::string & command_line)
    The command line for a job is stored in the database as a JSON array, for convenience. This deserializes it.

  • BatchJobCommand merge_batch_job_command(const BatchJobCommand & lhs, const BatchJobCommand & rhs)
    Merges two batch job commands, combining command line arguments and attributes. The “right hand side” wins over the “left hand side”.

  • std::vector< std::string > process_command_line(const CommandLineVariables & variables, const std::vector< std::string > & source)
    Processes the given command, replacing variables with their equivalent values.

  • std::string serialize_job_rules(const IngestJobRuleSet & rules)
    Serializes a set of job rules to JSON.

  • IngestJobRuleSet deserialize_job_rules(const std::string & rules_json)
    Deserializes a set of job rules from JSON.

  • bool does_artifact_pass_rules(const PrimaryArtifactMetadata & metadata, const std::map< std::string, std::string > & attributes, const IngestJobRuleSet & rules)
    Applies the set of rules to the given artifact (and attributes). Returns true if the rules apply to that artifact, or false otherwise.

  • std::string make_batch_job_artifact_name(const std::string & collection_name, size_t item_index_in_collection, const BatchJobCommand & common_command, const BatchJobCommand & specific_command)
    Computes the name for a batch job artifact, based on the commands and the original request/collection name.

Declared in “ark/catalog/sign.hh”:

  • std::string rsa_hash_and_sign_file(CatalogApi & client, const core::Guid & organization_id, const std::string & key_name, const std::filesystem::path & path)
    Signs and hashes the given path, using the given API gateway client. This will hash the file, then reach out to the catalog to sign the hash. The result will be an ‘Ark Signature’.

  • std::string rsa_hash_and_sign_message(CatalogApi & client, const core::Guid & organization_id, const std::string & key_name, const std::string_view & message)
    Signs and hashes the given message, using the given API gateway client. This will hash the message, then reach out to the catalog to sign the hash. The result will be an ‘Ark Signature’.

  • bool verify_signed_authorized_keys(const std::string & public_key, const std::string & content)
    Verifies the given authorized_keys file, based on the public key given. Returns true if verified, false otherwise.

Declared in “ark/catalog/tools/extract_catalog_annotations/annotation_attachment.hh”:

  • catalog::CreateInternalAttachmentRequest make_annotation_attachment_request(core::Url & log_url)
    Reads the log for annotations to generate a CreateInternalAttachmentRequest.

Declared in “ark/catalog/tools/extract_catalog_video/preferred_image.hh”:

  • std::optional< std::string > find_best_camera(const logging::Manifest & manifest)
    This routine looks through a manifest and tries to find a channel to use for camera images. It first looks for preferred images, then falls back to what is available.