Offload Stage

The offload stage allows you to offload Ark Logs from the onboard system to AWS storage for continued processing and analysis.

Requirements

You must be authenticated with valid credentials in order for the Offload Stage to migrate data.

Configuration

You need to configure where the onboard log data is located and the destination for the data.
Congfiguration is handled through the OffloadStageConfig structure, which can be populated with YAML:

---
config:
    # Set to the local storage location
    storage_base_path: "/tmp/ark_logs"

    # Set the name of the AWS S3 bucket to upload to
    destination_bucket: "ark-logs-rhq"

    # Remove the logs from local storage, after a successful offload
    remove_after_offload: true

This configuration sets the local path for the onboard logs to “/tmp/ark_logs”, this is where the OffloadStage will scan for Ark Logs. The destination is set to the AWS s3 bucket “ark-logs-rhq”.

The flag “remove_after_offload” set to “true” indicates that the data logs should be removed from the local storage, only after a successful offload has completed. If set to “false”, the data logs will simply not be removed.

Optionally, you can configure the server endpoint rather than using the default AWS endpoint. This is very useful for testing purposes, if you want to run your own mocked server locally.

---
  config:
    # The AWS server endpoint (configuable for local testing capabilities)
    server_endpoint: "127.0.0.1:38317"

Interacting

The OffloadStage requires the User to interact with the UI to initiate offload. The user must be authenticated with valid AWS credentials prior to starting offload. Be sure to authenticate through the Authentication Plugin:

The user may also use the command line tool ark-auth-tool to obtain valid access

Once valid credentials are set, use the Offload Plugin to interact with data logs stored onboard the system to select which logs to offload.

Initially, the OffloadStage will scan the storage_base_path directory specified for any ark logs. To force a rescan, simple press the Rescan System button. This will update what Ark Logs are found in the directory. Any non-Ark Log data found will be displayed to the user, and the system will ignore it. A simple summary of the Ark Logs found will be displayed, such as name, log start time, and the log duration.

Select the Ark Logs to offload using the checkboxes at the left of the plugin. Press the Begin Offload button to start the offload process. As the data is offloaded, a progress display will update on the percentage of data offloaded. When complete, the status will be modified to show Completition or Error.

Offload progress can be interrupted and restarted at any time.

As the Ark Data logs are offloaded, the Catalog ingestion job system will activate.

Metrics

The OffloadStage produces a message with a summary of status, offload progress and any errors encountered at 0.5hz rate of This is published on the status channel.