AWS

We use AWS for all of our cloud functionality at the moment. The AWS C++ SDK is built and available for use (although you may need to enable your module by modifying the cmake file – right now, only the S3 client is built).

Sevices are expected to make use of Terraform for all of their infrastructure, as this allows us to have repeatable infrastructure creation and migration. Don’t adjust service details manually; Terraform will just clobber it.

In general, you will want to source the env.sh file, as that places a lot of our common tools into your path, including the versions of aws and terraform that we test with.

Authenticating (CLI)

You should use the ark-auth-tool to communicate with the Catalog and related services. It works by fetching and storing JWT credentials in your ~/.cache/tbd/jwt.json file.

To obtain Catalog access:

~/ark$ ./build/ark-auth-tool -u dtascione
Your access token has expired or doesn't exist. You must reauthenticate.
Password:
Authentication complete.

There are a number of options in ark-auth-tool to make it easier to authenticate headlessly. In particular, you may pass you username and password in through the ARK_JWT_USERNAME and ARK_JWT_PASSWORD environment variables.

If you wish to remove the token from your local disk, use the --revoke command line flag. Please see --help for a full list of options.

If you need access to AWS resources directly (and you have a special account on Catalog), you can obtain AWS credentials using the ark-aws-auth-tool:

~/ark$ ./build/ark-aws-auth-tool --fetch-aws-creds
Checking cached credentials...
No cached credentials found.
Please reauthenticate. Using `dtascione` as username.
Password:
>> Access Token valid
>> AWS Credentials valid (expires at 2023-01-19 22:30:00 -- [11 hours 59 minutes])

Authenticating (API)

While generated rbuf code for accessing services will automatically try to refresh credentials, you may wish to do so manually.

    #include "ark/aws/authenticator.hh"
    #include "ark/aws/messages.hh"
    #include "ark/aws/refresh_credentials.hh"

    // Invoke to automatically refresh credentials, if possible
    ark::aws::try_refresh_credentials();

    // Invoke to retrieve the user's JWT (if valid)
    ark::aws::Authenticator authenticator;
    if (authenticator.access_token_valid())
    {
        auto jwt = authenticator.get_jwt();
    }

The above code will try to refresh your credentials; if they are valid, you can fetch the JWT that is useful for interacting with many services.

S3 API

A simple S3 API exists to allow you to easily upload or download data to S3 at ark::aws::S3Client.

#include "ark/aws/s3client.hh"

aws::S3Client client("my-bucket-name");

client.upload("/strings/test.txt", "Hello world! This is uploaded content!");

You can also upload files directly, or vectors of bytes.

Downloading works similarly:

auto downloaded_bytes = client.download("/key/name");

The APIs try to avoid making unnecessary copies of data where possible. They will throw if error conditions are encountered (such as not having the write credentials).

S3 is now strongly consistent, making it somewhat-useful to check if items exist or not. The exists(key_name); API exists to help with that. This runs an S3 HeadObject request in the background, to determine if the object exists or not.

For more advanced uploading cases, you can populate a S3UploadRequest structure and provide that to the upload series of APIs instead. This contains additional parameters:

  • key_name - The key to upload as
  • policy - An object policy (such as, making the object public or private)
  • content_type - The content type to assign
  • content_disposition - A content disposition to assign
  • progress_callback - A callback that is invoked as data is uploaded

Additionally, downloading objects can take an optional S3ProgressCallback that is invoked periodically with the number of bytes downloaded since the last time the callback was invoked.

Batch API

Similar to S3, the Batch API wraps several AWS SDK components to produce an easier-to-use (or at least, more consistent with the rest of our APIs) interface to AWS Batch.

With this interface, you can submit jobs and check on their state. For an example:

#include "ark/aws/batch.hh"

BatchClient client;

// Construct a request. This assumes that most of the job configuration
// is actually in your job definition. This allows you to override
// the command you execute.
BatchJobRequest request;

request.name = "MyJobName";
request.queue = "MyJobQueue";
request.definition = "MyJobDefinition";
request.command = {"/bin/echo", "Hello world!"};

auto job = batch.submit_job(request);

// submit_job will return the identifier of the newly-created
// job, which you can use to retrieve status...
auto state = batch.get_job_state(job);

std::cout << state.human_readable_status << ", logs at: " << state.log_stream_name << std::endl;

Cloudwatch API

Wrappers also exist for interacting with Cloudwatch. At the moment, they simply enable you to request events that were written out to a particular log stream (for example, if you wanted to get stdout from a batch job):

#include "ark/aws/logs_client.hh"

LogsClient logs;

auto stdout = logs.download_logs("/my/stream/name");