AWS
We use AWS for all of our cloud functionality at the moment. The AWS C++ SDK is built and available for use (although you may need to enable your module by modifying the cmake file – right now, only the S3 client is built).
Sevices are expected to make use of Terraform for all of their infrastructure, as this allows us to have repeatable infrastructure creation and migration. Don’t adjust service details manually; Terraform will just clobber it.
In general, you will want to source the env.sh
file, as that places a lot
of our common tools into your path, including the versions of aws
and
terraform
that we test with.
Authenticating (CLI)
You should use the ark-auth-tool
to communicate with the Catalog and
related services. It works by fetching and storing JWT credentials in
your ~/.cache/tbd/jwt.json
file.
To obtain Catalog access:
~/ark$ ./build/ark-auth-tool -u dtascione
Your access token has expired or doesn't exist. You must reauthenticate.
Password:
Authentication complete.
There are a number of options in ark-auth-tool
to make it easier to authenticate
headlessly. In particular, you may pass you username and password in through the
ARK_JWT_USERNAME
and ARK_JWT_PASSWORD
environment variables.
If you wish to remove the token from your local disk, use the --revoke
command line
flag. Please see --help
for a full list of options.
If you need access to AWS resources directly (and you have a special
account on Catalog), you can obtain AWS credentials using the ark-aws-auth-tool
:
~/ark$ ./build/ark-aws-auth-tool --fetch-aws-creds
Checking cached credentials...
No cached credentials found.
Please reauthenticate. Using `dtascione` as username.
Password:
>> Access Token valid
>> AWS Credentials valid (expires at 2023-01-19 22:30:00 -- [11 hours 59 minutes])
Authenticating (API)
While generated rbuf code for accessing services will automatically try to refresh credentials, you may wish to do so manually.
#include "ark/aws/authenticator.hh"
#include "ark/aws/messages.hh"
#include "ark/aws/refresh_credentials.hh"
// Invoke to automatically refresh credentials, if possible
ark::aws::try_refresh_credentials();
// Invoke to retrieve the user's JWT (if valid)
ark::aws::Authenticator authenticator;
if (authenticator.access_token_valid())
{
auto jwt = authenticator.get_jwt();
}
The above code will try to refresh your credentials; if they are valid, you can fetch the JWT that is useful for interacting with many services.
S3 API
A simple S3 API exists to allow you to easily upload or download data to S3
at ark::aws::S3Client
.
#include "ark/aws/s3client.hh"
aws::S3Client client("my-bucket-name");
client.upload("/strings/test.txt", "Hello world! This is uploaded content!");
You can also upload files directly, or vectors of bytes.
Downloading works similarly:
auto downloaded_bytes = client.download("/key/name");
The APIs try to avoid making unnecessary copies of data where possible. They will throw if error conditions are encountered (such as not having the write credentials).
S3 is now strongly consistent, making it somewhat-useful to check if items
exist or not. The exists(key_name);
API exists to help with that. This
runs an S3 HeadObject request in the background, to determine if the object
exists or not.
For more advanced uploading cases, you can populate a S3UploadRequest
structure
and provide that to the upload
series of APIs instead. This contains additional
parameters:
key_name
- The key to upload aspolicy
- An object policy (such as, making the object public or private)content_type
- The content type to assigncontent_disposition
- A content disposition to assignprogress_callback
- A callback that is invoked as data is uploaded
Additionally, downloading objects can take an optional S3ProgressCallback
that is
invoked periodically with the number of bytes downloaded since the last time the callback
was invoked.
Batch API
Similar to S3, the Batch API wraps several AWS SDK components to produce an easier-to-use (or at least, more consistent with the rest of our APIs) interface to AWS Batch.
With this interface, you can submit jobs and check on their state. For an example:
#include "ark/aws/batch.hh"
BatchClient client;
// Construct a request. This assumes that most of the job configuration
// is actually in your job definition. This allows you to override
// the command you execute.
BatchJobRequest request;
request.name = "MyJobName";
request.queue = "MyJobQueue";
request.definition = "MyJobDefinition";
request.command = {"/bin/echo", "Hello world!"};
auto job = batch.submit_job(request);
// submit_job will return the identifier of the newly-created
// job, which you can use to retrieve status...
auto state = batch.get_job_state(job);
std::cout << state.human_readable_status << ", logs at: " << state.log_stream_name << std::endl;
Cloudwatch API
Wrappers also exist for interacting with Cloudwatch. At the moment, they simply enable you to request events that were written out to a particular log stream (for example, if you wanted to get stdout from a batch job):
#include "ark/aws/logs_client.hh"
LogsClient logs;
auto stdout = logs.download_logs("/my/stream/name");