ark::comms

Enums

• HttpRequestType (uint32_t)
  The type of HTTP request (GET, PUT, etc).

• HttpResponseCode (uint32_t)
  An enumeration of the HTTP response codes that we support.

• SocketFamily
  Enumerates the various socket families that you can construct.

• SocketWriteFlags (int32_t)
  A set of flags that you can provide when writing data out over sockets.

• SocketType
  Enumerates the various socket types that you can construct. TCP sockets are “stream”, and UDP are packet-based.

• ThrottledWebsocketFrameContent (uint8_t)
  An enumeration representing different types of websocket frames.

• WebSocketOpcode
  For websocket frames, indicates the type of message being transmitted/received.

Typedefs

Defined in “ark/comms/forward.hh”:

• using HttpClientWriterPtr = std::shared_ptr< HttpClientWriter >

• using WebSocketClientPtr = std::shared_ptr< WebSocketClient >

• using ThrottledWebsocketWriterPtr = std::shared_ptr< ThrottledWebsocketWriter >

• using SocketPtr = std::shared_ptr< Socket >
  A shared-pointer wrapped socket.

Defined in “ark/comms/http_server.hh”:

• using HttpClientStatePtr = std::shared_ptr< HttpClientState >

• using HttpRegexMatcherPtr = std::shared_ptr< HttpRegexMatcher >

• using StaticMemoryCallback = std::function< HttpResponse(const HttpRequest &)>
  A definition of the callback that is used for URL requests.

• using StaticFileCallback = std::function< HttpResponse(const HttpRequest &, const std::filesystem::path &)>
  A definition for a callback that is used for URL requests. This callback will cause the request body to be stored in a file (which is provided to the callback).

• using StreamingConnectedCallback = std::function< void(const HttpClientWriterPtr &)>
  A definition of a callback that is invoked when a new streaming connection is created.

• using StreamingDisconnectedCallback = std::function< void(const HttpClientWriterPtr &)>
  A definition of a callback that is invoked when a streaming connection disconnects.

• using StreamingMessageReceivedCallback = std::function< void(const core::ByteBuffer &)>
  A definition for a callback that is invoked when a streaming connection receives a message. Note that the byte buffer received here is potentially just wrapping data make sure to check with ref_count() and make a clone(), as it might become invalidated once this callback returns.

Defined in “ark/comms/progress_callback.hh”:

• using HttpProgressCallback = std::function< void(uint64_t)>
  A callback used to report the number of bytes transferred since the previous time that the callback was invoked. (in other words, you might receive ‘4096’, ‘4096’, and ‘4096’ three times in a row, and that means you have transferred 12,288 bytes).

Defined in “ark/comms/throttled_websocket_writer.hh”:

• using ThrottledWebsocketWriteCallback = std::function< void(ThrottledWebsocketOutgoingMessage)>
  Function pointer that is invoked when data needs to be transmitted.

Defined in “ark/comms/websocket_protocol.hh”:

• using WebSocketMessagePtr = std::shared_ptr< WebSocketMessage >

Variables

Defined in “ark/comms/throttled_websocket_writer.hh”:

• constexpr size_t THROTTLED_WEBSOCKET_FRAME_HEADER_MAGIC= 0x10
  A constant expressing the version of the throttled websocket header.

• constexpr size_t THROTTLED_WEBSOCKET_FRAME_HEADER_MAX_SIZE= 4
  A constant expressing the size of the throttled websocket frame header.

Defined in “ark/comms/websocket_protocol.hh”:

• constexpr auto WEBSOCKET_EXTENSION_FIELD_NAME= “Sec-WebSocket-Extensions”
  The name of the websocket extension field.

• constexpr auto WEBSOCKET_PROTOCOL_FIELD_NAME= “Sec-WebSocket-Protocol”
  The name of the websocket protocol field.

• constexpr auto WEBSOCKET_ARK_LZ4_COMPRESSION_EXTENSION= “x-permessage-lz4”
  This is the websocket extension name that indicates if we should send compressed data (LZ4) or not.

• constexpr auto WEBSOCKET_ARK_ZSTD_COMPRESSION_EXTENSION= “x-permessage-zstd”
  This is the websocket extension name that indicates if we should send compressed data (ZSTD) or not.

• constexpr auto WEBSOCKET_ARK_THROTTLING_PROTOCOL= “x-ark-throttling-protocol”
  This is the websocket protocol name that indicates if we are using the Ark-specific windowing protocol.

• constexpr size_t WEBSOCKET_FRAME_HEADER_MAX_SIZE= 16
  The maximum size any websocket frame header can be, in bytes.

Classes

• ark::comms::BadRequestException
  An exception indicating that the client sent a bad request. Typically thrown by HTTP methods to better indicate that there was a problem with an incoming request (other exceptions result in a 500 error).

• ark::comms::DistillSocketStatsStage
  Class to distill SocketStats.

• ark::comms::HttpOutgoingBuffer
  This class represents an outgoing buffer it is really a collection of buffers internally, and we use it to store information if we are falling behind when writing data to clients.

• ark::comms::HttpRequestException
  An exception indicating that the client encountered an exception while processing a request. Contains additional information about the error.

• ark::comms::HttpRequestParser
  A very simple class meant for decoding HTTP requests. You can use this class to consume bytes from a socket and decode them into requests, or generate your own requests.

• ark::comms::HttpServer
  This is a simple HTTP server. It allows you to register callbacks at URLs, so that you can provide custom information to external clients.

  This class can also act as a websocket server, allowing you to register ‘streams’ that clients may subscribe to. This acts as a simple publish/subscribe mechanism.

• ark::comms::HttpServerStage
  This stage is used for communications. It provides an HTTP server that can give insight into the rest of the system providing streaming APIs for communication pathways that support it, and static APIs to do things like “list the available channels”.

• ark::comms::NetworkAddress
  A configuration for a network address. This includes an IP address and a port, which can be used for binding or connecting sockets.

• ark::comms::Socket
  A wrapper around socket routines this is a low level wrapper that turns libsock routines into something more modern. The semantics of these APIs are about the same as their POSIX equivalents.

• ark::comms::SocketStatsTracker
  Thread safe class for computing the socket statistics. Used to compute both overall (packets received/bytes received) and compute rate information.

• ark::comms::TcpClientStage
  This stage allows you to connect to a remote TCP socket, and it will publish “packet groups” to the rest of the system. While TCP is normally a stream-oriented protocol, this will essentially consider a packet as a single return from a recv() call.

• ark::comms::TcpServerStage
  This stage provides a TCP server, binding to the port of your choice. Any incoming packets will be published over the channel as TcpPacketGroups, and any ’egress’ packets will be sent to all clients.

• ark::comms::ThrottledWebsocketWriter
  This class is intended to act as a ’throttle’ for sending messages and content over a standard websocket (or anything that accepts byte buffer outputs).

  It works by keeping a local queue of messages of some fixed size, and then ensuring only some number of messages are on the wire (and unacknowledged) at once. Messages are FIFO, and we drop oldest first.

• ark::comms::UdpSocketStage
  A generic UDP socket interface stage that will listen to a given UDP port an publish any packets received onto the configured channel.

• ark::comms::WebSocketClient
  This class represents a websocket client it can talk to a websocket server and send/receive messages. This client will handle disconnects/reconnects transparently in the background.

• ark::comms::HttpClientOndiskRequestPayload
  This structure represents content for an HTTP payload. It maps an open file descriptor that we write to, to a path on-disk, and handles automatic cleanup when the client disconnects.

• ark::comms::HttpClientState
  Represents internal state for an HTTP client. This is just used within the server itself to manage the clients.

• ark::comms::HttpClientWriter
  Represents an internal writer for an HTTP client that is streaming. This allows external code (in callbacks) to “write” to a particular client.

• ark::comms::HttpFileBasedRequest
  Emitted from on-disk (path-based) publishers in the HTTP server stage. See the stage configuration for more information.

• ark::comms::HttpHelperResponse
  A response from the HTTP helpers, from the helpers that return a full response rather than just the body.

• ark::comms::HttpMultiPartBody
  A body containing a multi-part HTTP upload and its content type. You can pass this through the standard HTTP helpers.

• ark::comms::HttpMultiPartIndividualBody
  A request for a multi-part upload file. This can be provided to specify the raw file contents along with its type.

• ark::comms::HttpOutgoingBufferEntry
  An individual buffer entry, used by the HttpOutgoingBuffer class.

• ark::comms::HttpRequest
  A basic structure representing an HTTP request. These can be parsed from the HttpRequestParser, or converted into a valid request by using str() on it.

• ark::comms::HttpRequestOptions
  A structure containing more advanced options for complex HTTP requests.

• ark::comms::HttpResponse
  A basic structure representing a response to an HTTP request. This can be encoded with the str() method, and is then acceptable for a reply to a client.

• ark::comms::HttpServerConfig
  A configuration structure for the HTTP server. Allows you to define some common properties of the server.

• ark::comms::HttpServerStatistics
  Statistics reported back from the HTTP server. This reports some details on error/request rates that might be useful.

• ark::comms::StaticCallbacks
  A set of callbacks that could be used for static execution. Typically one of these will be set (a memory or file-based callback).

• ark::comms::StreamingCallbacks
  A definition of a streaming callback these are installed when an external party wants to listen for streaming (ie, websocket) requests coming into the HTTP server.

• ark::comms::ThrottledWebsocketFrame
  A structure representing a throttled websocket frame header.

• ark::comms::ThrottledWebsocketOutgoingMessage
  This structure represents a message that is going out over the wire.

• ark::comms::ThrottledWebsocketWriterConfig
  Configuration for the throttled websocket writer. This helps indicate how the writer should behave.

• ark::comms::WebSocketClientConfig
  Configuration for the websocket client. Allows you to adjust various settings for your connection.

• ark::comms::WebSocketClientStatistics
  Statistics related to a websocket client. This allows you to poll and understand how a websocket is doing.

• ark::comms::WebSocketFrameHeader
  Represents a frame of data to be transmitted over a websocket. This can be encoded or decoded.

• ark::comms::WebSocketMessage
  Represents a websocket message. Includes the opcode of the message, along with raw bytes that were contained in it.

Functions

Declared in “ark/comms/http_helpers.hh”:

• HttpHelperResponse make_http_request(const ark::core::Url & target, HttpRequestType request_type, const std::string_view & body, const HttpRequestOptions & options)
  A simple helper to execute an HTTP request against the given URL. If specified, the body will be attached to the request. This will return a helper response object, which can be inspected to determine status code, the response body, and any errors.

• HttpHelperResponse make_http_request_checked(const NetworkAddress & address, const std::string & path, HttpRequestType request_type, const std::string_view & body, const HttpRequestOptions & options)
  Similar to the standard make_http_request, but this will throw if there is an error during the process (such as a timeout, authentication issues, etc).

• HttpHelperResponse make_http_request_checked(const ark::core::Url & target, HttpRequestType request_type, const std::string_view & body, const HttpRequestOptions & options)
  Similar to the standard make_http_request, but this will throw if there is an error during the process (such as a timeout, authentication issues, etc).

• std::string http_get(const comms::NetworkAddress & address, const std::string & path, const std::string & body, const HttpRequestOptions & options)
  A simple helper to execute an HTTP GET against the given server. The ‘path’ will be queried. There is an optional ‘body’ that will be supplied with the GET request if specified.

  Throws if there is anything but a ‘200’ response, and will return the response body.

• std::string http_get(const core::Url & url, const std::string & body, const HttpRequestOptions & options)
  A simple helper to execute an HTTP GET against the given URL. There is an optional ‘body’ that will be supplied with the GET request if specified.

  Throws if there is anything but a ‘200’ response, and will return the response body.

• HttpHelperResponse http_get_to_file_request(const core::Url & url, const std::filesystem::path & path, const HttpRequestOptions & options)
  A simple helper to execute an HTTP GET against the given URL. Throws if there is anything but a ‘200’ response. The contents will be streamed to a file on disk.

• void http_get_to_file(const core::Url & url, const std::filesystem::path & path, const HttpRequestOptions & options)
  A simple helper to execute an HTTP GET against the given URL. Throws if there is anything but a ‘200’ response. The contents will be streamed to a file on disk.

• std::string http_post(const core::Url & url, const core::ByteBuffer & bytes, const HttpRequestOptions & options)
  A simple helper to execute an HTTP POST against the given server. The ‘path’ will be queried, and the ‘bytes’ will be used as a payload.

  Throws if there is anything but a ‘200’ response. Returns the body of the response.

• void http_put(const NetworkAddress & address, const std::string & path, const core::ByteBuffer & bytes, const HttpRequestOptions & options)
  A simple helper to execute an HTTP PUT against the given server. The ‘path’ will be queried, and the ‘bytes’ will be used as a payload.

  Throws if there is anything but a ‘200’ response.

• void http_put(const core::Url & url, const core::ByteBuffer & bytes, const HttpRequestOptions & options)
  A simple helper to execute an HTTP PUT against the given server. The bytes will be PUT into the given URL.

  Throws if there is anything but a ‘200’ response.

• HttpHelperResponse http_put_from_file_request(const core::Url & url, const std::filesystem::path & source, const HttpRequestOptions & options)
  A simple helper to execute an HTTP PUT against the given server. The file will be uploaded to the given URL.

• void http_put_from_file(const core::Url & url, const std::filesystem::path & source, const HttpRequestOptions & options)
  A simple helper to execute an HTTP PUT against the given server. The file will be uploaded to the given URL.

  Throws if there is anything but a ‘200’ response.

• HttpHelperResponse retry_http_request(const std::function< HttpHelperResponse()> & request_function, size_t retry_limit, std::chrono::milliseconds backoff_base)
  A helper function that will automatically retry the given request in the event that the HTTP request has failed with a non-4xx level status code. For example, this will not retry authentication failures (403), but will retry spurious DNS or 5xx series failures.

  After each failure, we back off longer and longer. The base time we use for this is defaulted to one second. Throws if the number of retry attempts is exhausted. Returns the final (successful) response otherwise.

• HttpMultiPartBody make_http_multipart_body(const std::vector< HttpMultiPartIndividualBody > & individual_parts)
  Constructs a multipart upload based on the given series of upload requests. Returns the overall content-type that you should use, along with a body for the request.

Declared in “ark/comms/http_protocol.hh”:

• std::map< std::string, std::string, core::CaseInsensitiveComparator > parse_http_fields(const std::vector< std::string_view > & lines, size_t start_offset)
  This can parse a list of key/value pairs from the given array of strings. It’s intended to be used to parse HTTP request/response fields.

Declared in “ark/comms/http_request_type.hh”:

• std::string to_string(HttpRequestType type)
  Translates the HTTP request type into a string suitable for sending over the wire.

Declared in “ark/comms/http_response.hh”:

• HttpResponse make_http_response(HttpResponseCode code)
  Helper to return a response that is marked ‘forbidden’.

Declared in “ark/comms/packet_timestamp.hh”:

• std::optional< std::chrono::steady_clock::time_point > get_first_timestamp_udp_packet(const UdpPacketGroup & msg)
  Function to extract the timestamp from the UDP PacketGroup message. This retrieve the timepoint associated with the first packet in the packet group.

• std::optional< std::chrono::steady_clock::time_point > get_first_timestamp_tcp_packet(const TcpPacketGroup & msg)
  Function to extract the timestamp from the TCP PacketGroup message. This retrieve the timepoint associated with the first packet in the packet group.

Declared in “ark/comms/pipeline_comms.hh”:

• OutputType fetch_latest_from_pipeline(const std::string & pipeline_hostname, const std::string & channel_name)
  Executes a remote fetch from a pipeline. This will grab the latest version of any message from the pipeline (if any) and return it.

• void publish_to_pipeline(const std::string & pipeline_hostname, const std::string & channel_name, const InputType & input)
  Writes the given message out to the pipeline at the given hostname on the specified channel.

Declared in “ark/comms/remote_api.hh”:

• OutputType remote_api_call(const comms::NetworkAddress & address, const std::string & path)
  Executes a remote get operation and deserializes the JSON result. Throws if there is some error deserializing or fetching.

• OutputType remote_api_call(const comms::NetworkAddress & address, const std::string & path, const InputType & input, const std::string & bearer_token)
  Executes a remote get operation and deserializes the JSON result. Throws if there is some error deserializing or fetching.

• OutputType remote_api_call(const core::Url & url, const std::string & bearer_token)
  Executes a remote get operation and deserializes the JSON result. Throws if there is some error deserializing or fetching.

• OutputType remote_api_call(const core::Url & url, const InputType & input, const std::string & bearer_token)
  Executes a remote get operation and deserializes the JSON result. Throws if there is some error deserializing or fetching.

Declared in “ark/comms/resolve.hh”:

• NetworkAddress resolve(const core::Url & url)
  Takes the given URL and resolves its hostname down to a valid IP address that we can connect to.

Declared in “ark/comms/socket_family.hh”:

• std::string to_string(SocketFamily family)
  Converts the given socket family into a human readable string.

Declared in “ark/comms/socket_type.hh”:

• std::string to_string(SocketType type)
  Converts the given socket type into a human readable string.

Declared in “ark/comms/stages/http_server_stage.hh”:

• HttpServerConfig make_http_server_stage_config(const HttpServerStageConfig & stage_config)
  Converts from the HttpServerStageConfig to a HttpServerConfig. Made public to allow for easier testing.

Declared in “ark/comms/url_encode.hh”:

• std::string encode_url_data(const std::string_view & data)
  Encodes the given data with URL-encoding scheme (suitable for being passed as certain form data or query parameters).

• std::string decode_url_data(const std::string_view & data)
  Decodes the previously URL-encoded data (such as passed in with form data or query parameters).

• std::string encode_url_query(const std::initializer_list< std::pair< std::string_view, std::string_view » & mapping)
  Encodes each of the key/value pairs and returns a single query string.

• std::map< std::string, std::string > decode_url_query(const std::string_view & query)
  Decodes the given URL query string into key/value pairs. Breaks up the query itself (using & as a delimiter) and then decodes the individual data components.

• std::string encode_url_query(const Mapping & mapping)
  Encodes each of the key/value pairs and returns a single query string.

Declared in “ark/comms/websocket_protocol.hh”:

• std::string construct_websocket_acceptance(const std::string & key)
  This routine will construct the appropriate handshake acceptance key when upgrading a normal HTTP connection into a websocket connection.

• bool has_websocket_extension(const std::map< std::string, std::string, core::CaseInsensitiveComparator > & fields, const std::string & extension)
  Returns true if the given mapping contains the extension name, or false otherwise.

• bool has_websocket_protocol(const std::map< std::string, std::string, core::CaseInsensitiveComparator > & fields, const std::string & protocol)
  Returns true if the given mapping contains the protocol name, or false otherwise.

• core::ByteBuffer apply_masking_key(uint32_t masking_key, const core::ByteBuffer & data)
  Returns the unmasked/masked message. Makes a copy of the data.