ark::serialization

Enums

• JsonReadValueType (int32_t)
  Potential types that a JSON value can be.

• RbufConstantType (int32_t)
  The type of constant value held.

• FieldType (int32_t)
  Indicates the type of the field this indicates what primitive ’type’ this field is (for example, if its a string or integer).

• FieldAttribute (uint32_t)
  This is a bitmask containing various attributes that could be set for fields.

• SchemaAttribute (uint32_t)
  This is a bitmask containing variou sattributes that could be set for schemas.

• EnumType (int8_t)
  Indicates an enumeration type (bitmask or plain values).

Typedefs

Defined in “ark/serialization/field_value.hh”:

• using ObjectPtr = std::shared_ptr< Object >
  Shared-pointer wrapper for Object.

Defined in “ark/serialization/schema.hh”:

• using FieldDescriptorPtr = std::shared_ptr< FieldDescriptor >
  A pointer for fields.

Variables

Defined in “ark/serialization/json_reader.hh”:

• concept GettableJsonType= std::is_same_v<ObjectType, std::string> ||
  std::is_same_v<ObjectType, bool> ||
  std::is_same_v<ObjectType, uint8_t> ||
  std::is_same_v<ObjectType, uint16_t> ||
  std::is_same_v<ObjectType, uint32_t> ||
  std::is_same_v<ObjectType, uint64_t> ||
  std::is_same_v<ObjectType, int8_t> ||
  std::is_same_v<ObjectType, int16_t> ||
  std::is_same_v<ObjectType, int32_t> ||
  std::is_same_v<ObjectType, int64_t> ||
  std::is_same_v<ObjectType, float> ||
  std::is_same_v<ObjectType, double> ||
  std::is_same_v<ObjectType, core::Guid> ||
  std::is_same_v<ObjectType, core::ByteBuffer> ||
  std::is_same_v<ObjectType, std::chrono::steady_clock::time_point> || std::is_same_v<ObjectType, std::chrono::system_clock::time_point> || std::is_same_v<ObjectType, std::chrono::nanoseconds>
  This concept can be used to ensure a template parameter is a supported type that we can get to.

• concept GettableEnumType= std::is_enum_v
  This concept can be used to ensure a template parameter is an enumerated type.

• concept ValueBasedEnumType= ark::serialization::is_value_based_enum::value
  This concept can be used to ensure a template parameter is a value-based enumerated type.

• concept DeserializableObject= ark::serialization::is_serializable::value && !std::is_same_v<ObjectType, ark::core::Guid> && !std::is_same_v<ObjectType, ark::core::ByteBuffer>
  This concept can be used to ensure a template parameter is an serializable type.

Defined in “ark/serialization/json_writer.hh”:

• concept Enumeration= std::is_enum_v
  This concept can be used to ensure a template parameter is an enumerated type.

• concept SerializableObject= ark::serialization::is_serializable::value && !std::is_same_v<ObjectType, ark::core::Guid> && !std::is_same_v<ObjectType, ark::core::ByteBuffer>
  This concept can be used to ensure a template parameter is an serializable type.

Defined in “ark/serialization/magic.hh”:

• constexpr uint8_t BITSTREAM_HEADER_MAGIC= 0xd0
  Magic bits that are at the top of every bitstream header.

• constexpr uint8_t BITSTREAM_GROUP_MAGIC= 0xe0
  Magic bits that start every group.

• constexpr uint8_t BITSTREAM_HAS_FOLLOWING_SECTION= 0x04
  Indicates (in the magic block) if there is a following block.

Classes

• ark::serialization::CcGenerator
  This generator allows you to take in a schema registry and a type name, and then emit all of the necessary C++ code to build code to automatically serialize and deserialize the schema (concretely).

• ark::serialization::FieldValue
  Our fields can be object of the specified values, depending on the field type. This is a clever wrapper around variant, so that we can support recursive field values (contained inside a container, for example).

• ark::serialization::Hasher
  This class wraps a hash algorithm and allows you to create a running hash from fields contained within a serialized structure.

• ark::serialization::IndexedVariant
  A simple class used to assist in rbuf serialization/deserialization of variants with custom index definitions.

• ark::serialization::InputStream
  An input stream allows you to deserialize data from a stream and into appropriate C++ primitives, safely. It is the lowest level primitive in the serialization stack.

• ark::serialization::JsGenerator
  This generator allows you to take in a generator configuration and then emit all of the necessary Javascript code to automatically serialize/deserialize that schema.

• ark::serialization::JsonArrayIterator
  Creates a read iterator over the given JSON document.

• ark::serialization::JsonBlob
  This class represents a container for a JSON object.

• ark::serialization::JsonDocumentWriter
  Creates a new JSON document.

• ark::serialization::JsonObjectIterator
  Creates a read iterator over the given JSON object.

• ark::serialization::JsonObjectWriter
  Creates a new JSON document, allowing you to write out objects/arrays/values into a JSON message.

• ark::serialization::JsonReadOnlyValue
  A wrapper over a JSON value, allowing you to check the type and get the underlying value out.

• ark::serialization::JsonWriteOnlyArray
  A wrapper over a mutable JSON array, allowing you to push values onto said array.

• ark::serialization::JsonWriteOnlyValue
  A wrapper over a mutable JSON value, allowing you to set the value to a primitive.

• ark::serialization::Object
  This is an ‘object’ type, which represents a object that we don’t know the concrete type of, but which we can decode from a schema and a bitstream.

  This is useful for decoding serialized data and displaying/working with it, assuming you have the schema but not generated code.

• ark::serialization::OutputStream
  An output stream allows you to serialize data in C++ to a stream that can later be read from, safely. It is the lowest level primitive in the serialization stack.

• ark::serialization::ParquetWriter
  This class can consume flat rbuf messages and convert them into rows within a parquet file. It’s typically used for emitting metrics. It uses reflection to retrieve values from the rbuf messages, and so performance is not as fast as writing directly from the structures.

• ark::serialization::PyGenerator
  This generator allows you to take in a generator configuration and then emit all of the necessary Python code to automatically serialize/deserialize that schema.

• ark::serialization::RbufSyntaxErrors
  An exception indicating that parsing rbuf failed.

• ark::serialization::SchemaRegistry
  A schema registry is a container that allows you to hold multiple schemas together in a single place. This allows you to quickly decode multiple “any” objects, even nested “any” objects, from one class.

  You must manually add schemas into this registry for it to be able to decode “any” objects.

• ark::serialization::EnumDescriptor
  An enum descriptor stores metadata about a particular enumeration. Specifically, it stores the name, and the mapping of values to names.

• ark::serialization::FieldDescriptor
  A field represents a particular object that will be serialized out to the bitstream.

• ark::serialization::OptionalGroup
  A versioned group is a list of field declarations that belong in a particular “grouping” that may or may not be present when deserializing.

• ark::serialization::OutputStreamState
  State of the output stream. Used to maintain pointers when nesting serialization calls (such as when writing out objects).

• ark::serialization::ParsedOptionalGroupHeader
  This is returned when reading group headers, allowing downstream clients to determine if they want to skip over this header or continue processing data.

• ark::serialization::RbufConstantValue
  A constant value (key/value pair) that is intended to be accessible from the top-level headers.

• ark::serialization::RbufGeneratorConfiguration
  Configuration for the code generators. This allows you to setup helper paths and configuration that affects the codegen.

• ark::serialization::RbufIncludedFile
  The result of parsing an include file. Contains the registry along with the resolved include path.

• ark::serialization::RbufOptions
  Configuration options for the rbuf parser use these for setting things like include paths or optional filenames.

• ark::serialization::RbufParseResult
  The result of parsing an rbuf file. This contains the resulting registry (all of the schemas & enumerations), along with a list of the types compiled out of the input directly (ie, not imported).

  The ordering of the types will be the same as the ordering within the rbuf itself.

• ark::serialization::RpcDeclaration
  An API function declaration is used to define a function within a service (a RPC).

• ark::serialization::RpcService
  An RPC service this is a collection of RPC declarations, and can be used to generate code that executes remote procedure calls.

• ark::serialization::Schema
  A schema represents a programmatic way to serialize and deserialize messages from just a schema (ie, without a concrete type). This can be helpful for external tools that want to decode data without compiling code.

• ark::serialization::SerializerType

• ark::serialization::SerializerType< T, typename std::enable_if< std::is_base_of_v< google::protobuf::MessageLite, T > >::type >

• ark::serialization::SerializerType< T, typename std::enable_if<::ros::message_traits::IsMessage< T >::value >::type >

• ark::serialization::SourceLocation
  Used to record the original file and line number this descriptor/definition came from.

• ark::serialization::has_schema
  This trait indicates if the given structure has a schema or not. If you have implemented the schema calls for a type, then you should have this trait available.

• ark::serialization::has_schema< T, typename std::enable_if< std::is_base_of_v< google::protobuf::MessageLite, T > >::type >

• ark::serialization::has_schema< T, typename std::enable_if<::ros::message_traits::IsMessage< T >::value >::type >

• ark::serialization::is_serializable
  This trait indicates if the given structure is serializable or not. If you have implemented serialize() and deserialize() calls, then you should also implement this trait.

• ark::serialization::is_serializable< EnumDescriptor >

• ark::serialization::is_serializable< FieldDescriptor >
  Traits for serialization.

• ark::serialization::is_serializable< OptionalGroup >

• ark::serialization::is_serializable< SchemaRegistry >
  Traits for serialization.

• ark::serialization::is_serializable< Schema >

• ark::serialization::is_serializable< SourceLocation >

• ark::serialization::is_serializable< T, typename std::enable_if< std::is_base_of_v< google::protobuf::MessageLite, T > >::type >

• ark::serialization::is_serializable< T, typename std::enable_if<::ros::message_traits::IsMessage< T >::value >::type >

• ark::serialization::is_serializable< core::ByteBuffer >
  Trait to indicate that this type is serializable.

• ark::serialization::is_serializable< core::Guid >
  Trait to indicate that this type is serializable.

• ark::serialization::is_value_based_enum
  A trait that is intended to indicate if a type is a value-based enumeration that is serializable. It provides certain guarantees on to_string/from_string functionality.

Functions

Declared in “ark/serialization/byte_buffer.hh”:

• void serialize< core::ByteBuffer >(OutputStream & stream, const core::ByteBuffer & source)
  Supports serializations serializes the given byte buffer into the specified output stream.

• void deserialize< core::ByteBuffer >(InputStream & stream, core::ByteBuffer & destination)
  Supports deserializations deserializes the given byte buffer into the specified output stream.

• void hash< core::ByteBuffer >(Hasher & hasher, const core::ByteBuffer & source)
  Supports hashing. Hashes the given GUID into the ongoing hash.

Declared in “ark/serialization/cast.hh”:

• OutputType cast_field_value(const FieldValue & value)
  Casts the given field value to the specified templated output type, based on the type that is contained within the field value itself.

  Throws if the value inside the field value is non-castable.

Declared in “ark/serialization/file.hh”:

• std::shared_ptr< serialization::Object > read_generic_object_from_serialized_file(const std::filesystem::path & path)
  Reads an unknown object from the given file, returning it as a generic object. Uses schema information to deserialize it.

• void write_serializable_type_to_file(const std::filesystem::path & path, const Type & type, core::WriteOptions options)
  Writes the given serializable type out to the specified file. This is just a helper routine wrapping serialize and writing to files.

  You can pass in additional write options if you wish to ensure data is ‘power safe’ (such as by passing in the WriteOptions::Synchronize flag).

• core::ByteBuffer write_serializable_type_to_buffer(const Type & type)
  Writes the given serializable type out a buffer, returning the results as a byte buffer. This can then be written to disk and read in via routines in this file.

  This may make copies of the data during serialization, so is not as efficient as writing directly to a file.

• Type read_serializable_type_from_file(const std::filesystem::path & path)
  Reads the given serializable type from the specified file. The path is loaded in from memory, and then deserialize is invoked on the resulting bytes.

Declared in “ark/serialization/generator_helpers.hh”:

• std::string enum_field_to_base_type(const SchemaRegistry & registry, const FieldDescriptor & field)
  Resolves the given enumeration field in the registry and emits its base type as a string.

• std::string enum_class_name(const SchemaRegistry & registry, const FieldDescriptor & field)
  Fetches the name of the enum, throws if the field isn’t an enum type.

• std::string object_to_base_type(const SchemaRegistry & registry, const std::string & object_type)
  Resolves the given object field in the registry and emits its base type as a string.

• std::string object_field_to_base_type(const SchemaRegistry & registry, const FieldDescriptor & field)
  Resolves the given object field in the registry and emits its base type as a string.

• std::string array_field_to_base_type(const SchemaRegistry & registry, const FieldDescriptor & field)
  Resolves the given array field container type in the registry and emits its base type as a string.

• bool has_field_type(const SchemaRegistry & registry, FieldType type)
  Returns true if we have any fields of the given type throughout the given registry.

• bool has_field_attribute(const SchemaRegistry & registry, FieldAttribute attribute)
  Returns true if we have any fields with the given attribute throughout the given registry.

Declared in “ark/serialization/guid.hh”:

• void serialize< core::Guid >(OutputStream & stream, const core::Guid & source)
  Supports serializations serializes the given GUID into the specified output stream.

• void deserialize< core::Guid >(InputStream & stream, core::Guid & destination)
  Supports deserializations deserializes the given GUID into the specified output stream.

• void hash< core::Guid >(Hasher & hasher, const core::Guid & source)
  Supports hashing. Hashes the given GUID into the ongoing hash.

Declared in “ark/serialization/helpers.hh”:

• core::ByteBuffer serialize(const Type & type)
  This is a helper routine that allows you to serialize an object and get back a byte buffer, as a one-liner.

• Type deserialize(const core::ByteBuffer & buffer)
  This is a helper routine that allows you to deserialize an object from a byte buffer, as a one liner.

• Type deserialize(const std::string_view & buffer)
  This is a helper routine that allows you to deserialize an object from a string, as a one liner.

• Type from_json(const std::string_view & buffer)
  This is a helper routine that can translate a JSON string into an object, with a one-liner.

• Type from_json(const serialization::JsonBlob & blob)
  This is a helper routine that can translate a JSON blob into an object, with a one-liner.

Declared in “ark/serialization/json_patch.hh”:

• Type apply_json_merge_patch(const Type & original, const std::string_view & patch)
  This allows you to take an existing serializable object and apply a ‘merge patch’ to it (for example, for changing a subset of the values in it).

• Type apply_json_patch(const Type & original, const std::string_view & patch)
  This allows you to take an existing serializable object and apply a ‘JSON patch’ to it (for example, for changing a subset of the values in it).

Declared in “ark/serialization/json_reader.hh”:

• const char * to_string(JsonReadValueType type)
  Converts the read value type into a string.

• void get_to(const JsonReadOnlyValue & value, std::vector< bool > & destination)
  Retrieves the current value into the given vector.

• void from_json(JsonObjectIterator iterator, Type & destination)
  A templated type that allows you to decode JSON into the given object, from an object iterator.

• void get_to(const JsonReadOnlyValue & value, ObjectType & destination)
  Retrieves the current value into the given object, throwing if there is a type mismatch that cannot be converted automatically.

• void get_to(const JsonReadOnlyValue & value, EnumType & destination)
  Retrieves the current value into the given object, throwing if there is a type mismatch that cannot be converted automatically.

• void get_to(const JsonReadOnlyValue & value, std::optional< Type > & destination)
  Retrieves the current value into the given object, throwing if there is a type mismatch that cannot be converted automatically.

• void get_to(const JsonReadOnlyValue & value, std::vector< Type > & destination)
  Retrieves the current value into the given vector.

• void get_to(const JsonReadOnlyValue & value, std::array< Type, Size > & destination)
  Retrieves the current value into the given array.

• void get_to(const JsonReadOnlyValue & value, std::map< std::string, Value > & destination)
  Retrieves the current value into the given dictionary.

• void get_to(const JsonReadOnlyValue & value, std::map< core::Guid, Value > & destination)
  Retrieves the current value into the given dictionary.

• void get_to(const JsonReadOnlyValue & value, std::map< Key, Value > & destination)
  Retrieves the current value into the given dictionary, converting the string to an enumerated value.

  Retrieves the current value into the given dictionary.

• void get_to(const JsonReadOnlyValue & value, IndexedVariant< VariantSubtypes… > & destination)
  Retrieves the current value into the given variant.

Declared in “ark/serialization/json_writer.hh”:

• void to_json(JsonDocumentWriter & writer, JsonObjectWriter root_object, const Type & source)
  Writes a serializable object into the given root object (overwriting fields, existing fields are not re-initialized).

Declared in “ark/serialization/object.hh”:

• FieldValue string_to_field_value(const std::string & value, const SchemaRegistry & registry, const Schema & schema, const FieldDescriptor & field)
  Converts a string to a field value, based on the type. This will automatically convert the string into the appropriate type (uint32_t, std::string, etc), based on the field type specified.

• std::chrono::nanoseconds duration_from_string(const std::string & value)
  Converts a string into a duration, applying rbuf-style formatting rules. This means that the string can end with a suffix to indicate units. In other words, if the string has no suffix, it will be interpreted as nanoseconds. ‘_s’ will be interpreted as seconds, ‘_ms’ will be interpreted as ‘milliseconds’, ‘_m’ will be interpreted as minutes, and ‘_h’ will be interpreted as hours.

• std::string field_value_to_string(const FieldValue & value, const SchemaRegistry & registry, const Schema & schema, const FieldDescriptor & field)
  Converts the given field value to a string, based on the field type and schema.

Declared in “ark/serialization/rbuf_depfiles.hh”:

• std::string emit_depfiles(const std::filesystem::path & rbuf_path, const std::filesystem::path & primary_output_path, const serialization::RbufParseResult & parse_result)
  Emit a depfile (as content that could be written to a file), containing all the dependencies of the primary output file, in the GCC depfile format suitable for being read from Ninja.

Declared in “ark/serialization/rbuf_grammar.hh”:

• const char * get_rbuf_grammar()
  Returns the PEG grammar used by the rbuf parser. Kept in a separate file to make rebuilds on grammar changes more managable.

Declared in “ark/serialization/rbuf_parser.hh”:

• RbufParseResult parse_rbuf(const std::string & input, RbufOptions options)
  Takes in data in the ‘rbuf’ format, and parses it into an appropriate schema structure. From there, you can either deserialize data encoded with that schema, or generate code that can parse data more efficiently.

Declared in “ark/serialization/schema.hh”:

• const char * field_type_to_string(FieldType type)
  A free-function that converts a field type into a human-readable string.

• FieldType field_type_from_string(const std::string & type)
  A free-function that converts a string into a field type.

• const char * enum_type_to_string(EnumType type)
  A free-function that converts an enumeration type into a human-readable string.

• EnumType enum_type_from_string(const std::string & type)
  A free-function that converts a string into an enumeration type.

• bool is_field_type_fundamental(FieldType type)
  Returns true if the given field type is a fundamental, false otherwise. A fundamental is defined as something like a uint or double, rather than a string or list.

• bool is_field_type_complex(FieldType type)
  Returns true if the given field type is ‘complex’, as in, is an object, list, container, or something that is nested. Primitives, strings, GUIDs, etc, are all considered “simple”.

• void serialize< FieldDescriptor >(OutputStream & stream, const FieldDescriptor & source)
  Supports serialization of the schemas.

• void deserialize< FieldDescriptor >(InputStream & stream, FieldDescriptor & destination)

• void serialize< EnumDescriptor >(OutputStream & stream, const EnumDescriptor & source)

• void deserialize< EnumDescriptor >(InputStream & stream, EnumDescriptor & destination)

• void serialize< OptionalGroup >(OutputStream & stream, const OptionalGroup & source)

• void deserialize< OptionalGroup >(InputStream & stream, OptionalGroup & destination)

• void serialize< Schema >(OutputStream & stream, const Schema & source)

• void deserialize< Schema >(InputStream & stream, Schema & destination)

• void serialize< SourceLocation >(OutputStream & stream, const SourceLocation & source)

• void deserialize< SourceLocation >(InputStream & stream, SourceLocation & destination)

• JsonBlob to_json< FieldDescriptor >(const FieldDescriptor & source)
  Supports conversion to/from JSON.

• void from_json< FieldDescriptor >(const JsonBlob & json, FieldDescriptor & destination)

• JsonBlob to_json< EnumDescriptor >(const EnumDescriptor & source)

• void from_json< EnumDescriptor >(const JsonBlob & json, EnumDescriptor & destination)

• JsonBlob to_json< OptionalGroup >(const OptionalGroup & source)

• void from_json< OptionalGroup >(const JsonBlob & json, OptionalGroup & destination)

• JsonBlob to_json< Schema >(const Schema & source)

• void from_json< Schema >(const JsonBlob & json, Schema & destination)

• JsonBlob to_json< SourceLocation >(const SourceLocation & source)

• void from_json< SourceLocation >(const JsonBlob & json, SourceLocation & destination)

• constexpr FieldAttribute operator|(FieldAttribute lhs, FieldAttribute rhs)
  Or operator for our bitmask.

• constexpr FieldAttribute operator&(FieldAttribute lhs, FieldAttribute rhs)
  And operator for our bitmask.

• constexpr FieldAttribute operator~(FieldAttribute attribute)
  Inverse operator for our bitmask.

• constexpr SchemaAttribute operator|(SchemaAttribute lhs, SchemaAttribute rhs)
  Or operator for our bitmask.

• constexpr SchemaAttribute operator&(SchemaAttribute lhs, SchemaAttribute rhs)
  And operator for our bitmask.

Declared in “ark/serialization/schema_registry.hh”:

• void serialize< SchemaRegistry >(OutputStream & stream, const SchemaRegistry & source)
  Supports serialization of the schemas.

• void deserialize< SchemaRegistry >(InputStream & stream, SchemaRegistry & destination)

• void to_json< SchemaRegistry >(JsonDocumentWriter & writer, JsonObjectWriter root_object, const SchemaRegistry & source)

• JsonBlob to_json< SchemaRegistry >(const SchemaRegistry & source)
  Supports conversion to/from JSON.

• void from_json< SchemaRegistry >(const JsonBlob & json, SchemaRegistry & destination)

• bool is_equivalent(const SchemaRegistry & lhs_registry, const SchemaRegistry & rhs_registry, const Schema & lhs, const Schema & rhs)
  Returns true if the two schemas are ’equivalent’. They are considered equivalent if they have the same base fields (by type, and in order), and any available groups also match.

Declared in “ark/serialization/serializers.hh”:

• void serialize(OutputStream & stream, const Type & source)
  A function that allows you to serialize a given C++ structure into the specified OutputStream. This will static assert unless the is_serializable trait is true.

• void deserialize(InputStream & stream, Type & destination)
  A function that allows you to deserialize a given C++ structure from the specified InputStream and into the given structure.

  This will static assert unless the is_serializable trait is true.

• JsonBlob to_json(const Type & source)
  A function that allows you to dump the given structure to JSON.

  This will static assert unless the is_serializable trait is true.

• void from_json(const JsonBlob & json, Type & destination)
  A function that allows you read JSON into the given type.

  This will static assert unless the is_serializable trait is true.

• std::string schema_protocol()
  A function that can return the schema type for a serialized type. This will be returned as an arbitrary string (e.g. rbuf, proto, rosmsg, etc).

  This will statically assert until the has_schema trait is true.

• std::string schema()
  A function that can return the schema for a serialized type. This will be returned as an arbitrary string, suitable for passing into the Schema object.

  This will statically assert until the has_schema trait is true.

• std::string type_name()
  A function that can return the type name for the schema of the serialized type. This is used as a mechanism to uniquely identify the type within schema registries.

  This will statically assert until the has_schema trait is true.

• void hash(Hasher & hasher, const Type & source)
  This function will use the specified Hasher to generate a hash for the object, recursively.

  This will statically assert until the has_schema trait is true.

• std::vector< ReturnType > values()
  This function returns the values of an enumeration. This will statically assert unless the has_schema trait is true.

• size_t count()
  This function returns the a count of the values of an enumeration. This will statically assert unless the has_schema trait is true. Returns the number of potential values known to this system; enums coming in from other systems (ie, via forwards/backwards compat) may have differing numbers.

• int64_t max()
  This function returns the maximum value of an enumeration. This will statically assert unless the has_schema trait is true. Returns the maximum value from the values known to this system; enums coming in from other systems (ie, via forwards/backwards compat) may have differing numbers.

Declared in “ark/serialization/timespec.hh”:

• uint64_t to_packed_timespec(const typename Clock::time_point & timestamp)
  Converts the given timepoint into a uint64 value, which consists of the lower bits being set as seconds, and the lower bits being set as nanoseconds.

• Clock::time_point from_packed_timespec(uint64_t timestamp)
  Converts the given uint64 value to a timepoint, assuming the lower bits are set as seconds, and lower bits as nanoseconds.