package cdc_ft.proto

Get desktop application:
View/edit binary Protocol Buffers messages

Describes the interface to fetch data from the asset streaming server running on the workstation.

• rpc GetContent (GetContentRequest, GetContentResponse)

  asset_stream_service.proto:28

  Requests the contents of a chunk by its id.

  message GetContentRequest

  asset_stream_service.proto:36

  • repeated ContentId id = 1

    IDs of the requested chunks.

  • uint64 thread_id = 2

    ID of the requesting thread. Used for statistics only.

  message GetContentResponse

  asset_stream_service.proto:44

  • repeated bytes data = 1

• rpc SendCachedContentIds (SendCachedContentIdsRequest, SendCachedContentIdsResponse)

  asset_stream_service.proto:32

  Send the contents of the chunk cache to the server. Used for statistics only.

  message SendCachedContentIdsRequest

  asset_stream_service.proto:48

  • repeated ContentId id = 1

  message SendCachedContentIdsResponse

  asset_stream_service.proto:52

  (message has no fields)

Describes the interface to receive manifest updates and prioritize processing of specific assets.

• rpc GetManifestId (GetManifestIdRequest, stream GetManifestIdResponse)

  asset_stream_service.proto:58

  Streaming channel to receive continuous manifest updates.

  message GetManifestIdRequest

  asset_stream_service.proto:70

  (message has no fields)

  message GetManifestIdResponse

  asset_stream_service.proto:72

  • optional ContentId id = 1

• rpc AckManifestIdReceived (AckManifestIdReceivedRequest, AckManifestIdReceivedResponse)

  asset_stream_service.proto:62

  Acknowledges that a specific manifest ID has been received.

  message AckManifestIdReceivedRequest

  asset_stream_service.proto:76

  • string gamelet_id = 1

  • optional ContentId manifest_id = 2

  message AckManifestIdReceivedResponse

  asset_stream_service.proto:81

  (message has no fields)

• rpc ProcessAssets (ProcessAssetsRequest, ProcessAssetsResponse)

  asset_stream_service.proto:67

  Requests the server to process the given in-progress assets as soon as possible.

  message ProcessAssetsRequest

  asset_stream_service.proto:83

  • repeated string relative_paths = 1

  message ProcessAssetsResponse

  asset_stream_service.proto:87

  (message has no fields)

An Asset represents a file, a directory, or a symlink. An asset can consist of many chunks. Directory asserts embed other assets (directly or indirectly) which describe their content.

Used in: AssetList, Manifest

• string name = 1

  The name of the asset (file or directory name).

• Asset.Type type = 2

  The type of this asset.

• int64 mtime_seconds = 3

  The last modification time of this asset, in seconds since epoche (UTC).

• uint32 permissions = 4

  The permission bits for this asset (RWX for user, group, world, in that order).

• uint64 file_size = 5

  For FILE assets only, the total size of this file, in bytes.

• repeated ChunkRef file_chunks = 6

  For FILE assets only, this is the list of chunks that make up the file contents. The chunk references must be sorted ascending by offset!

• repeated IndirectChunkList file_indirect_chunks = 7

  For FILE assets, an overflow list referencing additional chunks if this is very large asset. The list must be sorted ascending by offset!

• repeated Asset dir_assets = 8

  For DIRECTORY assets only, this is the list of assets referenced by this directory.

• repeated ContentId dir_indirect_assets = 9

  For DIRECTORY assets only, this list of indirect assets holds the content IDs of AssetList protos that hold additional assets which did not fit into the original message anymore due to size restrictions.

• string symlink_target = 10

  For SYMLINK assets only, this field holds the path to the file the symlink points to.

• bool in_progress = 11

  Indicates that this asset has not yet been fully processed and is still missing required information. This field is used for dynamic manifest updates to indicate to the client that it needs to wait for this asset to be fully processed.

Used in: Asset

• UNKNOWN = 0

  Default value for an empty asset. This should never be UNKNOWN in practice.

• FILE = 1

  A regular file.

• DIRECTORY = 2

  A directory.

• SYMLINK = 3

  A symlink.

A list of assets that belong to a directory. While a directory asset has a list of child assets embedded, additional assets might need to overflow into AssetList protos which are referenced by content ID from the parent directory asset.

• repeated Asset assets = 1

This message describes the CDC parameters that were used to create this manifest.

Used in: Manifest

• uint64 min_chunk_size = 1

• uint64 avg_chunk_size = 2

• uint64 max_chunk_size = 3

A list of chunks that an asset consists of. Large assets can consist of multiple lists which are linked.

• repeated ChunkRef chunks = 1

  List of chunk references. The chunks must be ordered by increasing offset. All offsets in this list are relative to the start of this chunk list, meaning that the first chunk always starts at offset zero. The absolute offset of this ChunkList is included in the enclosing IndirectChunkList message.

References a chunk by ContentId at a given offset within the asset. ChunkRefs never overlap or leave gaps, the offset of one chunk plus its size is the offset of the next chunk.

Used in: Asset, ChunkList

• uint64 offset = 1

  The offset of this chunk within its embedding asset.

• optional ContentId chunk_id = 2

  The content hash of the chunk.

This message wraps a hash of chunk contents. It is used as a key to uniquely identify, look up, and deduplicate chunks.

Used in: AckManifestIdReceivedRequest, Asset, ChunkRef, GetContentRequest, GetManifestIdResponse, IndirectChunkList, SendCachedContentIdsRequest

• bytes blake3_sum_160 = 1

  The first 20 byte (= 160 bit) of the BLAKE3 sum of the content. If we assume a total storage size of 1 EiB (2^60) and a chunk size of at least 64 KiB, the probability of a collision for a 160-bit hash is approximately 1e-22 (calculated using https://gist.github.com/zadam/6474221007e27705b3bde0a3a9323a84).

An IndirectChunkList stores additional chunks for very large assets. They contain the absolute offset of their beginning within the asset to allow a fast identification of the correct chunk list for a given position.

Used in: Asset

• uint64 offset = 1

  The offset within an asset where this chunk list begins.

• optional ContentId chunk_list_id = 2

  References a ChunkList proto with additional data chunks. The chunk offsets in the referenced ChunkList are relative, which means that the absolute offset of this IndirectChunkList must be added to each chunk's offset in order to obtain the absolute offset of each chunk.

A manifest is the entry point for a structured description of a hierarchical list of assets. The assets describe a file system hierarchy. The metadata describing those assets can be all embedded into the manifest, or they can be split into smaller chunks for streaming. Manifests can be identified by their content ID just like chunks, which allows them to be stored alongside the chunks.

• optional Asset root_dir = 1

  The root_dir is the entry point into the file system hierarchy described by a manifest. The root_dir asset must be of type DIRECTORY and has no name.

• optional CdcParameters cdc_params = 2

  The CDC parameters that were used to create this manifest.