package rerun.cloud.v1alpha1
Get desktop application:
View/edit binary Protocol Buffers messages
service RerunCloudService
cloud.proto:23The Rerun Cloud public API. ## Headers Most endpoints in the Rerun Cloud service require specific gRPC headers to be set. The so-called "standard dataset headers" correspond to at least one of the following headers: * x-rerun-entry-id: ID of the entry of interest, e.g. `1860390B087BC65F602d68eb646c385c`. * x-rerun-entry-name-bin: Name of the entry of interest, e.g. `droid:sample2k`. Headers with a -bin suffix must be base64-encoded (HTTP only supports ASCII values, UTF8 strings must binary encoded).
rpc CreateDatasetEntry (, )
cloud.proto:34message
cloud.proto:803optional string name = 1
Name of the dataset entry to create. The name should be a short human-readable string. It must be unique within all entries in the catalog. If an entry with the same name already exists, the request will fail. Entry names ending with `__manifest` are reserved.
optional id = 2
If specified, create the entry using this specific ID. Use at your own risk.
message
cloud.proto:816optional dataset = 1
rpc CreateIndex (, )
cloud.proto:110Creates a custom index for a specific column (vector search, full-text search, etc). This endpoint requires the standard dataset headers.
message
cloud.proto:340optional config = 3
message
cloud.proto:356optional index = 3
The properties/configuration of the newly created index.
bytes statistics_json = 4
Backend-specific statistics about the index. This is guaranteed to be valid JSON.
optional debug_info = 2
Optional debug information about the index-creation task
rpc CreateTableEntry (, )
cloud.proto:36message
cloud.proto:822string name = 1
Name of the dataset entry to create. The name should be a short human-readable string. It must be unique within all entries in the catalog. If an entry with the same name already exists, the request will fail. Entry names ending with `__manifest` are reserved.
optional provider_details = 2
Information about the table to register. This must be encoded message of one one of the following supported types: - LanceTable
optional schema = 3
Schema of the table to create
message
cloud.proto:839optional table = 1
rpc DeleteEntry (, )
cloud.proto:30message
cloud.proto:780optional id = 1
message
cloud.proto:784(message has no fields)
rpc DeleteIndexes (, )
cloud.proto:120Delete a custom index for a specific column. This endpoint requires the standard dataset headers.
message
cloud.proto:386optional column = 1
Which column to delete the indexes for.
message
cloud.proto:391repeated indexes = 1
Which indexes were actually deleted. Can be empty if no matching indexes were found.
rpc DoGlobalMaintenance (, )
cloud.proto:189Run global maintenance operations on the platform: this includes optimization of all datasets, garbage collection of unused data, and can include more in the future.
message
cloud.proto:732Request all maintenance operations to run on all datasets
(message has no fields)
message
cloud.proto:734(message has no fields)
rpc DoMaintenance (, )
cloud.proto:185Rerun Manifests maintenance operations: scalar index creation, compaction, etc. This endpoint requires the standard dataset headers.
message
cloud.proto:687bool optimize_indexes = 2
Optimize all builtin and user-defined indexes on this dataset. This merges all individual index deltas back in the main index, improving runtime performance of all indexes.
bool retrain_indexes = 6
Retrain all user-defined indexes on this dataset from scratch. This retrains all user-defined indexes from scratch for optimal runtime performance. This is faster than re-creating the indexes, and automatically keeps track of their configurations. This implies `optimize_indexes`.
bool compact_fragments = 3
Compact the underlying Lance fragments, for all Rerun Manifests. Hardcoded to the default (optimal) settings.
optional cleanup_before = 4
If set, all Lance fragments older than this date will be removed, for all Rerun Manifests. In case requested date is more recent than 1 hour, it will be ignored and 1 hour ago timestamp will be used. This is to prevent still used files (like recent transaction files) to be removed and cause Lance Dataset update issues. See https://docs.rs/lance/latest/lance/dataset/cleanup/index.html and https://docs.rs/lance/latest/lance/dataset/cleanup/fn.cleanup_old_versions.html
bool unsafe_allow_recent_cleanup = 5
Override default platform behavior and allow cleanup of recent files. This will respect the value of `cleanup_before` timestamp even if it's more recent than 1 hour. ⚠️ Do not ever use this unless you know exactly what you're doing. Improper use will lead to data loss.
message
cloud.proto:727string report = 1
rpc FetchChunks (, stream )
cloud.proto:152Fetch specific chunks from Rerun Cloud. In a 2-step query process, result of 1st phase, that is, the result of `QueryDataset` should include all the necessary information to send the actual chunk requests, which is the 2nd step of the query process. See `FetchChunksRequest` for details on the fields that describe each individual chunk.
message
cloud.proto:631repeated chunk_infos = 1
Information about the chunks to fetch. These dataframes have to include the following columns: * `chunk_id` - Chunk unique identifier * `partition_id` - partition this chunk belongs to. Currently needed as we pass this metadata back and forth * `partition_layer` - specific partition layer. Currently needed as we pass this metadata back and forth * `chunk_key` - chunk location details
message
cloud.proto:640repeated chunks = 1
Every gRPC response, even within the confines of a stream, involves HTTP2 overhead, which isn't cheap by any means, which is why we're returning a batch of `ArrowMsg` rather than a single one.
rpc FindEntries (, )
cloud.proto:28message
cloud.proto:770optional filter = 1
message
cloud.proto:774repeated entries = 1
Returns the schema of the dataset manifest. To inspect the data of the dataset manifest, which is guaranteed to match the schema returned by this endpoint, check out `ScanDatasetManifest`. This endpoint requires the standard dataset headers.
message
cloud.proto:305(message has no fields)
message
cloud.proto:307optional schema = 1
rpc GetDatasetSchema (, )
cloud.proto:103Returns the schema of the dataset. This is the union of all the schemas from all the underlying partitions. It will contain all the indexes, entities and components present in the dataset. This endpoint requires the standard dataset headers.
message
cloud.proto:329(message has no fields)
message
cloud.proto:334optional schema = 1
Returns the schema of the partition table. This is not to be confused with the schema of the dataset itself. For that, refer to `GetDatasetSchema`. To inspect the data of the partition table, which is guaranteed to match the schema returned by this endpoint, check out `ScanPartitionTable`. This endpoint requires the standard dataset headers.
message
cloud.proto:272(message has no fields)
message
cloud.proto:277optional schema = 1
rpc GetTableSchema (, )
cloud.proto:160message
cloud.proto:648optional table_id = 1
message
cloud.proto:652optional schema = 1
rpc ListIndexes (, )
cloud.proto:115List all user-defined indexes in this dataset. This endpoint requires the standard dataset headers.
message
cloud.proto:372(message has no fields)
message
cloud.proto:374repeated indexes = 1
The respective properties/configuration of the indexes.
repeated bytes statistics_json = 2
Backend-specific statistics about the indexes. This is guaranteed to be valid JSON. If non-empty, this is the same length as `indexes`, and in the same order.
rpc QueryDataset (, stream )
cloud.proto:145Perform Rerun-native queries on a dataset, returning the matching chunk IDs, as well as information that can be sent back to Rerun Cloud to fetch the actual chunks as part of `FetchChunks` request. In this 2-step query process, 1st step is getting information from the server about the chunks that contain relevant information. 2nd step is fetching those chunks (the actual data). These Rerun-native queries include: * Filtering by specific partition and chunk IDs. * Latest-at, range and dataframe queries. * Arbitrary Lance filters. To fetch the actual chunks themselves, see `FetchChunks`. Passing chunk IDs to this method effectively acts as a IF_EXIST filter. This endpoint requires the standard dataset headers.
message
cloud.proto:507repeated partition_ids = 2
Client can specify what partitions are queried. If left unspecified (empty list), all partitions will be queried.
repeated chunk_ids = 3
Client can specify specific chunk ids to include. If left unspecified (empty list), all chunks that match other query parameters will be included.
repeated entity_paths = 4
Which entity paths are we interested in? Leave empty, and set `select_all_entity_paths`, in order to query all of them.
bool select_all_entity_paths = 7
If set, the query will cover all existing entity paths. `entity_paths` must be empty, otherwise an error will be raised. Truth table: ```text select_all_entity_paths | entity_paths | result ------------------------+----------------+-------- false | [] | valid query, empty results (no entity paths selected) false | ['foo', 'bar'] | valid query, 'foo' & 'bar' selected true | [] | valid query, all entity paths selected true | ['foo', 'bar'] | invalid query, error ```
repeated string fuzzy_descriptors = 10
Which components are we interested in? If left unspecified, all existing components are considered of interest. This will perform a basic fuzzy match on the available columns' descriptors. The fuzzy logic is a simple case-sensitive `contains()` query. For example, given a `log_tick__SeriesLines:width` index, all of the following would match: `SeriesLines:width`, `Width`, `SeriesLines`, etc.
bool exclude_static_data = 8
If set, static data will be excluded from the results.
bool exclude_temporal_data = 9
If set, temporal data will be excluded from the results.
optional scan_parameters = 5
Generic parameters that will influence the behavior of the Lance scanner.
optional query = 6
message
cloud.proto:560optional data = 1
rpc QueryTasks (, )
cloud.proto:175Query the status of submitted tasks
message
cloud.proto:739`QueryTasksRequest` is the request message for querying tasks status
repeated ids = 1
Empty queries for all tasks if the server allows it.
message
cloud.proto:746`QueryTasksResponse` is the response message for querying tasks status encoded as a record batch
optional data = 1
rpc QueryTasksOnCompletion (, stream )
cloud.proto:178Query the status of submitted tasks as soon as they are no longer pending
message
cloud.proto:752`QueryTasksOnCompletionRequest` is the request message for querying tasks status. This is close-to-a-copy of `QueryTasksRequest`, with the addition of a timeout.
repeated ids = 1
Empty queries for all tasks if the server allows it.
optional timeout = 2
Time limit for the server to wait for task completion. The actual maximum time may be arbitrarily capped by the server.
message
cloud.proto:762`QueryTaskOnCompletionResponse` is the response message for querying tasks status encoded as a record batch. This is a copy of `QueryTasksResponse`.
optional data = 1
rpc ReadDatasetEntry (, )
cloud.proto:41Fetch metadata about a specific dataset. This endpoint requires the standard dataset headers.
message
cloud.proto:845(message has no fields)
message
cloud.proto:850optional dataset = 1
rpc ReadTableEntry (, )
cloud.proto:45message
cloud.proto:894optional id = 1
message
cloud.proto:898optional table = 1
rpc RegisterTable (, )
cloud.proto:158Register a foreign table as a new table entry in the catalog.
message
cloud.proto:871string name = 1
Name of the table entry to create. The name should be a short human-readable string. It must be unique within all entries in the catalog. If an entry with the same name already exists, the request will fail. Entry names ending with `__manifest` are reserved.
optional provider_details = 2
Information about the table to register. This must be encoded message of one one of the following supported types: - LanceTable
message
cloud.proto:887optional table_entry = 1
Details about the table that was created and registered.
rpc RegisterWithDataset (, )
cloud.proto:54Register new partitions with the Dataset. This endpoint requires the standard dataset headers.
message
cloud.proto:252repeated data_sources = 2
common.v1alpha1.IfDuplicateBehavior on_duplicate = 3
message
cloud.proto:260optional data = 1
rpc ScanDatasetManifest (, stream )
cloud.proto:95Inspect the contents of the dataset manifest. The data will follow the schema returned by `GetDatasetManifestSchema`. This endpoint requires the standard dataset headers.
message
cloud.proto:311repeated string columns = 3
A list of column names to be projected server-side. If empty, all columns are returned. If not empty, the returned `RecordBatch` are guaranteed to only have the requested column, in the order they were requested. If a projected column does not exist, or is projected more than once, the `ScanDatasetManifest` call will fail with an `InvalidArgument` error.
message
cloud.proto:324optional data = 1
The contents of the dataset manifest (i.e. information about layers) as Arrow RecordBatch.
rpc ScanPartitionTable (, stream )
cloud.proto:80Inspect the contents of the partition table. The data will follow the schema returned by `GetPartitionTableSchema`. This endpoint requires the standard dataset headers.
message
cloud.proto:281repeated string columns = 3
A list of column names to be projected server-side. If empty, all columns are returned. If not empty, the returned `RecordBatch` are guaranteed to only have the requested column, in the order they were requested. If a projected column does not exist, or is projected more than once, the `ScanPartitionTable` call will fail with an `InvalidArgument` error.
message
cloud.proto:300optional data = 1
Partitions metadata as Arrow RecordBatch.
rpc ScanTable (, stream )
cloud.proto:162message
cloud.proto:656optional table_id = 1
TODO(jleibs): support ScanParameters iff we can plumb them into Datafusion TableProvider Otherwise, just wait for Arrow Flight rerun.common.v1alpha1.ScanParameters scan_parameters = 2;
message
cloud.proto:663optional dataframe_part = 1
rpc SearchDataset (, stream )
cloud.proto:127Search a previously created index. This endpoint requires the standard dataset headers.
message
cloud.proto:461optional column = 2
Index column that is queried
optional query = 3
Query data - type of data is index specific. Caller must ensure to provide the right type. For vector search this should be a vector of appropriate size, for inverted index this should be a string. Query data is represented as a unit (single row) RecordBatch with 1 column.
optional properties = 4
Index type specific properties
optional scan_parameters = 5
Scan parameters
message
cloud.proto:481optional data = 1
Chunks as arrow RecordBatch
rpc UpdateDatasetEntry (, )
cloud.proto:43message
cloud.proto:856optional id = 1
The dataset to modify.
optional dataset_details = 2
The new values.
message
cloud.proto:864optional dataset = 1
The updated dataset entry
rpc UpdateEntry (, )
cloud.proto:32message
cloud.proto:788optional id = 1
The entry to modify.
optional entry_details_update = 2
The new values for updatable fields.
message
cloud.proto:796optional entry_details = 1
The updated entry details
rpc Version (, )
cloud.proto:24message
cloud.proto:194(message has no fields)
message
cloud.proto:196optional build_info = 1
rpc WriteChunks (stream , )
cloud.proto:61Write chunks to one or more partitions. The partition ID for each individual chunk is extracted from their metadata (`rerun:partition_id`). This endpoint requires the standard dataset headers.
message
cloud.proto:264optional chunk = 1
message
cloud.proto:268(message has no fields)
rpc WriteTable (stream , )
cloud.proto:170Write record batches to a table. This endpoint requires the standard dataset headers. TODO(#11645): endpoints with streaming input are not supported by `grpc-web`. A non-streaming shim will need to be added if/when the viewer uses this endpoint.
message
cloud.proto:678optional dataframe_part = 1
TableInsertMode insert_mode = 2
message
cloud.proto:683(message has no fields)
message
cloud.proto:455TODO(zehiko) add properties as needed
Used in:
(message has no fields)
message
cloud.proto:503TODO(zehiko) add properties as needed
Used in:
(message has no fields)
message
cloud.proto:229Used in:
optional string storage_url = 1
Where is the data for this data source stored (e.g. s3://bucket/file or file:///path/to/file)?
optional string layer = 3
/ Which Partition Layer should this data source be registered to? / / Defaults to `base` if unspecified.
bool prefix = 4
/ Is this a prefix URL (a directory)? / If true, all files of `typ` under this prefix will be / considered part of this data source.
DataSourceKind typ = 2
What kind of data is it (e.g. rrd, mcap, Lance, etc)?
Used in:
DATA_SOURCE_KIND_UNSPECIFIED = 0
DATA_SOURCE_KIND_RRD = 1
message
cloud.proto:946Used in: ,
optional blueprint_dataset = 3
The blueprint dataset associated with this dataset (if any).
optional default_blueprint = 4
The partition of the blueprint dataset corresponding to the default blueprint (if any).
message
cloud.proto:954Used in: , ,
optional details = 1
optional dataset_details = 4
Dataset-specific information, may be update with `UpdateDatasetEntry`
optional dataset_handle = 2
Read-only
message
cloud.proto:997Optional debug info
Used in:
optional uint64 memory_used = 1
The amount of memory used by the task or service call in bytes
message
cloud.proto:926Minimal info about an Entry for high-level catalog summary
Used in: , , ,
optional id = 1
The EntryId is immutable
optional string name = 2
The name of this entry.
EntryKind entry_kind = 3
The kind of entry
optional created_at = 4
optional updated_at = 5
message
cloud.proto:941Updatable fields of an Entry
Used in:
optional string name = 2
The name of this entry.
message
cloud.proto:902Used in:
optional id = 1
optional string name = 2
optional EntryKind entry_kind = 3
What type of entry. This has strong implication on which APIs are available for this entry.
Used in: ,
ENTRY_KIND_UNSPECIFIED = 0
Always reserve unspecified as default value
ENTRY_KIND_DATASET = 1
Order as TYPE, TYPE_VIEW so things stay consistent as we introduce new types.
ENTRY_KIND_DATASET_VIEW = 2
ENTRY_KIND_TABLE = 3
ENTRY_KIND_TABLE_VIEW = 4
ENTRY_KIND_BLUEPRINT_DATASET = 5
message
cloud.proto:201Application level error - used as `details` in the `google.rpc.Status` message
ErrorCode code = 1
error code
string id = 2
unique identifier associated with the request (e.g. recording id, recording storage url)
string message = 3
human readable details about the error
Error codes for application level errors
Used in:
ERROR_CODE_UNSPECIFIED = 0
unused
ERROR_CODE_OBJECT_STORE_ERROR = 1
object store access error
ERROR_CODE_METADATA_DB_ERROR = 2
metadata database access error
ERROR_CODE_CODEC_ERROR = 3
Encoding / decoding error
message
cloud.proto:412used to define which column we want to index
Used in: , ,
optional entity_path = 1
The path of the entity.
optional component = 2
Component details
message
cloud.proto:398Used in: , , ,
optional properties = 1
what kind of index do we want to create and what are its index specific properties.
optional column = 2
Component / column we want to index.
optional time_index = 3
What is the filter index i.e. timeline for which we will query the timepoints. TODO(zehiko) this might go away and we might just index across all the timelines
message
cloud.proto:420Used in:
oneof props
inverted = 1
vector = 2
btree = 3
message
cloud.proto:486Used in:
oneof props
specific index query properties based on the index type
inverted = 1
vector = 2
btree = 3
message
cloud.proto:428Used in:
optional bool store_position = 1
optional string base_tokenizer = 2
TODO(zehiko) add other properties as needed
message
cloud.proto:495TODO(zehiko) add properties as needed
Used in:
(message has no fields)
message
cloud.proto:991A foreign table stored as a Lance table.
string table_url = 1
The URL of the Lance table.
message
cloud.proto:564Used in:
optional latest_at = 1
If specified, will perform a latest-at query with the given parameters. You can combine this with a `QueryRange` in order to gather all the relevant chunks for a full-fledged dataframe query (i.e. they get OR'd together).
optional range = 2
If specified, will perform a range query with the given parameters. You can combine this with a `QueryLatestAt` in order to gather all the relevant chunks for a full-fledged dataframe query (i.e. they get OR'd together).
bool columns_always_include_everything = 3
If true, `columns` will contain the entire schema.
bool columns_always_include_chunk_ids = 4
If true, `columns` always includes `chunk_id`,
bool columns_always_include_byte_offsets = 5
If true, `columns` always includes `byte_offset` and `byte_size`.
bool columns_always_include_entity_paths = 6
If true, `columns` always includes `entity_path`.
bool columns_always_include_static_indexes = 7
If true, `columns` always includes all static component-level indexes.
bool columns_always_include_global_indexes = 8
If true, `columns` always includes all temporal chunk-level indexes.
bool columns_always_include_component_indexes = 9
If true, `columns` always includes all component-level indexes.
message
cloud.proto:602A chunk-level latest-at query, aka `LatestAtRelevantChunks`. This has the exact same semantics as the query of the same name on our `ChunkStore`.
Used in:
optional index = 1
Which index column should we perform the query on? E.g. `log_time`. Leave this empty to query for static data.
optional int64 at = 2
What index value are we looking for? Leave this empty to query for static data.
message
cloud.proto:620/ A chunk-level range query, aka `RangeRelevantChunks`. This has the exact same semantics as the query of the same name on our `ChunkStore`.
Used in:
optional index = 1
Which index column should we perform the query on? E.g. `log_time`.
optional index_range = 2
What index range are we looking for?
message
cloud.proto:986SystemTableKind kind = 1
Used in:
SYSTEM_TABLE_KIND_UNSPECIFIED = 0
Always reserve unspecified as default value
SYSTEM_TABLE_KIND_NAMESPACES = 1
Not used yet
SYSTEM_TABLE_KIND_ENTRIES = 2
All of the entries in the associated namespace
message
cloud.proto:967Used in: , ,
optional details = 1
optional provider_details = 3
Details specific to the table-provider
Used in:
TABLE_INSERT_MODE_UNSPECIFIED = 0
Always reserve unspecified as default value
TABLE_INSERT_MODE_APPEND = 1
Appends new rows to the existing table without modifying any existing rows.
TABLE_INSERT_MODE_OVERWRITE = 2
Overwrites all existing rows in the table with the new rows.
Used in:
VECTOR_DISTANCE_METRIC_UNSPECIFIED = 0
VECTOR_DISTANCE_METRIC_L2 = 1
VECTOR_DISTANCE_METRIC_COSINE = 2
VECTOR_DISTANCE_METRIC_DOT = 3
VECTOR_DISTANCE_METRIC_HAMMING = 4
message
cloud.proto:499Used in:
optional uint32 top_k = 1
message
cloud.proto:434Used in:
optional uint32 num_partitions = 1
num_partitions is deprecated. Use target_partition_num_rows instead TODO(RR-2798): Remove in 0.6
optional uint32 num_sub_vectors = 2
VectorDistanceMetric distance_metrics = 3
optional uint32 target_partition_num_rows = 4
Target size of the IVF partition in rows