Proto commits in rerun-io/rerun

These commits are when the Protocol Buffers files have changed: (only the last 100 relevant commits are shown)

2025-05-09

Remove deprecated `Scalar`, `SeriesLine`, `SeriesPoint` archetypes (#9892)

The documentation is generated from this commit.

2025-05-08

create partition manifests now requires storage url and partition type (#9902)

2025-05-06

fix other mentions of removed archetypes

2025-05-05

protos: support parallel partition manifest creation during dataset registration (#9852) Protos change to support parallel partition manifests creation in https://github.com/rerun-io/dataplatform/pull/663 --------- Signed-off-by: Andrea Reale <andrea@rerun.io>

2025-05-02

update dataset registration proto to support It now supports parallel partition manifest creation. Signed-off-by: Andrea Reale <andrea@rerun.io>

2025-04-30

update dataset registration proto to support It now supports parallel partition manifest creation. Signed-off-by: Andrea Reale <andrea@rerun.io>

wip

2025-04-28

Remove dead proto objects and their extensions (#9823) Remove a bunch of dead proto objects that had me very confused.

remove dead proto objects and their extensions

wip

2025-04-22

Remove `FetchPartition` (#9773) This was a temporary hack while waiting for `GetChunks`. I can confirm connecting to redap and watching recordings still work. * Sibling: https://github.com/rerun-io/dataplatform/pull/609

2025-04-14

dataplatform tasks: rename task related methods (#9730) When exposed from the dataplatform forntend, it became clare that methods like `Query` would not work well :) Renaming to clearer `QueryTasks` etc.. ### Related * https://github.com/rerun-io/dataplatform/pull/583 Signed-off-by: Andrea Reale <andrea@rerun.io>

re_protos: use async registration by default (protos) (#9537) ### Related * Part of [dataplatform#527](https://github.com/issues/assigned?issue=rerun-io%7Cdataplatform%7C527) * Related [dataplatform PR](https://github.com/rerun-io/dataplatform/pull/543) <!-- Include links to any related issues/PRs in a bulleted list, for example: * Closes #1234 * Part of #1337 --> ### What Make async dataset and partition registrations the default registration method for redap. Keep supporting synchronous registration via the ManifestRegistry direct endpoint only. * **This is a breaking change of the redap API** Signed-off-by: Andrea Reale <andrea@rerun.io>

2025-04-11

re_protos: expose Tasks endpoints via the redap_frontend (#9578) ### Related * Depends on: https://github.com/rerun-io/rerun/pull/9572 * Part of: https://github.com/rerun-io/dataplatform/issues/545 * Proto definition for: https://github.com/rerun-io/dataplatform/pull/574 ### What Expose the Task Service endpoint via the frontend. Signed-off-by: Andrea Reale <andrea@rerun.io>

redap_protos: add endpoint for the tasks service to blocking-query tasks (#9572) ### Related * Part of https://github.com/rerun-io/dataplatform/issues/545 * Follows https://github.com/rerun-io/dataplatform/pull/560 * Protos definition for: https://github.com/rerun-io/dataplatform/pull/570 ### What We are adding a gRPC endpoint for the internal TasksService that allows to block until a task is completed and then return its status. Signed-off-by: Andrea Reale <andrea@rerun.io>

2025-04-07

Implement `send_table` in `re_grpc_server` and `re_viewer` (#9510) ### Related * Pulled out from #9456 ### What The server-side and viewer logic for `send_table`. Also contains, the required changes for the notebook. --------- Co-authored-by: Emil Ernerfeldt <emil.ernerfeldt@gmail.com>

Merge branch 'main' into grtlr/send-table-grpc-server

redap manifest registry: add endpoints for asynchronous dataset/partitions registration (#9487) ### Related * [dataplatform/#528](https://github.com/rerun-io/dataplatform/pull/528) ### What Introduces a new `ManifestRegistry` endpoint called `RegisterWithDatasetAsync`, which - surprise- is the async version of `RegisterWithDataset`. It also introduces new methods for the `Tasks` service to query the output of a task once completed. **Recommended reviewing commit-by-commit** <!-- Make sure the PR title and labels are set to maximize their usefulness for the CHANGELOG, and our `git log`. If you have noticed any breaking changes, include them in the migration guide. We track various metrics at <https://build.rerun.io>. For maintainers: * To run all checks from `main`, comment on the PR with `@rerun-bot full-check`. * To deploy documentation changes immediately after merging this PR, add the `deploy docs` label. --> --------- Signed-off-by: Andrea Reale <andrea@rerun.io>

lints

Initial changes to viewer and `re_grpc_server` Pulled out from #9456.

2025-04-04

move `TableId` to `common`

Initial gRPC proposal

Feedback from @jprochazk

move `TableId` to `common`

Initial gRPC proposal

Feedback from @jprochazk

2025-04-03

add entity paths to Frontend definitions (#9481) Forgot to do it in https://github.com/rerun-io/rerun/pull/9480

add entity paths as param to GetChunksRequest (#9480) Title.

Impl `QueryDataset` (#9475) Nothing fancy here, see the sibling. * https://github.com/rerun-io/dataplatform/pull/515

2025-04-02

Minimal viable case for table provider via flight responses with current grpc service. This doesn't yet use flight sql.

Preparing for re_protos for dataset queries (#9462) Just clean up all the type shenanigans, add extensions, kill duplicate objects, etc. * https://github.com/rerun-io/dataplatform/pull/509

Initial commit for flight protos

2025-04-01

Partition manifests basics (#9434) Basic definitions to get the ball rolling on the new partition manifests. * For the interesting stuff, see https://github.com/rerun-io/dataplatform/pull/487

gRPC spec update: use Schema for schema responses (#9443)

Align all RPCs with latest spec (#9438) Title. * https://github.com/rerun-io/rerun/pull/9438

2025-03-31

Add new gRPC definitions related to Tables (#9416) ### What This is a very minimal generic Table API. It gives us the schema and lets us scan the table.

Remove legacy API protos and their use, including the legacy Python API (#9423) ### Related - Sibling: https://github.com/rerun-io/dataplatform/pull/480 ### What Remove the legacy proto and the legacy server Python wrapper. This means that `PyRecordingHandle` can now only by a local recording (which will have to be fixed with he new api anyways, see https://github.com/rerun-io/rerun/issues/9139)

2025-03-28

gRPC spec update: remove ListPartitionIndexes as we have ListPartitions (#9420) ListPartitions has everything we wanted from ListPartitionIndexes and more.

2025-03-27

Creating a dataset should not require a full `EntryDetails` (#9403) Title. * Sibling: https://github.com/rerun-io/dataplatform/pull/461

2025-03-26

Proto extension industrialization template/demo (#9383) What: * New extensions for protobuf helper objects: * `re_protos::common::v1alpha1::ext` * `re_protos::catalog::v1alpha1::ext` * Basic integration with `jiff` and `pyo3::Error` * New `EntryId` type to prevent the naked `Tuid`: * Makes use of the new extensions to have a decent DX The proposed design is very simple: * Implement all your native helper objects in `re_protos::*::ext` with all the type conversion / parsing / etc logic * Do not ever under any circumstance leak the raw protobuf objects beyond service/client trait definitions * Profit as your business code becomes _much_ nicer to work with I've ported a non-trivial chunk of the new catalog APIs to that design, so as to have a decent enough sample size. * https://github.com/rerun-io/dataplatform/pull/459 --------- Co-authored-by: Antoine Beyeler <49431240+abey79@users.noreply.github.com>

small gRPC spec update: make CreatePartitionIndexes response return dataframe (#9389)

gRPC spec update: expose FetchPartition on the frontend (#9386)

Rename `EntryType` to `EntryKind` (#9378) ### Related - Part of #9360 - Part of #9350 - Sibling https://github.com/rerun-io/dataplatform/pull/455 ### What Title. "Type" is annoying because `type` is usually a reserved keyword. Note: intentionally breaking grpc compatibility

Fix `Tuid` lacking explicit presence bits (#9379) Fix this before things get completely out of hand. * https://github.com/rerun-io/dataplatform/pull/454

fetch chunk manifest for a Dataset (#9376) ### What Add a an API that enables fetching chunk manifest data for a particular Dataset index. This won't be a publicly exposed API, but it is very useful for developers (both for CLI tools and testing).

2025-03-25

Change `DeleteDatasetEntry` to `DeleteEntry` in both the protos and the Python SDK (#9364) ### Related - Fixes #9344 - Sibling https://github.com/rerun-io/dataplatform/pull/444 ### What All entries should be deletable, so it was a mistake to make it specific to dataset in the first place.

`IfDuplicateBehavior` belongs in `common` (#9372) Title.

redap_tasks: introduce grpc definitions (#9359) ### Related * Part of [redap #276 ](https://github.com/rerun-io/dataplatform/issues/276) <!-- Include links to any related issues/PRs in a bulleted list, for example: * Closes #1234 * Part of #1337 --> ### What This PR introduces gRPC definitions for the service that will enable interaction with redap persistent tasks. See [redap #276 ](https://github.com/rerun-io/dataplatform/issues/276).

introduce the `Frontend` service (#9365) Introduces the `Frontend` service, which is the public gRPC service for the entire platform. It does very little beyond basic auth and proxying at the moment, except for one very important thing: it resolves dataset IDs into physical dataset handles for downstream services that don't have this information. It also acts as the last encapsulation barrier: all the services downstream are effectively private and thus have less sensitive to breaking changes. On the DP side, this service is implemented by the `redap_frontend` crate, which we expect will do a lot more in the future (e.g. auth, tls termination, rerun-specific load-balancing, even support other protocols such as Arrow Flight or raw HTTP). * Implementation: https://github.com/rerun-io/dataplatform/pull/445

2025-03-20

Remove `re_log_types::Time` (#9293) ### Related * ~Is waiting for https://github.com/rerun-io/rerun/pull/9249 for the final purge~ * Follows https://github.com/rerun-io/rerun/pull/9292 * Part of https://github.com/rerun-io/rerun/issues/8635 * Sibling PR: https://github.com/rerun-io/dataplatform/pull/428 ### What `struct Time` has been replaced by `struct Duration` and `struct Timestamp`

manifest registry service spec update: make partition id a string type (#9324) ### What Partition ids can be user defined fields (episode id, recording id, etc). For simplicity we now represents in the manifest registry service with a simple string type. What actual field and type defines the partition id will most likely be defined as part of creating the ``Dataset``. This will also simplify the initial integration with ``rrd`` and the ``StoreId::id()``. We might revisit how we represent and deal with the partition ids e2e, but it will be simpler to do that as concrete cases and client integration comes along.

Prevent automatic opening of views for archetypes in `RecordingProperties` (#9315) ### Related * Closes #9313 ### What This PR adds `- /__partition_properties/**` to the filter list unless overwritten. This unifies our dataframe APIs again and is a first step to logging arbitrary data to properties. ### Test To test, it's probably best to look at one of the `recording_properties.{cpp|py|rs}` snippets of the follow up PR: * #9316 Here is a demo of the UI interactions: https://github.com/user-attachments/assets/6158b77b-6200-453c-b5e8-80b9a854d25c

2025-03-19

make partition ids string instead of Tuid

Initial gRPC protos for the new catalog API: dataset CRUD (#9299) ### What This PR introduces the initial set of gRPC protos for the new catalog API with a focus on dataset CRUD. In a nutshell: ```proto service CatalogService { rpc FindEntries(FindEntriesRequest) returns (FindEntriesResponse) {} rpc CreateDatasetEntry(CreateDatasetEntryRequest) returns (CreateDatasetEntryResponse) {} rpc ReadDatasetEntry(ReadDatasetEntryRequest) returns (ReadDatasetEntryResponse) {} rpc DeleteDatasetEntry(DeleteDatasetEntryRequest) returns (DeleteDatasetEntryResponse) {} } ```

2025-03-18

manifest registry service - small fixes (#9309) ### What * make new common DataframePart encode-able * add on_duplicate to partitions registration req

lint

small fixes for the new manifest registry service

2025-03-17

Recording properties for naming recordings and adding information (#9249) ### Related * Supersedes #9176 * Closes #6264 * Part of #8776 Corresponding project issue: * #9215 ### What This is the first step of the project issue above and removes part of `StoreInfo`. > [!IMPORTANT] > Older `.rrd` will still load, but might show `<unknown>` as recording name. <img width="334" alt="image" src="https://github.com/user-attachments/assets/13726b9e-9097-4199-87ef-cb8e71d5898b" /> ### TODOs * [x] Add `set_name` method to all SDKs. * [x] pass full checks ### Open questions * [x] Naming: `RecordingProperties` vs `Properties`? * [x] Should all components be optional to better support `Dataframe` and the likes? * [x] Which fields should be mandatory? From a viewer perspective, should we allow omitting `RecordingProperties` entirely? Again, an example for this is `Dataframe`, where `started` does not make any sense. --------- Co-authored-by: Andreas Reich <andreas@rerun.io> Co-authored-by: Andreas Reich <r_andreas2@web.de>

2025-03-14

Remove `EntryKey` in favor using Tuid directly

Merge branch 'main' into antoine/new-catalog-proto # Conflicts: # crates/store/re_protos/proto/rerun/v1alpha1/common.proto # crates/store/re_protos/src/v1alpha1/rerun.common.v1alpha1.rs

Manifest registry service spec update: make few more calls stream back the response (#9286) ### What Quick follow up to * https://github.com/rerun-io/rerun/pull/9273. I forgot to make few endpoints stream back the responses (noticed as soon as I started stubbing server side), hence fixing that here. I initially made all remaining APIs stream back responses, but then I realized that create/register/unregister endpoints should actually return a handle. So for now making only the ones I obviously missed (listing).

gRPC spec update: introduce Manifest registry service (#9273) ### What We're introducing new gRPC service - ManifestRegistryService. This change is mostly aligned with the current proposal for the Dataset APIs, there is simply few things missing (describe dataset, stream data in), that we will follow up on. Co-authored-by: Clement Rey <cr.rey.clement@gmail.com>

2025-03-13

Make `optional` fields with primitive types

Remove EntryId in favor of using Tuid directly.

Updated/created EntryId (now a TUID) and DatasetHandle to `common.proto`, this is the only shared message with the ManifestRegistryService (for now).

2025-03-12

Use EntryId type in EntryKey

Use EntryId type in filter

The field-name type is annoying to work with, so use entry_type instead

Pass pb-lints

WIP protos for the new catalog APIs

WIP

2025-03-11

Batch registration of recordings (#9234) All the necessary stuff to get batch registration up and running. Most of the changes is just new arrow utils for working with record batches. * `pixi run rerun-release rerun://clement.redap.rerun.io` * DP: https://github.com/rerun-io/dataplatform/pull/358

2025-03-10

Stop inspecting `is_official_example` in `StoreInfo` (#9247) ### Related * Closes #9189. ### What Title. _This change should be non-breaking._

2025-03-06

show me your best breakage

Protobuf tooling & associated breaking changes (#9202) Title. * :warning: Extremely conflict prone -- to be merged quickly, at the same time as https://github.com/rerun-io/dataplatform/pull/335 --- Checks: * [x] Codegen is out of sync (we've never implemented it...) * [x] Modified Proto, stale Rust * [x] Stale Proto, modified Rust * [x] Protobuf definitions and snapshot are out of sync * [x] Modified Proto, stale snapshot * [x] Stale Proto, modified snapshot * [x] Breaking change detector is working * [x] Lint detector is working * [x] Formatting detector is working

2025-03-05

gRPC spec update: Fetch range of chunks (#9178) ### What With the introduction of manifest, ``GetChunkIds`` is no longer needed, hence we get rid of it. We introduce a new call for fetching range of chunks that basically merged query manifest + get chunks into a single call. In order to support fetching chunk ranges directly from disk, added few helper methods for rrd decoder.

stash again

2025-02-28

add catalog entry property to remaining APIs (#9158) ### What In order to get rid of "default collection" concept from the ReDap server, making remaining APIs catalog entry aware. There's plenty of renaming of actual calls ahead of us, but this is unrelated effort, for now the goal is to make everything specific collection aware.

2025-02-25

Add support for collection enumeration to the remote store protocol (#9121) ### What The remote store protocol is being extended to support enumeration of collections. This PR also aims to prototype a more scalable organization of proto files: * A catalog model is centrally defined for reuse by related request and response messages * When a catalog model is provided in a request, it is repeated in the response, providing access to its finalized field values such as an assigned id or created at timestamp

2025-02-24

Make remaining data fetching APIs catalog entry aware (#9118) Title.

Add entity paths to manifest APIs (#9117) * See https://github.com/rerun-io/dataplatform/pull/293

2025-02-20

Fix `QueryManifest`'s missing catalog entry parameter (#9099)

gRPC spec update: make resource register / unregister API calls be catalog entry aware (#9095)

2025-02-19

RRD manifests bootstrap (#9053)

2025-02-13

gRPC spec update: Add support for fetching temporal range of recordings (#9010) ### What Adding support for fetching chunks that belong to specific range of the recording. This consists of the 2 APIs calls: * fetching chunk ids for a given range * fetching chunks for the given chunk ids (which we might want to leverage separately in the future) As there is a limitation for the client side gRPC streaming on the web (see [here](https://github.com/rerun-io/rerun/issues/8877#issuecomment-2652986044)), we've decided to send a batch of chunk ids to the server. To make the APIs somewhat symmetrical, the 1st API call will stream batches of chunk ids. Closes * https://github.com/rerun-io/rerun/issues/8877

2025-02-11

Add reindex capability to remote store protocol (#8985) ### Related Part of https://github.com/rerun-io/dataplatform/issues/156. ### What We want to be able to manually request the reindexing of an existing index. This could be used after a recording is added, which leaves the index in a stale state.

2025-02-10

Migrate SDK comms to gRPC (#8838) Co-authored-by: Jeremy Leibs <jeremy@rerun.io>

2025-02-07

Splitting service and more docs

DRAFT: Propose new uniform gRPC service for Catalog / StorageNode

2025-02-05

Workspace configuration to improve working with protobufs in vscode (#8920) ### What These settings give us proper syntax validation on imports and sane format-on-save behavior.

2025-02-04

Rename the index APIs from `query` to `search` (#8901) Cleaning up the APIs to replace query with search and. Also replace collection w/ entry where appropriate.

2025-01-30

Update to latest egui (#8865) ## Changes * When painting a rect stroke, you must chose if the stroke is inside, outside, or on the rectangle. * TextEdit and ColorButton have been shrunk to fit inside their widget rects ## TODO * [x] Check that the choices of `StrokeKind` makes sense everywhere

initial gRPC spec changes for supporting index creation and querying (#8829) ### What Expose creating and querying an index over an entire collection of data. We add support for: * vector index * inverted index for full text search * btree index Query data is always represented with a record batch with just 1 column that carries the query data.

2025-01-28

Use Arrow IPC to encode the column schema (#8821) ### Related * Part of https://github.com/rerun-io/rerun/issues/8744 * Sibling PR: https://github.com/rerun-io/dataplatform/pull/154

2025-01-21

Use serde to encode arrow datatype (#8747) ### Related * https://github.com/rerun-io/rerun/pull/8719 Apparently `arrow::Datatype::to_string/parse` doesn't roundtrip, despite claiming so. Let's hope serde/json works better. We probably want to replace the whole of `crates/store/re_protos/proto/rerun/v0/common.proto` with just a IPC-encoded RecordBath at some point, but not today.

2025-01-17

Even less arrow2 (#8719) * Part of #3741 Just one small piece at a time

2025-01-14

gRPC spec update: define recording schema protobuf types and expose the schema via new endpoint (#8651) ### What Define column descriptors in the protobuf spec and the relevant type conversions. Add new endpoint for fetching the schema that returns a list of column descriptors. Also updated the example script to use the next endpoint and print the schema.

2025-01-09

Implement `re_grpc_server` (#8622)

2025-01-08

Add `rpc` calls to unregister recordings from catalog (#8566) ### What This adds two simple `rpc` calls, `UnregisterRecording` and `UnregisterAllRecordings`, to the storage node interface.

2024-12-18

Update the remote APIs to take and send a Table (#8521) This allows the update APIs to now support multi-recording updates, and generally gives users more direct control over things like the column metadata, etc.