Proto commits in chroma-core/chroma

These commits are when the Protocol Buffers files have changed: (only the last 100 relevant commits are shown)

2026-07-08

[ENH](work-queue): require compaction offset

2026-07-07

[ENH](logservice): add conditional push and insert offset to proto (#7317) ## Description of changes Extend PushLogsRequest and PushLogsResponse to support conditional log appends: - Add optional PushLogsCondition with observed_log_offset and read_ids - Add optional first_inserted_record_offset to PushLogsResponse Update all PushLogsRequest construction sites and the log-service response to populate the new fields with None defaults. ## Test plan CI ## Migration plan Backwards compatible ## Observability plan N/A ## Documentation Changes N/A Co-authored-by: AI

The documentation is generated from this commit.

[CHORE](sysdb): simplify finish API

[BUG](work-queue): expose compaction frontier Return the queue compaction frontier from GetWork while completion_offset is no longer persisted in the WQS state file. Keep the field optional on the wire so older consumers remain compatible during rollout.

2026-07-06

[ENH](work-queue): require compaction offset

[CHORE](sysdb): simplify finish API

Revert "[BUG](fn-consumer): Read offsets from sysdb" This reverts commit f4ea0ff9c6e4c494a6ddf912cf985be639ade204.

[BUG](fn-consumer): Read offsets from sysdb

[ENH](work-queue): require compaction offset

[BUG](work-queue): expose compaction frontier Return the queue compaction frontier from GetWork while completion_offset is no longer persisted in the WQS state file. Keep the field optional on the wire so older consumers remain compatible during rollout.

[CHORE](sysdb): simplify finish API

[BUG](work-queue): expose compaction frontier Return the queue compaction frontier from GetWork while completion_offset is no longer persisted in the WQS state file. Keep the field optional on the wire so older consumers remain compatible during rollout.

Revert "[BUG](fn-consumer): Read offsets from sysdb" This reverts commit f4ea0ff9c6e4c494a6ddf912cf985be639ade204.

[BUG](fn-consumer): Read offsets from sysdb

[ENH](work-queue): require compaction offset

2026-07-02

[ENH](work-queue): add optional compaction offset (#7354) ## Description of changes Part 1 of a stack of changes to persist the input collection compaction offset instead of the function completion offset in the WQS file. This diff starts off by requiring this compaction offset to be sent in the PushWork API. - Improvements & Bug fixes - - New functionality - ... ## Test plan _How are these changes tested?_ - [ ] Tests pass locally with `pytest` for python, `yarn test` for js, `cargo test` for rust ## Migration plan _Are there any migrations, or any forwards/backwards compatibility changes needed in order to make sure this change deploys reliably?_ ## Observability plan _What is the plan to instrument and monitor this change?_ ## Documentation Changes _Are all docstrings for user-facing APIs updated if required? Do we need to make documentation changes in the_ [_docs section](https://github.com/chroma-core/chroma/tree/main/docs/docs.trychroma.com)?_

[CHORE](sysdb): simplify finish API

[BUG](fn-consumer): Read offsets from sysdb

[BUG](work-queue): expose compaction frontier Return the queue compaction frontier from GetWork while completion_offset is no longer persisted in the WQS state file. Keep the field optional on the wire so older consumers remain compatible during rollout.

Revert "[BUG](fn-consumer): Read offsets from sysdb" This reverts commit f4ea0ff9c6e4c494a6ddf912cf985be639ade204.

[ENH](work-queue): require compaction offset

[CHORE](sysdb): simplify finish API

[BUG](fn-consumer): Read offsets from sysdb

2026-07-01

[ENH](work-queue): require compaction offset

2026-06-30

document new fields

[CHORE](sysdb): simplify finish API

[ENH](work-queue): require compaction offset

2026-06-29

[CHORE](sysdb): simplify finish API

[ENH](work-queue): require compaction offset

[ENH](work-queue): add optional compaction offset

[CHORE](sysdb): simplify finish API

[ENH](work-queue): require compaction offset

[ENH](work-queue): add optional compaction offset

2026-06-28

[ENH](logservice): add conditional push and insert offset to proto Extend PushLogsRequest and PushLogsResponse to support conditional log appends: - Add optional PushLogsCondition with observed_log_offset and read_ids - Add optional first_inserted_record_offset to PushLogsResponse Update all PushLogsRequest construction sites and the log-service response to populate the new fields with None defaults. Co-authored-by: AI

2026-06-25

[ENH](logservice): add conditional push and insert offset to proto Extend PushLogsRequest and PushLogsResponse to support conditional log appends: - Add optional PushLogsCondition with observed_log_offset and read_ids - Add optional first_inserted_record_offset to PushLogsResponse Update all PushLogsRequest construction sites and the log-service response to populate the new fields with None defaults. Co-authored-by: AI

[ENH](logservice): add conditional push and insert offset to proto Extend PushLogsRequest and PushLogsResponse to support conditional log appends: - Add optional PushLogsCondition with observed_log_offset and read_ids - Add optional first_inserted_record_offset to PushLogsResponse Update all PushLogsRequest construction sites and the log-service response to populate the new fields with None defaults. Co-authored-by: AI

[ENH](logservice): add conditional push and insert offset to proto Extend PushLogsRequest and PushLogsResponse to support conditional log appends: - Add optional PushLogsCondition with observed_log_offset and read_ids - Add optional first_inserted_record_offset to PushLogsResponse Update all PushLogsRequest construction sites and the log-service response to populate the new fields with None defaults. Co-authored-by: AI

[ENH](logservice): add conditional push and insert offset to proto Extend PushLogsRequest and PushLogsResponse to support conditional log appends: - Add optional PushLogsCondition with observed_log_offset and read_ids - Add optional first_inserted_record_offset to PushLogsResponse Update all PushLogsRequest construction sites and the log-service response to populate the new fields with None defaults. Co-authored-by: AI

2026-06-04

[ENH]: Finish_work and output collection flush are atomic

[ENH]: Add scaffolding for functions with many inputs in fn consumer (#7143) ## Description of changes This PR upgrades the fn consumer pipeline to support functions with **multiple input collections**. Previously, `AttachedFunctionExecutor::execute` received a single `Chunk` of records; now it receives a `Vec<Chunk>` — one chunk per input collection. The orchestration layer is restructured to carry batches of `(collection_id, materialized_logs)` pairs through the pipeline, and the fn consumer manager groups work items by function before dispatching them as a single batch. The key structural changes are: 1. `AttachedFunctionExecutor::execute` signature: `Chunk<…>` → `Vec<Chunk<…>>` 2. New `FunctionExecutionBatch` / `FunctionExecutionProgress` / `FunctionContext` types extracted into a new `function_execution` module 3. `ExecuteAttachedFunctionInput` gains `input_batches: Vec<ExecuteAttachedFunctionBatchInput>` replacing the flat `materialized_logs` + `input_record_segment` 4. `FinishAsyncWorkInput` now carries `Vec<FinishAsyncWorkItem>` instead of a single offset 5. `FnConsumerManager` groups work items by function ID and dispatches them as a batch via `FunctionExecutionContext::run` - Improvements & Bug fixes - ... - New functionality - ... ## Test plan _How are these changes tested?_ - [ ] Tests pass locally with `pytest` for python, `yarn test` for js, `cargo test` for rust ## Migration plan _Are there any migrations, or any forwards/backwards compatibility changes needed in order to make sure this change deploys reliably?_ ## Observability plan _What is the plan to instrument and monitor this change?_ ## Documentation Changes _Are all docstrings for user-facing APIs updated if required? Do we need to make documentation changes in the_ [_docs section](https://github.com/chroma-core/chroma/tree/main/docs/docs.trychroma.com)?_

[CLN](worker): Box function execution fetch future

[ENH]: Add route to allow multiple inputs on an async function (#7142) ## Description of changes This PR adds a new `add_input` route that lets callers associate additional input collections with an existing async attached function. The key data model insight: the `attached_functions` table uses a composite primary key of `(id, input_collection_id)`, so one logical attached function can have multiple rows — one per input collection. Only async functions support multiple inputs. The change spans the full stack: protobuf → Go sysdb coordinator → Rust sysdb client → Rust frontend server → Python API client. - Improvements & Bug fixes - ... - New functionality - ... ## Test plan _How are these changes tested?_ - [ ] Tests pass locally with `pytest` for python, `yarn test` for js, `cargo test` for rust ## Migration plan _Are there any migrations, or any forwards/backwards compatibility changes needed in order to make sure this change deploys reliably?_ ## Observability plan _What is the plan to instrument and monitor this change?_ ## Documentation Changes _Are all docstrings for user-facing APIs updated if required? Do we need to make documentation changes in the_ [_docs section](https://github.com/chroma-core/chroma/tree/main/docs/docs.trychroma.com)?_

[TEST](coordinator): Fix attach function collection mocks

2026-05-27

[BUG]: WQS repairs on bootup (#7126) ## Description of changes This PR renames `AreInvocationsDone` (which returned `[]bool`) to `CheckInvocationStatus` (which returns a three-state enum). The key semantic change is that the old binary done/not-done is replaced with three states: - `NOT_DONE` (0) — default, work still in progress - `DONE` (1) — completed (soft/hard deleted, or offset advanced with no pending heap entry) - `NEEDS_REPAIR` (2) — offset advanced but `heap_entry_pending=true`, meaning the WQS crashed mid-repair The new `NEEDS_REPAIR` state enables the work queue manager to automatically repair stale entries on bootup, which is the core bug fix this PR addresses. In the proto we replaced the RPC for `AreInvocationsDone` with ``CheckInvocationStatus` this is not backwards compatible but that is fine as there is not production use for the former RPC yet. - Improvements & Bug fixes - ... - New functionality - ... ## Test plan _How are these changes tested?_ - [ ] Tests pass locally with `pytest` for python, `yarn test` for js, `cargo test` for rust ## Migration plan _Are there any migrations, or any forwards/backwards compatibility changes needed in order to make sure this change deploys reliably?_ ## Observability plan _What is the plan to instrument and monitor this change?_ ## Documentation Changes _Are all docstrings for user-facing APIs updated if required? Do we need to make documentation changes in the_ [_docs section](https://github.com/chroma-core/chroma/tree/main/docs/docs.trychroma.com)?_

[BUG]: WQS repairs on bootup

2026-05-22

[BUG]: Fn consumer manager keys its job scheduler on output collection id (#7117) ## Description of changes The `FnConsumerManager` was keying its "in-progress" job scheduler on `(fn_id, input_coll_id)` — a `FnJobKey`. The bug: multiple different attached functions could share the same output collection, and the old key didn't prevent two jobs from writing to the same output collection concurrently. The fix re-keys the in-progress map on `output_coll_id` (`CollectionUuid`), which is the actual resource that must be serialized. To support this, `GetAttachedFunctions` gains a batch `ids` parameter so the manager can look up all output collection IDs in a single sysdb call rather than one per work item. - Improvements & Bug fixes - ... - New functionality - ... ## Test plan _How are these changes tested?_ - [ ] Tests pass locally with `pytest` for python, `yarn test` for js, `cargo test` for rust ## Migration plan _Are there any migrations, or any forwards/backwards compatibility changes needed in order to make sure this change deploys reliably?_ ## Observability plan _What is the plan to instrument and monitor this change?_ ## Documentation Changes _Are all docstrings for user-facing APIs updated if required? Do we need to make documentation changes in the_ [_docs section](https://github.com/chroma-core/chroma/tree/main/docs/docs.trychroma.com)?_

[BUG]: Fn consumer manager keys its job scheduler on output collection id

2026-05-14

[ENH] implement stratified sampling endpoint for collections Add a new sample API that randomly selects records from a collection using stratified sampling over SPANN index strata. Records can be filtered by ID, metadata, or document predicates before sampling. The endpoint accepts a limit and optional seed for reproducibility. Full-stack implementation across all layers: - API types and request/response models (Rust, Python, JS/TS) - Protobuf IDL for distributed query execution - Frontend server routing, metrics, and executor integration - Sample operator with SPANN-aware stratified selection - Sample orchestrator with filter, projection, and shard fan-out - SPANN index reader support for head-count enumeration - Python and JS client bindings Co-authored-by: AI

2026-05-13

[ENH]: Function consumer server

[ENH]: Function consumer server

[ENH]: Function consumer server

2026-05-11

[ENH]: Function consumer server

2026-05-08

[ENH]: Add sysdb endpoint to check if invocations are done (#7016) ## Description of changes This PR adds a new `AreInvocationsDone` endpoint to sysdb that lets callers batch-check whether attached function invocations have completed. The check works by comparing `completion_offset > invocation_id AND heap_repair_needed` in the database. - Improvements & Bug fixes - ... - New functionality - New SysDB endpoint as described. ## Test plan _How are these changes tested?_ Go tests added to test the filtering logic of this endpoint. - [ ] Tests pass locally with `pytest` for python, `yarn test` for js, `cargo test` for rust ## Migration plan _Are there any migrations, or any forwards/backwards compatibility changes needed in order to make sure this change deploys reliably?_ ## Observability plan _What is the plan to instrument and monitor this change?_ ## Documentation Changes _Are all docstrings for user-facing APIs updated if required? Do we need to make documentation changes in the_ [_docs section](https://github.com/chroma-core/chroma/tree/main/docs/docs.trychroma.com)?

2026-05-07

[ENH]: Add sysdb endpoint to check if invocations are done

[ENH]: Sysdb calls for finishing function invocation (#7015) ## Description of changes This PR adds two new sysdb RPC endpoints for finishing async attached function invocations: 1. **`TryFinishAsyncAttachedFunctionInvocation`** — records a new completion offset for an async function. If the offset is behind the collection's current log position, it signals that a repair is needed (returns `NeedsRepair`); otherwise it returns `Success`. 2. **`FinalizeAsyncAttachedFunctionRepair`** — called after repair is complete to flip `can_gc` back to `true`. A new `can_gc` column is added to the `attached_functions` table to track whether the function's output is safe to garbage-collect. - Improvements & Bug fixes - N/A - New functionality - Aforementioned new endpoints ## Test plan _How are these changes tested?_ Go tests have been added to test the repair and non-repair cases of these new methods. - [ ] Tests pass locally with `pytest` for python, `yarn test` for js, `cargo test` for rust ## Migration plan _Are there any migrations, or any forwards/backwards compatibility changes needed in order to make sure this change deploys reliably?_ A migration has been added to add a 'can_gc' column to the `attached_functions` table. ## Observability plan _What is the plan to instrument and monitor this change?_ ## Documentation Changes _Are all docstrings for user-facing APIs updated if required? Do we need to make documentation changes in the_ [_docs section](https://github.com/chroma-core/chroma/tree/main/docs/docs.trychroma.com)?_

[ENH]: Add work queue proto definitions and core types (#7020) ## Description of changes This is the base PR for the work queue service stack. It introduces two things: 1. A protobuf contract (`workqueue.proto`) defining the gRPC API surface 2. Rust core types (`WorkQueueRecord`, `WorkQueueError`, `FinishResult`) that the rest of the stack builds on Nothing is wired up in this change. This is scaffolding for later PRs. - Improvements & Bug fixes - N/A - New functionality - N/A ## Test plan N/A - [ ] Tests pass locally with `pytest` for python, `yarn test` for js, `cargo test` for rust ## Migration plan _Are there any migrations, or any forwards/backwards compatibility changes needed in order to make sure this change deploys reliably?_ ## Observability plan _What is the plan to instrument and monitor this change?_ ## Documentation Changes _Are all docstrings for user-facing APIs updated if required? Do we need to make documentation changes in the_ [_docs section](https://github.com/chroma-core/chroma/tree/main/docs/docs.trychroma.com)?_

[ENH]: Sysdb call for function invocation

[ENH]: Add async mode to functions (#7014) ## Description of changes This PR adds an `is_async` field to the `Function` model, propagating it through the full stack: database schema → Go model → protobuf → Rust types → orchestrator logic. The core behavioral change is in the orchestrator: async functions skip updating the `completion_offset` during processing, because they'll be handled through a separate queue mechanism in later PRs. - Improvements & Bug fixes - N/A - New functionality - Can define a function to be async, attached functions runs of such functions will do the aforementioned async behavior. ## Test plan _How are these changes tested?_ Existing function go tests have been adjusted for this new field. - [x] Tests pass locally with `pytest` for python, `yarn test` for js, `cargo test` for rust ## Migration plan _Are there any migrations, or any forwards/backwards compatibility changes needed in order to make sure this change deploys reliably?_ A new `is_async` boolean column is added to `attached_functions` with `NOT NULL DEFAULT false`, ensuring backward compatibility — all existing rows default to synchronous behavior. ## Observability plan _What is the plan to instrument and monitor this change?_ ## Documentation Changes _Are all docstrings for user-facing APIs updated if required? Do we need to make documentation changes in the_ [_docs section](https://github.com/chroma-core/chroma/tree/main/docs/docs.trychroma.com)?_

2026-05-06

feat: Expose work queue service in Kubernetes - Add port 50054 to compaction service container ports - Create work-queue-service Service to expose the gRPC endpoint - Enable work queue via environment variable CHROMA_WORK_QUEUE__ENABLED - Remove separate work-queue-config.yaml (configuration via env vars)

[ENH]: Add async mode to functions

2026-05-05

feat: Expose work queue service in Kubernetes - Add port 50054 to compaction service container ports - Create work-queue-service Service to expose the gRPC endpoint - Enable work queue via environment variable CHROMA_WORK_QUEUE__ENABLED - Remove separate work-queue-config.yaml (configuration via env vars)

2026-04-30

[ENH]: Add metadata support to Database entity This adds optional metadata field to databases, following the pattern used for Collections. Users can now store arbitrary key-value metadata on databases for organizational and tagging purposes. Changes: - New database_metadata table in Go PostgreSQL and Rust SQLite - Proto updates to Database message and CreateDatabaseRequest - Go backend: dbmodel, DAO, catalog, and gRPC handlers - Rust backend: table definitions, sqlite.rs queries - Python client: AdminAPI, FastAPI endpoints - JavaScript client: AdminClient.createDatabase - Integration tests for database metadata https://claude.ai/code/session_019rHB7PwwuyJvyypxuJVCCs

2026-04-17

Cherry-pick with conflicts: fdcd2165154b6bd6394983c49a9ba02cc71ff925

[ENH]: Enable rebuilds for sharded collections (#6916) ## Description of changes This change adds rebuild capabilities for sharded collections. A rebuild command can now specify a shard index it wants to rebuild. An unspecified one defaults to 0 (first shard). This shard's filepaths are set to empty during rebuild compaction, analogous to all of the segment's filepaths being set to empty before this change. A new struct packaging the rebuild command parameters called `RebuildInfo` is passed to `CompactionContext` now. This contains the desired shard index to be rebuilt. - Improvements & Bug fixes - ... - New functionality - ... ## Test plan Tests have been added in compact.rs. They test rebuilding various shards in the full and partial rebuild scenarios. - [ ] Tests pass locally with `pytest` for python, `yarn test` for js, `cargo test` for rust ## Migration plan _Are there any migrations, or any forwards/backwards compatibility changes needed in order to make sure this change deploys reliably?_ ## Observability plan _What is the plan to instrument and monitor this change?_ ## Documentation Changes _Are all docstrings for user-facing APIs updated if required? Do we need to make documentation changes in the_ [_docs section](https://github.com/chroma-core/chroma/tree/main/docs/docs.trychroma.com)?_

2026-04-16

[ENH]: Enable rebuilds for sharded collections

[HOTFIX] applying PR #6914 to release/2026-04-03 (#6920) This PR cherry-picks the commit f749f58f881830ad5ed4d728529b5d491656f825 onto release/2026-04-03. If there are unresolved conflicts, please resolve them manually. Co-authored-by: Macronova <60079945+Sicheng-Pan@users.noreply.github.com>

2026-04-15

[ENH] Add IndexAndBoundedWal read level (#6914) ## Description of changes Add a new `IndexAndBoundedWal` read level for collection search and count APIs. This level reads from the index plus up to a server-configured number of write-ahead log entries, providing a consistent prefix of the WAL with bounded query latency. Unlike `IndexAndWal` (which reads the entire WAL) or `IndexOnly` (which skips it entirely), `IndexAndBoundedWal` caps the log fetch via the existing `maximum_fetch_count` on `FetchLogOperator`. The limit is controlled by a new `bounded_wal_limit` config field on `QueryServiceConfig` (default 250). - New functionality - `INDEX_AND_BOUNDED_WAL` proto enum variant in `ReadLevel` - `IndexAndBoundedWal` Rust type with bidirectional proto conversions - `bounded_wal_limit` config field on `QueryServiceConfig` (default 250) - Bounded WAL fetch logic in `CountOrchestrator` and `KnnFilterOrchestrator` — clones `FetchLogOperator` with `maximum_fetch_count = Some(bounded_wal_limit)` ## Test plan - [x] Behavioral unit test (`test_read_level_semantics`) in `count.rs` — sets up 5 compacted + 10 uncompacted records and verifies: - `IndexAndWal` → 15 (all records) - `IndexOnly` → 5 (compacted only) - `IndexAndBoundedWal` with limit 3 → 8 (partial WAL) - `IndexAndBoundedWal` with limit 10 → 15 (full WAL) - `IndexAndBoundedWal` with limit 100 → 15 (limit exceeds WAL, reads all available)

[ENH] Add IndexAndBoundedWal read level (#6914) ## Description of changes Add a new `IndexAndBoundedWal` read level for collection search and count APIs. This level reads from the index plus up to a server-configured number of write-ahead log entries, providing a consistent prefix of the WAL with bounded query latency. Unlike `IndexAndWal` (which reads the entire WAL) or `IndexOnly` (which skips it entirely), `IndexAndBoundedWal` caps the log fetch via the existing `maximum_fetch_count` on `FetchLogOperator`. The limit is controlled by a new `bounded_wal_limit` config field on `QueryServiceConfig` (default 250). - New functionality - `INDEX_AND_BOUNDED_WAL` proto enum variant in `ReadLevel` - `IndexAndBoundedWal` Rust type with bidirectional proto conversions - `bounded_wal_limit` config field on `QueryServiceConfig` (default 250) - Bounded WAL fetch logic in `CountOrchestrator` and `KnnFilterOrchestrator` — clones `FetchLogOperator` with `maximum_fetch_count = Some(bounded_wal_limit)` ## Test plan - [x] Behavioral unit test (`test_read_level_semantics`) in `count.rs` — sets up 5 compacted + 10 uncompacted records and verifies: - `IndexAndWal` → 15 (all records) - `IndexOnly` → 5 (compacted only) - `IndexAndBoundedWal` with limit 3 → 8 (partial WAL) - `IndexAndBoundedWal` with limit 10 → 15 (full WAL) - `IndexAndBoundedWal` with limit 100 → 15 (limit exceeds WAL, reads all available)

[ENH](faults): add fault injection control plane (#6795) ## Description of changes Add the fault injection proto service and the new chroma-faults crate for in-process fault state management. Implement inject, list, and clear RPCs over a shared FaultRegistry, and wire log-service to build and expose the new tonic service while registering the shared Arc in the config registry. ## Test plan CI + further dev ## Migration plan N/A ## Observability plan N/A ## Documentation Changes N/A Co-authored-by: AI

2026-04-05

[ENH]: Pass down shard_index, num_shards and log_upper_bound_offset to query (#6810) ## Description of changes _Summarize the changes made by this PR._ - Improvements & Bug fixes - New functionality - Introduces new fields - shard_index, num_shards and log_upper_bound_offset in ScanProto and Scan types. - Currently FE just sets shard_index=0, num_shards=1 and log_upper_bound_offset=0. The query nodes don't use these fields - These fields will be used by worker in a future PR to execute logic for only its shard - Handled backwards and forward compatibility. If num_shards is set to 0 (i.e. absent) then query node sets it to 1 ## Test plan _How are these changes tested?_ - [x] Tests pass locally with `pytest` for python, `yarn test` for js, `cargo test` for rust ## Migration plan None ## Observability plan None ## Documentation Changes None

2026-04-03

add log offset upper bound

[ENH]: Pass down shard id and num shards to query

2026-03-30

[ENH] Add description field to collections Add an optional text description field to collections across the full stack: - Python types, proto definitions, and conversion functions - Go model, DB model, DAO, catalog, and gRPC layers - Postgres and SQLite migrations - FastAPI server endpoints and HTTP clients (sync + async) - Abstract API interfaces and SegmentAPI implementation - Collection model classes (sync + async) with modify() support https://claude.ai/code/session_01XmGiw5KPfuugfHgSbPL2pX

2026-03-17

[CLN]: Remove ListDeadJobs endpoint (#6633) ## Description of changes Compactors don't maintain an in-memory DLQ anymore. Remove the ListDeadJobs endpoint to reflect this - Improvements & Bug fixes - ... - New functionality - ... ## Test plan _How are these changes tested?_ - [ ] Tests pass locally with `pytest` for python, `yarn test` for js, `cargo test` for rust ## Migration plan _Are there any migrations, or any forwards/backwards compatibility changes needed in order to make sure this change deploys reliably?_ ## Observability plan _What is the plan to instrument and monitor this change?_ ## Documentation Changes _Are all docstrings for user-facing APIs updated if required? Do we need to make documentation changes in the_ [_docs section](https://github.com/chroma-core/chroma/tree/main/docs/docs.trychroma.com)?_

2026-03-14

[ENH]: MCMR batch_get_collection_soft_delete_status (#6422) ## Description of changes Implements the `batch_get_collection_soft_delete_status` endpoint for rust sysdb and adds a few debugging messages I found helpful when testing end to end GC functionality on MCMR. - Improvements & Bug fixes - ... - New functionality - ... ## Test plan Unit tests are added in spanner.rs - [ ] Tests pass locally with `pytest` for python, `yarn test` for js, `cargo test` for rust ## Migration plan _Are there any migrations, or any forwards/backwards compatibility changes needed in order to make sure this change deploys reliably?_ ## Observability plan _What is the plan to instrument and monitor this change?_ ## Documentation Changes _Are all docstrings for user-facing APIs updated if required? Do we need to make documentation changes in the_ [_docs section](https://github.com/chroma-core/chroma/tree/main/docs/docs.trychroma.com)?_

2026-03-13

[ENH]: MCMR batch_get_collection_soft_delete_status

2026-03-12

[ENH]: delete_collection_versions in MCMR (#6420) ## Description of changes This change implements delete_collection_version in rust sysdb. - Improvements & Bug fixes - ... - New functionality - ... ## Test plan _How are these changes tested?_ - [ ] Tests pass locally with `pytest` for python, `yarn test` for js, `cargo test` for rust ## Migration plan _Are there any migrations, or any forwards/backwards compatibility changes needed in order to make sure this change deploys reliably?_ ## Observability plan _What is the plan to instrument and monitor this change?_ ## Documentation Changes _Are all docstrings for user-facing APIs updated if required? Do we need to make documentation changes in the_ [_docs section](https://github.com/chroma-core/chroma/tree/main/docs/docs.trychroma.com)?_

[ENH]: delete_collection_versions in MCMR

[ENH]: mark_version_to_gc in MCMR (#6419) ## Description of changes This PR implements `mark_version_for_deletion` for rust sysdb to support MCMR GC. - Improvements & Bug fixes - ... - New functionality - ... ## Test plan _How are these changes tested?_ - [ ] Tests pass locally with `pytest` for python, `yarn test` for js, `cargo test` for rust ## Migration plan _Are there any migrations, or any forwards/backwards compatibility changes needed in order to make sure this change deploys reliably?_ ## Observability plan _What is the plan to instrument and monitor this change?_ ## Documentation Changes _Are all docstrings for user-facing APIs updated if required? Do we need to make documentation changes in the_ [_docs section](https://github.com/chroma-core/chroma/tree/main/docs/docs.trychroma.com)?_

2026-03-11

[ENH]: mark_version_to_gc in MCMR (WIP)

[CLN]: Remove ListDeadJobs from compaction client

[ENH]: batch_get_collection_version_file_paths for MCMR (#6417) ## Description of changes This change adds the `batch_get_collection_version_file_paths` endpoint in rust sysdb for garbage collection. - Improvements & Bug fixes - ... - New functionality - ... ## Test plan _How are these changes tested?_ - [ ] Tests pass locally with `pytest` for python, `yarn test` for js, `cargo test` for rust ## Migration plan _Are there any migrations, or any forwards/backwards compatibility changes needed in order to make sure this change deploys reliably?_ ## Observability plan _What is the plan to instrument and monitor this change?_ ## Documentation Changes _Are all docstrings for user-facing APIs updated if required? Do we need to make documentation changes in the_ [_docs section](https://github.com/chroma-core/chroma/tree/main/docs/docs.trychroma.com)?_

[ENH]: batch_get_collection_version_file_paths for MCMR

[ENH]: Get database name from get_collections_to_gc mcmr endpoint (#6392) ## Description of changes This change adds the get_collections_to_gc endpoint to spanner and has the garbage collector retrieve collections and database names from this endpoint. It edits the Go sysdb ListCollectionsToGC endpoint to also return a database name with each endpoint. This database name is saved in a GarbageCollectionOrchestrator's state. This is a breaking change on the Go sysdb protos but it is not a big deal as garbage collection might just error while this change is being deployed. - Improvements & Bug fixes - ... - New functionality - ... ## Test plan _How are these changes tested?_ - [ ] Tests pass locally with `pytest` for python, `yarn test` for js, `cargo test` for rust ## Migration plan _Are there any migrations, or any forwards/backwards compatibility changes needed in order to make sure this change deploys reliably?_ ## Observability plan _What is the plan to instrument and monitor this change?_ ## Documentation Changes _Are all docstrings for user-facing APIs updated if required? Do we need to make documentation changes in the_ [_docs section](https://github.com/chroma-core/chroma/tree/main/docs/docs.trychroma.com)?_

2026-03-09

wip

2026-03-05

[ENH]: Get database name from get_collections_to_gc mcmr endpoint

[ENH]: Compaction endpoint to return where a collection would be assigned (#6564) ## Description of changes - Improvements & Bug fixes - N/A - New functionality - `./compaction-client --url http://localhost:50051 get-collection-assignment --collection-id <UUID>` returns where the concerned pod thinks the given collection id would be assigned and also returns what it thinks its memberlist is. ## Test plan Manual local testing. - [ ] Tests pass locally with `pytest` for python, `yarn test` for js, `cargo test` for rust ## Migration plan N/A ## Observability plan N/A ## Documentation Changes _Are all docstrings for user-facing APIs updated if required? Do we need to make documentation changes in the [docs section](https://github.com/chroma-core/chroma/tree/main/docs/docs.trychroma.com)?_

[ENH]: Compaction endpoint to return where a collection would be assigned

[ENH]: ListInProgressJobs endpoint for compactor (#6552) ## Description of changes _Summarize the changes made by this PR._ - Improvements & Bug fixes - Adds a ListInProgressJobs RPC endpoint that returns the list of in progress jobs (collection ids) within the compaction's scheduler along with its expiration time and database name. - New functionality - This endpoint can be accessed by the compaction client by doing `compaction_client --url \<compactor-url> list-in-progress-jobs` ## Test plan Manual testing - [ ] Tests pass locally with `pytest` for python, `yarn test` for js, `cargo test` for rust ## Migration plan _Are there any migrations, or any forwards/backwards compatibility changes needed in order to make sure this change deploys reliably?_ ## Observability plan _What is the plan to instrument and monitor this change?_ ## Documentation Changes _Are all docstrings for user-facing APIs updated if required? Do we need to make documentation changes in the_ [_docs section](https://github.com/chroma-core/chroma/tree/main/docs/docs.trychroma.com)?_

[HOTFIX] applying PR #6539 to release/2026-02-20 (#6555) This PR cherry-picks the commit cc7d74a034f5db17845ffea5e4922518f05a23d9 onto release/2026-02-20. If there are unresolved conflicts, please resolve them manually. Co-authored-by: Hammad Bashir <HammadB@users.noreply.github.com>

[ENH] Add ReadLevel to count in be (#6539) ## Description of changes _Summarize the changes made by this PR._ - Improvements & Bug fixes - ... - New functionality - Introduces read level to count, mirrors existing search read level ## Test plan _How are these changes tested?_ See #6541 which had tests merged into there (Stacked) - [x] Tests pass locally with `pytest` for python, `yarn test` for js, `cargo test` for rust ## Migration plan Noner equried ## Observability plan None new required ## Documentation Changes None

[ENH] Add ReadLevel to count in be (#6539) ## Description of changes _Summarize the changes made by this PR._ - Improvements & Bug fixes - ... - New functionality - Introduces read level to count, mirrors existing search read level ## Test plan _How are these changes tested?_ See #6541 which had tests merged into there (Stacked) - [x] Tests pass locally with `pytest` for python, `yarn test` for js, `cargo test` for rust ## Migration plan Noner equried ## Observability plan None new required ## Documentation Changes None

[HOTFIX] applying PR #6533 to release/2026-02-20 (#6553) This PR cherry-picks the commit 3160e30a9a45e8ea7b98bba009350b633b2a80b5 onto release/2026-02-20. If there are unresolved conflicts, please resolve them manually. --------- Co-authored-by: Robert Escriva <robert@trychroma.com> Co-authored-by: macronova <sicheng@trychroma.com>

[CLN] Remove deprecated in-memory DLQ for compaction dead jobs Dead jobs are now tracked persistently in sysdb via compaction_failure_count. This removes the deprecated ListDeadJobs endpoint and in-memory tracking. - Remove ListDeadJobsMessage from types.rs - Remove Handler<ListDeadJobsMessage> from compaction_manager.rs - Remove list_dead_jobs RPC from compaction_server.rs - Remove ListDeadJobs CLI command from compaction_client.rs - Remove ListDeadJobs proto definitions from compactor.proto Slack thread: https://trychroma.slack.com/archives/C09RE2LFEJY/p1772678305697779?thread_ts=1772678258.865279&cid=C09RE2LFEJY https://claude.ai/code/session_01WkMf7PYLmZLAdcdWZsBxLB

[ENH](log) add pointer-based log fetch via ScoutLogFragments (#6533) ## Description of changes Introduce a new ScoutLogFragments RPC that returns fragment pointers so query and compactor nodes can read log data directly from object storage instead of streaming through gRPC PullLogs. Key changes: - Add ScoutLogFragments RPC and LogFragmentPointer message to proto - Implement server-side handler in log-service using wal3 scan - Add scout_log_fragments client method to grpc_log and Log enum - Create FragmentFetcher in rust/worker that dereferences pointers, reads parquet fragments from storage, and caches raw bytes - Extend FetchLogOperator with use_pointer_fetch toggle to select between gRPC pull and pointer-based fetch strategies - Thread use_pointer_fetch config through compactor and query service - Default use_pointer_fetch to false for safe incremental rollout ## Test plan CI ## Migration plan N/A ## Observability plan Staging ## Documentation Changes N/A Co-authored-by: AI

[ENH]: ListInProgressJobs endpoint for compactor

2026-03-04

[ENH] Add ReadLevel to count in be

2026-02-27

[ENH] Purge dirty via Spanner when topology is specified (#6504) ## Description of changes Make topology_name optional in PurgeDirtyForCollectionRequest. When a topology is provided, purge dirty state directly in the Spanner manifests table by setting ignore_dirty = true, bypassing the dirty log. When no topology is provided, fall back to the existing dirty log append path for backward compatibility. Add purge_dirty_for_collection and unpurge_dirty_for_collection methods to ReplManifestManager that set the ignore_dirty flag in Spanner. ## Test plan CI ## Migration plan N/A ## Observability plan N/A ## Documentation Changes N/A Co-authored-by: AI

[ENH] Thread topology name through purge-dirty pipeline (#6503) ## Description of changes Pass topology_name alongside collection IDs when purging dirty log entries so the log service can scope purges to the correct topology. - Add topology_name field to PurgeDirtyForCollectionRequest proto - Change purge_dirty_for_collection to accept (CollectionUuid, Option<TopologyName>) pairs and group requests by topology - Convert Scheduler.deleted_collections from HashSet to HashMap to track each deleted collection's topology name - Update PurgeDirtyLogInput to carry topology information ## Test plan CI ## Migration plan N/A ## Observability plan N/A ## Documentation Changes N/A Co-authored-by: AI

2026-02-22

[HOTFIX] applying PR #6478 to release/2026-02-19 (#6480) This PR cherry-picks the commit 116642308e2bcad3af63f5b6ce5373ed2065a2ff onto release/2026-02-19. If there are unresolved conflicts, please resolve them manually. Co-authored-by: tanujnay112 <tanujnay112@live.com>