Proto commits in chroma-core/chroma

These commits are when the Protocol Buffers files have changed: (only the last 100 relevant commits are shown)

2026-08-19

[ENH](wqs): Add manual function DLQ RPC (#7612)

The documentation is generated from this commit.

[ENH](wqs): Add manual function DLQ RPC

The documentation is generated from this commit.

2026-08-18

[HOTFIX] applying PR #7582 to release/2026-08-18 (#7604) This PR cherry-picks the commit 290a4788fa773b49ce899c57eda52240659ac6a5 onto release/2026-08-18. If there are unresolved conflicts, please resolve them manually. Co-authored-by: tanujnay112 <tanujnay112@live.com>

The documentation is generated from this commit.

[ENH](wqs): Add manual function DLQ RPC (#7582) ## Summary - Adds a SysDB RPC to set an attached function’s failure count to an exact non-negative value, scoped by `(fn_id, input_coll_id)`. - Adds WQS `SetFunctionFailureCount`: updates SysDB first, then durably mirrors the returned count into the WQS Parquet snapshot. - Returns `NotFound` when the WQS entry is absent, while leaving SysDB as the authoritative count. ## Testing - Go SysDB DAO test for absolute assignment. - Tilt-gated WQS integration test covering SysDB state and `GetWork` DLQ filtering. - Go package compile check.

The documentation is generated from this commit.

[ENH](wqs): Add manual function DLQ RPC (#7582) ## Summary - Adds a SysDB RPC to set an attached function’s failure count to an exact non-negative value, scoped by `(fn_id, input_coll_id)`. - Adds WQS `SetFunctionFailureCount`: updates SysDB first, then durably mirrors the returned count into the WQS Parquet snapshot. - Returns `NotFound` when the WQS entry is absent, while leaving SysDB as the authoritative count. ## Testing - Go SysDB DAO test for absolute assignment. - Tilt-gated WQS integration test covering SysDB state and `GetWork` DLQ filtering. - Go package compile check.

The documentation is generated from this commit.

[ENH](wqs): Add manual function DLQ RPC

The documentation is generated from this commit.

2026-08-15

[BUG](worker): Skip in-progress function work

2026-08-14

[ENH](wqs): Add manual work insertion RPC

[ENH](wqs): Add manual work deletion RPC

[ENH](wqs): Add manual function DLQ RPC

The documentation is generated from this commit.

[ENH](fn-consumer): Dead-letter fn work (#7572) ## Summary Adds fn-consumer DLQ behavior for attached-function work. - Reports a failed function batch to WQS for each input collection. - Before dispatch, reads the attached function from SysDB. - Logs and skips functions whose failure count has reached the configurable threshold (default: 5). - Leaves skipped queue records in WQS, per the current design.

[ENH](wqs): Persist function failure counts

[ENH](wqs): Report fn failures (#7571) ## Summary Adds the WQS failure-reporting boundary. - Adds `FailFunction(fn_id, input_coll_id)` to the WQS API. - Validates IDs and forwards failures to SysDB for durable count increments. - Does not add WQS-specific DLQ persistence; queue records remain in WQS. - Adds Kubernetes integration coverage for WQS → SysDB failure reporting.

[ENH](wqs): Report fn failures

[ENH](sysdb): Track fn failures (#7570) ## Summary Adds durable DLQ failure tracking for attached functions in SysDB. - Adds `attached_functions.failure_count` with an Atlas migration and updated checksum. - Adds the SysDB failure-reporting API used by WQS. - Resets `failure_count` in the existing successful invocation-completion SQL update; no separate reset query is issued. - Adds DAO coverage for failure-count increments.

2026-08-11

[ENH](wqs): Persist function failure counts

[ENH](wqs): Report fn failures

[ENH](sysdb): Return attached failure count

[ENH](sysdb): Track fn failures

2026-07-31

[CLN](sysdb): Remove min_records_for_invocation The field claimed to control how many new records accumulate before an attached function is invoked. Nothing ever compared it to a record count — every reference across Go and Rust was storage, a re-attach idempotency check, a pass-through, or a display line. Foundation set it to 100 and got the compactor's default of 10. The real gate is `should_compact_at` in the log service: uncompacted record count against the compactor's `min_compaction_size`, plus a reinsert threshold and a time-on-log timeout. None of that is per-function, so there was no knob here to keep. An inert setting is worse than a missing one — it invites tuning that does nothing and misleads anyone reading the config to understand cadence. Proto fields 8 and 9 are reserved rather than freed, so the wire numbers can never be recycled. Removal is safe in both directions during a rolling deploy precisely because the value was never read: an old client still sending it is ignored, and a new client omitting it changes no behavior. The DB column is deliberately left in place. Dropping it in the same release would break old sysdb pods mid-rollout, which still SELECT it; the column is NOT NULL with a default, so new writes fill it harmlessly. A follow-up migration should drop it once this release has rolled out. One unrelated fix was required to land: examples/task_api_example.py treated attach_function's `(function, created)` tuple as a single object, which mypy rejects. Staging the file for the comment update surfaced it. Verified: `cargo check --workspace --all-targets` and `cargo fmt --check` clean; `go build` and `go vet` clean over pkg/sysdb, and vet compiles the test files so the test edits are covered.

2026-07-20

[CHORE](sysdb): simplify finish API

[BUG](work-queue): expose compaction frontier to consumer (#7394) ## Description of changes This PR fixes a bug where the fn-consumer work queue was not exposing the **compaction frontier** — the offset up to which the input collection has been compacted. Without this, the fn-consumer couldn't tell whether a queued work item was stale (i.e., the attached function had already processed past the queued offset), leading to redundant or incorrect re-execution. The fix threads `compaction_offset` through the entire pipeline: 1. Proto: add `compaction_offset` to `WorkItemResult` 2. Work queue server: populate it when serving work items 3. Fn-consumer manager: use it to build `FunctionExecutionInput` (replacing the raw `(CollectionUuid, i64)` tuple) 4. Log fetch orchestrator: fetch attached function state first (when in fn-consumer mode), then use the resolved `completion_offset` from sysdb instead of the stale value from the work queue 5. Function execution context: after fetching, skip work items where the attached function has already reached or passed the queued frontier - Improvements & Bug fixes - ... - New functionality - ... ## Test plan _How are these changes tested?_ - [ ] Tests pass locally with `pytest` for python, `yarn test` for js, `cargo test` for rust ## Migration plan _Are there any migrations, or any forwards/backwards compatibility changes needed in order to make sure this change deploys reliably?_ ## Observability plan _What is the plan to instrument and monitor this change?_ ## Documentation Changes _Are all docstrings for user-facing APIs updated if required? Do we need to make documentation changes in the_ [_docs section](https://github.com/chroma-core/chroma/tree/main/docs/docs.trychroma.com)?_

[BUG](work-queue): expose compaction frontier Return the queue compaction frontier from GetWork while completion_offset is no longer persisted in the WQS state file. Keep the field optional on the wire so older consumers remain compatible during rollout.

2026-07-10

[BUG](work-queue): expose compaction frontier Return the queue compaction frontier from GetWork while completion_offset is no longer persisted in the WQS state file. Keep the field optional on the wire so older consumers remain compatible during rollout.

2026-07-09

[ENH](work-queue): require compaction offset (#7358) ## Description of changes This PR makes `compaction_offset` a required field throughout the work queue stack — removing the `Option<i64>` wrapper from the proto, Rust types, and all call sites. This also gets rid of sysdb-based filtering in get_work. - Improvements & Bug fixes - ... - New functionality - ... ## Test plan _How are these changes tested?_ - [ ] Tests pass locally with `pytest` for python, `yarn test` for js, `cargo test` for rust ## Migration plan _Are there any migrations, or any forwards/backwards compatibility changes needed in order to make sure this change deploys reliably?_ ## Observability plan _What is the plan to instrument and monitor this change?_ ## Documentation Changes _Are all docstrings for user-facing APIs updated if required? Do we need to make documentation changes in the_ [_docs section](https://github.com/chroma-core/chroma/tree/main/docs/docs.trychroma.com)?_

[ENH](work-queue): require compaction offset

[BUG](work-queue): expose compaction frontier Return the queue compaction frontier from GetWork while completion_offset is no longer persisted in the WQS state file. Keep the field optional on the wire so older consumers remain compatible during rollout.

2026-07-08

[ENH](work-queue): require compaction offset

2026-07-07

[ENH](logservice): add conditional push and insert offset to proto (#7317) ## Description of changes Extend PushLogsRequest and PushLogsResponse to support conditional log appends: - Add optional PushLogsCondition with observed_log_offset and read_ids - Add optional first_inserted_record_offset to PushLogsResponse Update all PushLogsRequest construction sites and the log-service response to populate the new fields with None defaults. ## Test plan CI ## Migration plan Backwards compatible ## Observability plan N/A ## Documentation Changes N/A Co-authored-by: AI

[CHORE](sysdb): simplify finish API

[BUG](work-queue): expose compaction frontier Return the queue compaction frontier from GetWork while completion_offset is no longer persisted in the WQS state file. Keep the field optional on the wire so older consumers remain compatible during rollout.

2026-07-06

[ENH](work-queue): require compaction offset

[CHORE](sysdb): simplify finish API

Revert "[BUG](fn-consumer): Read offsets from sysdb" This reverts commit f4ea0ff9c6e4c494a6ddf912cf985be639ade204.

[BUG](fn-consumer): Read offsets from sysdb

[ENH](work-queue): require compaction offset

[BUG](work-queue): expose compaction frontier Return the queue compaction frontier from GetWork while completion_offset is no longer persisted in the WQS state file. Keep the field optional on the wire so older consumers remain compatible during rollout.

[CHORE](sysdb): simplify finish API

[BUG](work-queue): expose compaction frontier Return the queue compaction frontier from GetWork while completion_offset is no longer persisted in the WQS state file. Keep the field optional on the wire so older consumers remain compatible during rollout.

Revert "[BUG](fn-consumer): Read offsets from sysdb" This reverts commit f4ea0ff9c6e4c494a6ddf912cf985be639ade204.

[BUG](fn-consumer): Read offsets from sysdb

[ENH](work-queue): require compaction offset

2026-07-02

[ENH](work-queue): add optional compaction offset (#7354) ## Description of changes Part 1 of a stack of changes to persist the input collection compaction offset instead of the function completion offset in the WQS file. This diff starts off by requiring this compaction offset to be sent in the PushWork API. - Improvements & Bug fixes - - New functionality - ... ## Test plan _How are these changes tested?_ - [ ] Tests pass locally with `pytest` for python, `yarn test` for js, `cargo test` for rust ## Migration plan _Are there any migrations, or any forwards/backwards compatibility changes needed in order to make sure this change deploys reliably?_ ## Observability plan _What is the plan to instrument and monitor this change?_ ## Documentation Changes _Are all docstrings for user-facing APIs updated if required? Do we need to make documentation changes in the_ [_docs section](https://github.com/chroma-core/chroma/tree/main/docs/docs.trychroma.com)?_

[CHORE](sysdb): simplify finish API

[BUG](fn-consumer): Read offsets from sysdb

[BUG](work-queue): expose compaction frontier Return the queue compaction frontier from GetWork while completion_offset is no longer persisted in the WQS state file. Keep the field optional on the wire so older consumers remain compatible during rollout.

Revert "[BUG](fn-consumer): Read offsets from sysdb" This reverts commit f4ea0ff9c6e4c494a6ddf912cf985be639ade204.

[ENH](work-queue): require compaction offset

[CHORE](sysdb): simplify finish API

[BUG](fn-consumer): Read offsets from sysdb

2026-07-01

[ENH](work-queue): require compaction offset

2026-06-30

document new fields

[CHORE](sysdb): simplify finish API

[ENH](work-queue): require compaction offset

2026-06-29

[CHORE](sysdb): simplify finish API

[ENH](work-queue): require compaction offset

[ENH](work-queue): add optional compaction offset

[CHORE](sysdb): simplify finish API

[ENH](work-queue): require compaction offset

[ENH](work-queue): add optional compaction offset

2026-06-28

[ENH](logservice): add conditional push and insert offset to proto Extend PushLogsRequest and PushLogsResponse to support conditional log appends: - Add optional PushLogsCondition with observed_log_offset and read_ids - Add optional first_inserted_record_offset to PushLogsResponse Update all PushLogsRequest construction sites and the log-service response to populate the new fields with None defaults. Co-authored-by: AI

2026-06-25

[ENH](logservice): add conditional push and insert offset to proto Extend PushLogsRequest and PushLogsResponse to support conditional log appends: - Add optional PushLogsCondition with observed_log_offset and read_ids - Add optional first_inserted_record_offset to PushLogsResponse Update all PushLogsRequest construction sites and the log-service response to populate the new fields with None defaults. Co-authored-by: AI

[ENH](logservice): add conditional push and insert offset to proto Extend PushLogsRequest and PushLogsResponse to support conditional log appends: - Add optional PushLogsCondition with observed_log_offset and read_ids - Add optional first_inserted_record_offset to PushLogsResponse Update all PushLogsRequest construction sites and the log-service response to populate the new fields with None defaults. Co-authored-by: AI

[ENH](logservice): add conditional push and insert offset to proto Extend PushLogsRequest and PushLogsResponse to support conditional log appends: - Add optional PushLogsCondition with observed_log_offset and read_ids - Add optional first_inserted_record_offset to PushLogsResponse Update all PushLogsRequest construction sites and the log-service response to populate the new fields with None defaults. Co-authored-by: AI

[ENH](logservice): add conditional push and insert offset to proto Extend PushLogsRequest and PushLogsResponse to support conditional log appends: - Add optional PushLogsCondition with observed_log_offset and read_ids - Add optional first_inserted_record_offset to PushLogsResponse Update all PushLogsRequest construction sites and the log-service response to populate the new fields with None defaults. Co-authored-by: AI

2026-06-04

[ENH]: Finish_work and output collection flush are atomic

[ENH]: Add scaffolding for functions with many inputs in fn consumer (#7143) ## Description of changes This PR upgrades the fn consumer pipeline to support functions with **multiple input collections**. Previously, `AttachedFunctionExecutor::execute` received a single `Chunk` of records; now it receives a `Vec<Chunk>` — one chunk per input collection. The orchestration layer is restructured to carry batches of `(collection_id, materialized_logs)` pairs through the pipeline, and the fn consumer manager groups work items by function before dispatching them as a single batch. The key structural changes are: 1. `AttachedFunctionExecutor::execute` signature: `Chunk<…>` → `Vec<Chunk<…>>` 2. New `FunctionExecutionBatch` / `FunctionExecutionProgress` / `FunctionContext` types extracted into a new `function_execution` module 3. `ExecuteAttachedFunctionInput` gains `input_batches: Vec<ExecuteAttachedFunctionBatchInput>` replacing the flat `materialized_logs` + `input_record_segment` 4. `FinishAsyncWorkInput` now carries `Vec<FinishAsyncWorkItem>` instead of a single offset 5. `FnConsumerManager` groups work items by function ID and dispatches them as a batch via `FunctionExecutionContext::run` - Improvements & Bug fixes - ... - New functionality - ... ## Test plan _How are these changes tested?_ - [ ] Tests pass locally with `pytest` for python, `yarn test` for js, `cargo test` for rust ## Migration plan _Are there any migrations, or any forwards/backwards compatibility changes needed in order to make sure this change deploys reliably?_ ## Observability plan _What is the plan to instrument and monitor this change?_ ## Documentation Changes _Are all docstrings for user-facing APIs updated if required? Do we need to make documentation changes in the_ [_docs section](https://github.com/chroma-core/chroma/tree/main/docs/docs.trychroma.com)?_

[CLN](worker): Box function execution fetch future

[ENH]: Add route to allow multiple inputs on an async function (#7142) ## Description of changes This PR adds a new `add_input` route that lets callers associate additional input collections with an existing async attached function. The key data model insight: the `attached_functions` table uses a composite primary key of `(id, input_collection_id)`, so one logical attached function can have multiple rows — one per input collection. Only async functions support multiple inputs. The change spans the full stack: protobuf → Go sysdb coordinator → Rust sysdb client → Rust frontend server → Python API client. - Improvements & Bug fixes - ... - New functionality - ... ## Test plan _How are these changes tested?_ - [ ] Tests pass locally with `pytest` for python, `yarn test` for js, `cargo test` for rust ## Migration plan _Are there any migrations, or any forwards/backwards compatibility changes needed in order to make sure this change deploys reliably?_ ## Observability plan _What is the plan to instrument and monitor this change?_ ## Documentation Changes _Are all docstrings for user-facing APIs updated if required? Do we need to make documentation changes in the_ [_docs section](https://github.com/chroma-core/chroma/tree/main/docs/docs.trychroma.com)?_

[TEST](coordinator): Fix attach function collection mocks

2026-05-27

[BUG]: WQS repairs on bootup (#7126) ## Description of changes This PR renames `AreInvocationsDone` (which returned `[]bool`) to `CheckInvocationStatus` (which returns a three-state enum). The key semantic change is that the old binary done/not-done is replaced with three states: - `NOT_DONE` (0) — default, work still in progress - `DONE` (1) — completed (soft/hard deleted, or offset advanced with no pending heap entry) - `NEEDS_REPAIR` (2) — offset advanced but `heap_entry_pending=true`, meaning the WQS crashed mid-repair The new `NEEDS_REPAIR` state enables the work queue manager to automatically repair stale entries on bootup, which is the core bug fix this PR addresses. In the proto we replaced the RPC for `AreInvocationsDone` with ``CheckInvocationStatus` this is not backwards compatible but that is fine as there is not production use for the former RPC yet. - Improvements & Bug fixes - ... - New functionality - ... ## Test plan _How are these changes tested?_ - [ ] Tests pass locally with `pytest` for python, `yarn test` for js, `cargo test` for rust ## Migration plan _Are there any migrations, or any forwards/backwards compatibility changes needed in order to make sure this change deploys reliably?_ ## Observability plan _What is the plan to instrument and monitor this change?_ ## Documentation Changes _Are all docstrings for user-facing APIs updated if required? Do we need to make documentation changes in the_ [_docs section](https://github.com/chroma-core/chroma/tree/main/docs/docs.trychroma.com)?_

[BUG]: WQS repairs on bootup

2026-05-22

[BUG]: Fn consumer manager keys its job scheduler on output collection id (#7117) ## Description of changes The `FnConsumerManager` was keying its "in-progress" job scheduler on `(fn_id, input_coll_id)` — a `FnJobKey`. The bug: multiple different attached functions could share the same output collection, and the old key didn't prevent two jobs from writing to the same output collection concurrently. The fix re-keys the in-progress map on `output_coll_id` (`CollectionUuid`), which is the actual resource that must be serialized. To support this, `GetAttachedFunctions` gains a batch `ids` parameter so the manager can look up all output collection IDs in a single sysdb call rather than one per work item. - Improvements & Bug fixes - ... - New functionality - ... ## Test plan _How are these changes tested?_ - [ ] Tests pass locally with `pytest` for python, `yarn test` for js, `cargo test` for rust ## Migration plan _Are there any migrations, or any forwards/backwards compatibility changes needed in order to make sure this change deploys reliably?_ ## Observability plan _What is the plan to instrument and monitor this change?_ ## Documentation Changes _Are all docstrings for user-facing APIs updated if required? Do we need to make documentation changes in the_ [_docs section](https://github.com/chroma-core/chroma/tree/main/docs/docs.trychroma.com)?_

[BUG]: Fn consumer manager keys its job scheduler on output collection id

2026-05-14

[ENH] implement stratified sampling endpoint for collections Add a new sample API that randomly selects records from a collection using stratified sampling over SPANN index strata. Records can be filtered by ID, metadata, or document predicates before sampling. The endpoint accepts a limit and optional seed for reproducibility. Full-stack implementation across all layers: - API types and request/response models (Rust, Python, JS/TS) - Protobuf IDL for distributed query execution - Frontend server routing, metrics, and executor integration - Sample operator with SPANN-aware stratified selection - Sample orchestrator with filter, projection, and shard fan-out - SPANN index reader support for head-count enumeration - Python and JS client bindings Co-authored-by: AI

2026-05-13

[ENH]: Function consumer server

[ENH]: Function consumer server

[ENH]: Function consumer server

2026-05-11

[ENH]: Function consumer server

2026-05-08

[ENH]: Add sysdb endpoint to check if invocations are done (#7016) ## Description of changes This PR adds a new `AreInvocationsDone` endpoint to sysdb that lets callers batch-check whether attached function invocations have completed. The check works by comparing `completion_offset > invocation_id AND heap_repair_needed` in the database. - Improvements & Bug fixes - ... - New functionality - New SysDB endpoint as described. ## Test plan _How are these changes tested?_ Go tests added to test the filtering logic of this endpoint. - [ ] Tests pass locally with `pytest` for python, `yarn test` for js, `cargo test` for rust ## Migration plan _Are there any migrations, or any forwards/backwards compatibility changes needed in order to make sure this change deploys reliably?_ ## Observability plan _What is the plan to instrument and monitor this change?_ ## Documentation Changes _Are all docstrings for user-facing APIs updated if required? Do we need to make documentation changes in the_ [_docs section](https://github.com/chroma-core/chroma/tree/main/docs/docs.trychroma.com)?

2026-05-07

[ENH]: Add sysdb endpoint to check if invocations are done

[ENH]: Sysdb calls for finishing function invocation (#7015) ## Description of changes This PR adds two new sysdb RPC endpoints for finishing async attached function invocations: 1. **`TryFinishAsyncAttachedFunctionInvocation`** — records a new completion offset for an async function. If the offset is behind the collection's current log position, it signals that a repair is needed (returns `NeedsRepair`); otherwise it returns `Success`. 2. **`FinalizeAsyncAttachedFunctionRepair`** — called after repair is complete to flip `can_gc` back to `true`. A new `can_gc` column is added to the `attached_functions` table to track whether the function's output is safe to garbage-collect. - Improvements & Bug fixes - N/A - New functionality - Aforementioned new endpoints ## Test plan _How are these changes tested?_ Go tests have been added to test the repair and non-repair cases of these new methods. - [ ] Tests pass locally with `pytest` for python, `yarn test` for js, `cargo test` for rust ## Migration plan _Are there any migrations, or any forwards/backwards compatibility changes needed in order to make sure this change deploys reliably?_ A migration has been added to add a 'can_gc' column to the `attached_functions` table. ## Observability plan _What is the plan to instrument and monitor this change?_ ## Documentation Changes _Are all docstrings for user-facing APIs updated if required? Do we need to make documentation changes in the_ [_docs section](https://github.com/chroma-core/chroma/tree/main/docs/docs.trychroma.com)?_

[ENH]: Add work queue proto definitions and core types (#7020) ## Description of changes This is the base PR for the work queue service stack. It introduces two things: 1. A protobuf contract (`workqueue.proto`) defining the gRPC API surface 2. Rust core types (`WorkQueueRecord`, `WorkQueueError`, `FinishResult`) that the rest of the stack builds on Nothing is wired up in this change. This is scaffolding for later PRs. - Improvements & Bug fixes - N/A - New functionality - N/A ## Test plan N/A - [ ] Tests pass locally with `pytest` for python, `yarn test` for js, `cargo test` for rust ## Migration plan _Are there any migrations, or any forwards/backwards compatibility changes needed in order to make sure this change deploys reliably?_ ## Observability plan _What is the plan to instrument and monitor this change?_ ## Documentation Changes _Are all docstrings for user-facing APIs updated if required? Do we need to make documentation changes in the_ [_docs section](https://github.com/chroma-core/chroma/tree/main/docs/docs.trychroma.com)?_

[ENH]: Sysdb call for function invocation

[ENH]: Add async mode to functions (#7014) ## Description of changes This PR adds an `is_async` field to the `Function` model, propagating it through the full stack: database schema → Go model → protobuf → Rust types → orchestrator logic. The core behavioral change is in the orchestrator: async functions skip updating the `completion_offset` during processing, because they'll be handled through a separate queue mechanism in later PRs. - Improvements & Bug fixes - N/A - New functionality - Can define a function to be async, attached functions runs of such functions will do the aforementioned async behavior. ## Test plan _How are these changes tested?_ Existing function go tests have been adjusted for this new field. - [x] Tests pass locally with `pytest` for python, `yarn test` for js, `cargo test` for rust ## Migration plan _Are there any migrations, or any forwards/backwards compatibility changes needed in order to make sure this change deploys reliably?_ A new `is_async` boolean column is added to `attached_functions` with `NOT NULL DEFAULT false`, ensuring backward compatibility — all existing rows default to synchronous behavior. ## Observability plan _What is the plan to instrument and monitor this change?_ ## Documentation Changes _Are all docstrings for user-facing APIs updated if required? Do we need to make documentation changes in the_ [_docs section](https://github.com/chroma-core/chroma/tree/main/docs/docs.trychroma.com)?_

2026-05-06

feat: Expose work queue service in Kubernetes - Add port 50054 to compaction service container ports - Create work-queue-service Service to expose the gRPC endpoint - Enable work queue via environment variable CHROMA_WORK_QUEUE__ENABLED - Remove separate work-queue-config.yaml (configuration via env vars)

[ENH]: Add async mode to functions

2026-05-05

feat: Expose work queue service in Kubernetes - Add port 50054 to compaction service container ports - Create work-queue-service Service to expose the gRPC endpoint - Enable work queue via environment variable CHROMA_WORK_QUEUE__ENABLED - Remove separate work-queue-config.yaml (configuration via env vars)

2026-04-30

[ENH]: Add metadata support to Database entity This adds optional metadata field to databases, following the pattern used for Collections. Users can now store arbitrary key-value metadata on databases for organizational and tagging purposes. Changes: - New database_metadata table in Go PostgreSQL and Rust SQLite - Proto updates to Database message and CreateDatabaseRequest - Go backend: dbmodel, DAO, catalog, and gRPC handlers - Rust backend: table definitions, sqlite.rs queries - Python client: AdminAPI, FastAPI endpoints - JavaScript client: AdminClient.createDatabase - Integration tests for database metadata https://claude.ai/code/session_019rHB7PwwuyJvyypxuJVCCs

2026-04-17

Cherry-pick with conflicts: fdcd2165154b6bd6394983c49a9ba02cc71ff925

[ENH]: Enable rebuilds for sharded collections (#6916) ## Description of changes This change adds rebuild capabilities for sharded collections. A rebuild command can now specify a shard index it wants to rebuild. An unspecified one defaults to 0 (first shard). This shard's filepaths are set to empty during rebuild compaction, analogous to all of the segment's filepaths being set to empty before this change. A new struct packaging the rebuild command parameters called `RebuildInfo` is passed to `CompactionContext` now. This contains the desired shard index to be rebuilt. - Improvements & Bug fixes - ... - New functionality - ... ## Test plan Tests have been added in compact.rs. They test rebuilding various shards in the full and partial rebuild scenarios. - [ ] Tests pass locally with `pytest` for python, `yarn test` for js, `cargo test` for rust ## Migration plan _Are there any migrations, or any forwards/backwards compatibility changes needed in order to make sure this change deploys reliably?_ ## Observability plan _What is the plan to instrument and monitor this change?_ ## Documentation Changes _Are all docstrings for user-facing APIs updated if required? Do we need to make documentation changes in the_ [_docs section](https://github.com/chroma-core/chroma/tree/main/docs/docs.trychroma.com)?_

2026-04-16

[ENH]: Enable rebuilds for sharded collections

[HOTFIX] applying PR #6914 to release/2026-04-03 (#6920) This PR cherry-picks the commit f749f58f881830ad5ed4d728529b5d491656f825 onto release/2026-04-03. If there are unresolved conflicts, please resolve them manually. Co-authored-by: Macronova <60079945+Sicheng-Pan@users.noreply.github.com>

2026-04-15

[ENH] Add IndexAndBoundedWal read level (#6914) ## Description of changes Add a new `IndexAndBoundedWal` read level for collection search and count APIs. This level reads from the index plus up to a server-configured number of write-ahead log entries, providing a consistent prefix of the WAL with bounded query latency. Unlike `IndexAndWal` (which reads the entire WAL) or `IndexOnly` (which skips it entirely), `IndexAndBoundedWal` caps the log fetch via the existing `maximum_fetch_count` on `FetchLogOperator`. The limit is controlled by a new `bounded_wal_limit` config field on `QueryServiceConfig` (default 250). - New functionality - `INDEX_AND_BOUNDED_WAL` proto enum variant in `ReadLevel` - `IndexAndBoundedWal` Rust type with bidirectional proto conversions - `bounded_wal_limit` config field on `QueryServiceConfig` (default 250) - Bounded WAL fetch logic in `CountOrchestrator` and `KnnFilterOrchestrator` — clones `FetchLogOperator` with `maximum_fetch_count = Some(bounded_wal_limit)` ## Test plan - [x] Behavioral unit test (`test_read_level_semantics`) in `count.rs` — sets up 5 compacted + 10 uncompacted records and verifies: - `IndexAndWal` → 15 (all records) - `IndexOnly` → 5 (compacted only) - `IndexAndBoundedWal` with limit 3 → 8 (partial WAL) - `IndexAndBoundedWal` with limit 10 → 15 (full WAL) - `IndexAndBoundedWal` with limit 100 → 15 (limit exceeds WAL, reads all available)

[ENH] Add IndexAndBoundedWal read level (#6914) ## Description of changes Add a new `IndexAndBoundedWal` read level for collection search and count APIs. This level reads from the index plus up to a server-configured number of write-ahead log entries, providing a consistent prefix of the WAL with bounded query latency. Unlike `IndexAndWal` (which reads the entire WAL) or `IndexOnly` (which skips it entirely), `IndexAndBoundedWal` caps the log fetch via the existing `maximum_fetch_count` on `FetchLogOperator`. The limit is controlled by a new `bounded_wal_limit` config field on `QueryServiceConfig` (default 250). - New functionality - `INDEX_AND_BOUNDED_WAL` proto enum variant in `ReadLevel` - `IndexAndBoundedWal` Rust type with bidirectional proto conversions - `bounded_wal_limit` config field on `QueryServiceConfig` (default 250) - Bounded WAL fetch logic in `CountOrchestrator` and `KnnFilterOrchestrator` — clones `FetchLogOperator` with `maximum_fetch_count = Some(bounded_wal_limit)` ## Test plan - [x] Behavioral unit test (`test_read_level_semantics`) in `count.rs` — sets up 5 compacted + 10 uncompacted records and verifies: - `IndexAndWal` → 15 (all records) - `IndexOnly` → 5 (compacted only) - `IndexAndBoundedWal` with limit 3 → 8 (partial WAL) - `IndexAndBoundedWal` with limit 10 → 15 (full WAL) - `IndexAndBoundedWal` with limit 100 → 15 (limit exceeds WAL, reads all available)

[ENH](faults): add fault injection control plane (#6795) ## Description of changes Add the fault injection proto service and the new chroma-faults crate for in-process fault state management. Implement inject, list, and clear RPCs over a shared FaultRegistry, and wire log-service to build and expose the new tonic service while registering the shared Arc in the config registry. ## Test plan CI + further dev ## Migration plan N/A ## Observability plan N/A ## Documentation Changes N/A Co-authored-by: AI

2026-04-05

[ENH]: Pass down shard_index, num_shards and log_upper_bound_offset to query (#6810) ## Description of changes _Summarize the changes made by this PR._ - Improvements & Bug fixes - New functionality - Introduces new fields - shard_index, num_shards and log_upper_bound_offset in ScanProto and Scan types. - Currently FE just sets shard_index=0, num_shards=1 and log_upper_bound_offset=0. The query nodes don't use these fields - These fields will be used by worker in a future PR to execute logic for only its shard - Handled backwards and forward compatibility. If num_shards is set to 0 (i.e. absent) then query node sets it to 1 ## Test plan _How are these changes tested?_ - [x] Tests pass locally with `pytest` for python, `yarn test` for js, `cargo test` for rust ## Migration plan None ## Observability plan None ## Documentation Changes None

2026-04-03

add log offset upper bound

[ENH]: Pass down shard id and num shards to query

2026-03-30

[ENH] Add description field to collections Add an optional text description field to collections across the full stack: - Python types, proto definitions, and conversion functions - Go model, DB model, DAO, catalog, and gRPC layers - Postgres and SQLite migrations - FastAPI server endpoints and HTTP clients (sync + async) - Abstract API interfaces and SegmentAPI implementation - Collection model classes (sync + async) with modify() support https://claude.ai/code/session_01XmGiw5KPfuugfHgSbPL2pX