Proto commits in chroma-core/chroma

These commits are when the Protocol Buffers files have changed: (only the last 100 relevant commits are shown)

2025-05-09

[CLN] Rename $matches to $regex

2025-05-08

[ENH]: add batch get version file paths method to Sysdb

[ENH] Add RPC on SysDB to get fork count for a collection (#4484) ## Description of changes _Summarize the changes made by this PR._ - Improvements & Bug fixes - N/A - New functionality - Adds a new RPC to the SysDB, `CountForks`, that returns the number of forks for a given collection. ## Test plan _How are these changes tested?_ - [ ] Tests pass locally with `pytest` for python, `yarn test` for js, `cargo test` for rust ## Documentation Changes _Are all docstrings for user-facing APIs updated if required? Do we need to make documentation changes in the [docs section](https://github.com/chroma-core/chroma/tree/main/docs/docs.trychroma.com)?_

The documentation is generated from this commit.

2025-05-07

[ENH] Add RPC on SysDB to get fork count for a collection

The documentation is generated from this commit.

[ENH] A route and tool to inspect the dirty log. (#4461) ## Description of changes To get ground truth on the dirty log, add a tool that can print it as it gets interpreted in the 'get all to compact' call. ## Test plan - [ ] Tests pass locally with `pytest` for python, `yarn test` for js, `cargo test` for rust ## Documentation Changes N/A

2025-05-06

[ENH]: add batch get version file paths method to Sysdb

wip

[ENH] A route and tool to inspect the dirty log. To get ground truth on the dirty log, add a tool that can print it as it gets interpreted in the 'get all to compact' call.

2025-05-05

[ENH]: add batch get version file paths method to Sysdb

2025-05-03

[ENH]: add batch get version file paths method to Sysdb

2025-05-01

[ENH] Wire up regex filter from client to query node (#4410) ## Description of changes _Summarize the changes made by this PR._ - Improvements & Bug fixes - N/A - New functionality - Introduce the regex filters for documents: - `{"$matches": "<regex>"}`: Evaluates to true for documents matching the regex - `{"$not_matches": "<regex>"}`: The exact opposite of `$matches`. (i.e. it will evaluate to true on records without documents) - Propagates the regex filters from the client to the query node. Current the implementation is `todo!()` for local and distributed chroma, and the code will panic when reached. The `todo!`s will be replaced with actual implementations with future impls. ## Test plan _How are these changes tested?_ - [ ] Tests pass locally with `pytest` for python, `yarn test` for js, `cargo test` for rust ## Documentation Changes _Are all docstrings for user-facing APIs updated if required? Do we need to make documentation changes in the [docs section](https://github.com/chroma-core/chroma/tree/main/docs/docs.trychroma.com)?_

Reuse existing field for rename

2025-04-30

[ENH] Wire up regex filter from client to query node

2025-04-22

[ENH] Implement log forking (#4326) ## Description of changes *Summarize the changes made by this PR.* - Improvements & Bug fixes - N/A - New functionality - Implement log forking. Currently it is only implemented for legacy log service, by naively cloning the log offsets and records. ## Test plan *How are these changes tested?* - [ ] Tests pass locally with `pytest` for python, `yarn test` for js, `cargo test` for rust ## Documentation Changes *Are all docstrings for user-facing APIs updated if required? Do we need to make documentation changes in the [docs repository](https://github.com/chroma-core/docs)?*

[ENH] Implement log forking

Do not lock log when fork

[ENH] Wire up proto defs for sysdb fork endpoint (#4299) ## Description of changes *Summarize the changes made by this PR.* - Improvements & Bug fixes - N/A - New functionality - Defines the SysDB fork service request and response - Wires up the request upto `table_catalog.go` ## Test plan *How are these changes tested?* - [ ] Tests pass locally with `pytest` for python, `yarn test` for js, `cargo test` for rust ## Documentation Changes *Are all docstrings for user-facing APIs updated if required? Do we need to make documentation changes in the [docs repository](https://github.com/chroma-core/docs)?*

Do not lock log when fork

[ENH] Implement log forking

Add log offsets to fork request

Specify new collection id in request for retriable fork

Update lineage file format

Wire up proto defs for sysdb fork endpoint

2025-04-21

[ENH] Implement log forking

Update lineage file format

Wire up proto defs for sysdb fork endpoint

Specify new collection id in request for retriable fork

2025-04-18

[ENH] Implement log forking

Update lineage file format

Specify new collection id in request for retriable fork

Wire up proto defs for sysdb fork endpoint

[ENH] grpc changes for update collection config (#4083) ## Description of changes *Summarize the changes made by this PR.* - Improvements & Bug fixes - ... - New functionality - add `configuration_json_str` to go grpc implementation and enable writing to grpc service from rust - add logic to overwrite existing configuration attributes with the updated configuration attributes ## Test plan *How are these changes tested?* - [ ] Tests pass locally with `pytest` for python, `yarn test` for js, `cargo test` for rust ## Documentation Changes *Are all docstrings for user-facing APIs updated if required? Do we need to make documentation changes in the [docs repository](https://github.com/chroma-core/docs)?*

add grpc changes for update

2025-04-17

Specify new collection id in request for retriable fork

Update lineage file format

Wire up proto defs for sysdb fork endpoint

add grpc changes for update

Specify new collection id in request for retriable fork

Wire up proto defs for sysdb fork endpoint

Update lineage file format

2025-04-16

add grpc changes for update

2025-04-15

add grpc changes for update

Revert "[BUG] OBO between log service and compaction. (#4276)" This reverts commit e33f7703c3fbe6819553a228f77b794eee217708.

[BUG] OBO between log service and compaction. (#4276) Compaction assumes that enumeration position t_i means t_i was the last record seen and therefore next reader should read from t_i + 1. Log service was built and tested for t_i meaning t_i was the first record to return. Note: I changed the go code, but only by moving a +1 out a layer. The inner version was inconsistent with convention, so I updated it.

2025-04-10

[ENH] Add a scout-logs function to find the max log position. (#4232) This PR makes a currently-not-used scout-logs function. It adds it to the go service, the rust log service over grpc, and the rust log client over grpc and in-memory.

2025-04-09

[BUG]: use absolute cutoff time when listing collections to garbage collect (#4245) ## Description of changes We currently send a relative cutoff time in seconds to the Sysdb when it expects an absolute cutoff time. This updates the garbage collection service to send an absolute cutoff time instead. ## Test plan *How are these changes tested?* - [x] Tests pass locally with `pytest` for python, `yarn test` for js, `cargo test` for rust ## Documentation Changes *Are all docstrings for user-facing APIs updated if required? Do we need to make documentation changes in the [docs repository](https://github.com/chroma-core/docs)?* n/a

2025-04-07

[ENH]: allow overriding default garbage collection mode per tenant & clean up cutoff time parameters (#4135) ## Description of changes The main change here is allowing setting different garbage collection modes per tenant. This will allow us to enable delete mode for one tenant while testing and keep everyone else in dry run mode. I added a new test, but for it to work I ended up needing to make some changes around how cutoff time is handled (was planning to make these changes, just in a later PR). ## Test plan *How are these changes tested?* - [x] Tests pass locally with `pytest` for python, `yarn test` for js, `cargo test` for rust Added a new E2E test to validate tenant overrides. ## Documentation Changes *Are all docstrings for user-facing APIs updated if required? Do we need to make documentation changes in the [docs repository](https://github.com/chroma-core/docs)?*

2025-04-05

[ENH]: allow overriding default garbage collection mode per tenant

[ENH]: rename GC `ListOnly` mode -> `DryRun`, do not delete versions in `DryRun` mode (#4126) ## Description of changes - Renames garbage collection `ListOnly` mode to `DryRun` for better clarity. - Does not delete versions in `DryRun` mode, only marks them for deletion. These changes make it possible to transition from running `DryRun` mode to running `Rename` or `Delete` modes with no additional tooling. ## Test plan *How are these changes tested?* Added a new E2E test for dry run mode. ## Documentation Changes *Are all docstrings for user-facing APIs updated if required? Do we need to make documentation changes in the [docs repository](https://github.com/chroma-core/docs)?* n/a

2025-04-04

[ENH] Implement rebuild endpoint for compaction server (#4132) ## Description of changes *Summarize the changes made by this PR.* - Improvements & Bug fixes - Updates compaction job definition - New functionality - Add the rebuild endpoint for compaction server ## Test plan *How are these changes tested?* - [ ] Tests pass locally with `pytest` for python, `yarn test` for js, `cargo test` for rust ## Documentation Changes *Are all docstrings for user-facing APIs updated if required? Do we need to make documentation changes in the [docs repository](https://github.com/chroma-core/docs)?*

[ENH]: rename GC `ListOnly` mode -> `DryRun`, do not delete versions in `DryRun` mode

[ENH] Prevent tracing:error calls for soft deleted collection in compaction. (#4146)

[ENH] Wire up the log service's offline path. (#4084) This makes the log-service "compact" the dirty log.

2025-04-01

Adding new Sysdb Grpc for fetching single collection. The grpc returns FailedPreCondition if the collection has been soft deleted.

Allow collections to be forgotten; they will be purged from dirty log on delete from sysdb.

2025-03-25

.

Merge branch 'main' into jai/update-coll-config

2025-03-24

[ENH] Calculate and update collection logical size during compaction (#4071) Fixes issues in https://github.com/chroma-core/chroma/pull/4019

Revert "[ENH] Calculate and update collection logical size during compaction" (#4070) Reverts chroma-core/chroma#4019

[ENH] Calculate and update collection logical size during compaction (#4019) ## Description of changes *Summarize the changes made by this PR.* - Improvements & Bug fixes - N/A - New functionality - Propagate the new collection logical size and compaction time to SysDB ## Test plan *How are these changes tested?* - [ ] Tests pass locally with `pytest` for python, `yarn test` for js, `cargo test` for rust ## Documentation Changes *Are all docstrings for user-facing APIs updated if required? Do we need to make documentation changes in the [docs repository](https://github.com/chroma-core/docs)?* --------- Co-authored-by: Sicheng Pan <sicheng@trychroma.com>

2025-03-19

add update collection config plumbing

2025-03-18

Prop Tests for GC. Use RefState and SUT to test GC. Update TestSysDb.

[ENH] Make SysDb return eligible collections to GC. (#3916) ## Description of changes *Summarize the changes made by this PR.* - Improvements & Bug fixes - Limit the number of collections, and return those that have at least 1 version that is elligible for GC. - New functionality - ... ## Test plan *How are these changes tested?* - [ ] Tests pass locally with `pytest` for python, `yarn test` for js, `cargo test` for rust ## Documentation Changes *Are all docstrings for user-facing APIs updated if required? Do we need to make documentation changes in the [docs repository](https://github.com/chroma-core/docs)?*

2025-03-17

Prop Tests for GC. Use RefState and SUT to test GC. Update TestSysDb.

2025-03-12

[ENH] Add log bytes pulled as part of query result (#3972) ## Description of changes *Summarize the changes made by this PR.* - Improvements & Bug fixes - N/A - New functionality - Added the size of pulled log on the read endpoint results ## Test plan *How are these changes tested?* - [ ] Tests pass locally with `pytest` for python, `yarn test` for js, `cargo test` for rust ## Documentation Changes *Are all docstrings for user-facing APIs updated if required? Do we need to make documentation changes in the [docs repository](https://github.com/chroma-core/docs)?* --------- Co-authored-by: Sicheng Pan <sicheng@trychroma.com>

2025-03-05

[ENH] Add size_bytes_post_compaction and last_compaction_time_secs to SysDB collection (#3886) ## Description of changes *Summarize the changes made by this PR.* - Improvements & Bug fixes - N/A - New functionality - Adds `size_bytes_post_compaction` and `last_compaction_time_secs` fields to the SysDB's Collection. ## Test plan *How are these changes tested?* - [ ] Tests pass locally with `pytest` for python, `yarn test` for js, `cargo test` for rust ## Documentation Changes *Are all docstrings for user-facing APIs updated if required? Do we need to make documentation changes in the [docs repository](https://github.com/chroma-core/docs)?*

2025-03-04

[ENH] Add size_bytes_post_compaction and last_compaction_time_secs to SysDB collection

2025-03-03

[ENH] Garbage Collection property tests. (#3881) ## Description of changes *Summarize the changes made by this PR.* - Improvements & Bug fixes - ... - New functionality - ... ## Test plan *How are these changes tested?* - [ ] Tests pass locally with `pytest` for python, `yarn test` for js, `cargo test` for rust ## Documentation Changes *Are all docstrings for user-facing APIs updated if required? Do we need to make documentation changes in the [docs repository](https://github.com/chroma-core/docs)?*

2025-03-02

[ENH] GC - Property test. Property tested : Create mulitple batches of embeddings. All records should exist irrespective of number of GC runs.

2025-02-16

[ENH] GC Tool - Tilt based tests. (#3749) ## Description of changes Includes changes to SysDb. Added operators for GC. Contains tests in garbage collection orchestrator. *Summarize the changes made by this PR.* - Improvements & Bug fixes - ... - New functionality - ... ## Test plan *How are these changes tested?* - [ ] Tests pass locally with `pytest` for python, `yarn test` for js, `cargo test` for rust ## Documentation Changes *Are all docstrings for user-facing APIs updated if required? Do we need to make documentation changes in the [docs repository](https://github.com/chroma-core/docs)?*

2025-02-15

[ENH] GC - Property test. Property tested : Create mulitple batches of embeddings. All records should exist irrespective of number of GC runs.

2025-02-14

Resolve merge conflicts

2025-02-13

Cherry-pick with conflicts: 3a9a76ee1f9e0e7464ff9e40a0b55ec49792a468

[ENH] Count collections implementation (#3785) ## Description of changes *Summarize the changes made by this PR.* - Improvements & Bug fixes - Introduces and Implements count_collections rpc in sysdb - Previously, we were suboptimally performing a len(get_collections()) which results in fan out of multiple db queries to fetch collection metadata one per collection - Both the python sysdb client and rust sysdb client consume the rpc introduced in sysdb - Both the rust frontend and python frontend consume the new impl - New functionality - ... ## Test plan *How are these changes tested?* - Added a test in test_system.py - [x] Tests pass locally with `pytest` for python, `yarn test` for js, `cargo test` for rust ## Documentation Changes None

Make database name optional

[ENH] Count collections implementation

2025-02-12

Fixes and tests.

2025-01-27

[ENH] Add GRPCs in SysDB to help with garbage collection (#3539) ## Description of changes *Summarize the changes made by this PR.* - Improvements & Bug fixes - ... - New functionality - New GRPCs to mark versions for deletion, and to delete versions. ## Test plan *How are these changes tested?* - [ ] Tests pass locally with `pytest` for python, `yarn test` for js, `cargo test` for rust ## Documentation Changes *Are all docstrings for user-facing APIs updated if required? Do we need to make documentation changes in the [docs repository](https://github.com/chroma-core/docs)?*

2025-01-25

[ENH] Serialize Where clause to protobuf (#3565) ## Description of changes *Summarize the changes made by this PR.* - Improvements & Bug fixes - Refactor the deserialization logic for `Where` clause - New functionality - Implement the serialization logic for `Where` clause ## Test plan *How are these changes tested?* - [x] Tests pass locally with `pytest` for python, `yarn test` for js, `cargo test` for rust ## Documentation Changes *Are all docstrings for user-facing APIs updated if required? Do we need to make documentation changes in the [docs repository](https://github.com/chroma-core/docs)?*

2025-01-23

Implementation and test cases of the GRPCs.

Added the GRPCs

[ENH] Create VersionFiles in S3 from Sysdb. GI#822 (#3475) ## Description of changes SysDB will create version files in S3 as part of FlushCollectionCompaction call. CreateCollection will also create a version file. *Summarize the changes made by this PR.* - Improvements & Bug fixes - - New functionality - Creation of version files to aid with GC and Restore Ops. ## Test plan *How are these changes tested?* - [ ] Tests pass locally with `pytest` for python, `yarn test` for js, `cargo test` for rust Added New unit tests. ## Documentation Changes *Are all docstrings for user-facing APIs updated if required? Do we need to make documentation changes in the [docs repository](https://github.com/chroma-core/docs)?*

2025-01-22

Addressed review comments.

2025-01-21

[ENH] Implement compactor server interface (#3375) ## Description of changes *Summarize the changes made by this PR.* - Improvements & Bug fixes - N/A - New functionality - We would like to manually start a compaction on certain collections. This PR is the first in the stack to implement this feature. Specifically, it: - Introduces the grpc interface for compactor. Currently there is only a single variant of request that allows the user to specify an optional list of collection ids for compaction. - Implements the `CompactionServer` struct in rust that implements this interface. It receives the request and send it to the running `CompactionManager` - The next PRs will handle the request in `CompactionManager`. ## Test plan *How are these changes tested?* - [ ] Tests pass locally with `pytest` for python, `yarn test` for js, `cargo test` for rust ## Documentation Changes *Are all docstrings for user-facing APIs updated if required? Do we need to make documentation changes in the [docs repository](https://github.com/chroma-core/docs)?*

[ENH] Implement Get collections to GC + add to rust sysdb client (#3508) ## Description of changes *Summarize the changes made by this PR.* - New functionality - Adds an rpc on Go side that is dumb and gets all collections - Adds this rpc to the rust sysdb client + uses this rpc from rust client in gc ## Test plan *How are these changes tested?* - [x] Tests pass locally with `pytest` for python, `yarn test` for js, `cargo test` for rust ## Documentation Changes None

Get collections to GC

Added the implementation of the GRPCs.

Added the GRPCs

2025-01-18

[ENH] GetCollectionSize on SysDB read replica (#3503) ## Description of changes *Summarize the changes made by this PR.* - Improvements & Bug fixes - None - New functionality - Hooks up `SysDB` Go coordinator to a DB read replica - Adds a new RPC, `GetCollectionSize`, to return the total records in a collection, pulling from the reader DB ## Test plan *How are these changes tested?* - [ ] Tests pass locally with `pytest` for python, `yarn test` for js, `cargo test` for rust ## Documentation Changes *Are all docstrings for user-facing APIs updated if required? Do we need to make documentation changes in the [docs repository](https://github.com/chroma-core/docs)?*

2025-01-17

[ENH] GetCollectionSize on SysDB read replica

2025-01-16

idl makefile generates read coordinator

[ENH] Expose read replica on SysDB coordinator

[ENH] Update compactor to flush total records upon compaction (#3483) ## Description of changes Updates the compactor to calculate the total records per collection and flush to the sysdb upon every compaction. *Summarize the changes made by this PR.* - New functionality - `FlushCollectionCompaction` struct includes `TotalRecordsPostCompaction` - `SysDB` populates the `total_records_post_compaction` column when receiving a flush - `ArrowBlockfileFlusher` contains a new attribute, `total_keys` - `ArrowUnorderedBlockfileWriter` sums the total keys using the `SparseIndexWriter` and returns an `ArrowBlockfileFlusher` with the summed count - `RegisterInput` contains a new attribute, `total_records_post_compaction ` - If `CompactOrchestrator`, when handling `CommitSegmentWriterOutput`, receives a `ChromaSegmentFlusher::RecordSegment`, it reads `total_keys()` and sets it as an attribute on itself. - `ChromaSegmentFlusher::RecordSegment` has `total_keys()` through its `ArrowBlockfileFlusher` - `CompactOrchestrator` sends its `num_records_last_compaction` value to a `RegisterInput` to be flushed to the `SysDB` ## Test plan *How are these changes tested?* - [x] Tests pass locally with `pytest` for python, `yarn test` for js, `cargo test` for rust - [x] Tested locally and confirmed that compaction correctly updates the column in the `SysDB` ## Documentation Changes *Are all docstrings for user-facing APIs updated if required? Do we need to make documentation changes in the [docs repository](https://github.com/chroma-core/docs)?*

[ENH] Update compactor to flush total records upon compaction

i64 -> u64

[ENH] Add num_records_last_compaction to sysdb (#3463) ## Description of changes *Summarize the changes made by this PR.* - New functionality - Adds `num_records_last_compaction` to distributed sysdb ## Test plan *How are these changes tested?* - [x] Tests pass locally with `pytest` for python, `yarn test` for js, `cargo test` for rust ## Documentation Changes *Are all docstrings for user-facing APIs updated if required? Do we need to make documentation changes in the [docs repository](https://github.com/chroma-core/docs)?*

2025-01-14

i64 -> u64

[ENH] add distributed implementation for database deletion (#3460) ## Description of changes Adds implementation for database deletion API to distributed Chroma. ## Test plan *How are these changes tested?* - [x] Tests pass locally with `pytest` for python, `yarn test` for js, `cargo test` for rust Added a new Go test. Mostly tested with existing Python tests. ## Documentation Changes *Are all docstrings for user-facing APIs updated if required? Do we need to make documentation changes in the [docs repository](https://github.com/chroma-core/docs)?* n/a

Add distributed impl

2025-01-10

make num records required