Proto commits in TraceMachina/nativelink

These 51 commits are when the Protocol Buffers files have changed:

2026-07-07

Add REAPI content-defined chunking (SplitBlob/SpliceBlob) support (#2497) * Add REAPI content-defined chunking (SplitBlob/SpliceBlob) support Implements the server side of the remote-apis blob split/splice extension used by Bazel's --experimental_remote_cache_chunking (Bazel 8.7.0+/9.1.0+), fixes #2496. - Vendor SplitBlob/SpliceBlob RPCs, ChunkingFunction, FastCdc2020Params and CacheCapabilities fields 8-12 from upstream remote-apis. - SpliceBlob re-assembles chunked uploads: verifies chunk existence and the spliced digest before committing, materializes the blob so non-chunking clients stay correct, and persists the chunk layout in a configurable index store. Chunk reads are pipelined while hashing stays in chunk order. - SplitBlob serves stored layouts (validated against the blob size so corrupt or truncated index entries are never served), or chunks blobs on demand with FastCDC 2020 (fastcdc crate, normalization level 2) so outputs uploaded whole by remote execution workers also get chunked downloads. Unusable layouts fall back to re-chunking. - Capabilities advertise split/splice support and FastCDC 2020 parameters per instance, collected across all server blocks, gated behind the new opt-in experimental_chunking CAS service config (off by default; zero behavior change when unset). - Reject foot-gun configs at startup: index_store == cas_store (chunk layouts stored under blob digests would overwrite blob content) and chunking on grpc proxy stores (would download/re-upload entire blobs instead of forwarding RPCs). - Conformance-test the chunker against the official REAPI fastcdc2020_test_vectors.txt (offsets, lengths, sha256s and gear fingerprints, seeds 0 and 666, in-memory and streaming). - Add ChunkingMetrics (splice/split totals, hit/miss/on-demand rates, byte counters, digest verification failures). * Forward SplitBlob/SpliceBlob natively through grpc proxy stores For grpc-store-backed CAS instances the chunking RPCs are now forwarded verbatim to the backend (with instance-name rewriting and the store's usual retry handling) instead of being rejected at startup. This makes NativeLink relays transparent for content-defined chunking: one RPC in, one RPC out, the backend owns chunking and the layout index. - Add GrpcStore::split_blob/splice_blob following the existing find_missing_blobs/batch_*/get_tree forwarding pattern. - Shortcut to the proxy in the CAS handlers before any local chunking machinery is consulted, mirroring the other four CAS RPCs. - Make experimental_chunking.index_store optional: required for locally chunked instances, rejected for grpc-store instances where the backend owns the chunk layouts. The capabilities service still advertises split/splice + FastCDC params from the same config block, so relay operators set avg_chunk_size_bytes to match their backend. - Test forwarding against a fake CAS backend over a real gRPC round trip (verifies passthrough and instance-name rewriting) and the new constructor rules. * Fix pre-commit hooks and make max chunk count configurable - Reuse the FastCDC test fixture already vendored at nativelink-util/tests/data/SekienAkashita.jpg (and already excluded from the forbid-binary-files hook) instead of adding a duplicate binary copy; export it from nativelink-util for the conformance test. - Format nativelink-service/Cargo.toml per taplo. - Replace the hardcoded 50k chunk cap with a per-instance experimental_chunking.max_chunk_count knob (default 50000). Blobs above the cap are served without chunking; the layout read cap is derived from the configured count so the two can never disagree. * Add chunking configuration example and clarify optionality Address review feedback: state explicitly that experimental_chunking is optional (with unchanged behavior when unset) and add a complete, test-validated configuration example at nativelink-config/examples/chunking_cas.json5. * Format chunking example per formatjson5 * Add chunking integration test to CI Now that the repo's Bazel is 9.1.1 (which supports --experimental_remote_cache_chunking), exercise the SplitBlob/SpliceBlob paths end-to-end in the existing integration-tests job: enable experimental_chunking in the docker-compose CAS config and add chunking_cache_test.sh, which uploads a ~6.9MB artifact as chunks, asserts the server registered a chunk layout, then re-fetches it through the chunked download path and verifies it is byte-identical. * Infer digest function when unset in chunking RPCs Bazel 9.1.1 leaves digest_function unset in SplitBlob/SpliceBlob even when running with --digest_function=blake3 (surfaced by the new CI chunking integration test, which runs with the repo's blake3 default). REAPI length-based inference cannot disambiguate SHA256 from BLAKE3 (both 32 bytes), so: - SpliceBlob hashes the re-assembled blob with both candidates when the field is unset and accepts whichever reproduces the expected digest. - SplitBlob's on-demand chunking infers the blob's digest function with an extra content pass so chunk digests use the right function. Explicitly-set digest functions keep the single-hasher fast path. * Fix artifact path resolution in chunking integration test The test runner's working directory is deployment-examples/docker-compose, not the workspace root, so the bazel-bin convenience symlink is not at the script's cwd. Resolve the output path through bazel info instead. * Add docs-site page for content-defined chunking Address review: document the feature where customers look — what it is, measured savings, client requirements (Bazel 9.1.1+/8.7+), how to enable it, tuning knobs, and when it does or does not help. * Disable async cache uploads in chunking integration test Bazel has uploaded cache entries in the background by default since Bazel 8 (--remote_cache_async), so `bazel build` can return before the chunked upload and SpliceBlob complete. The CI logs show exactly this: the splice reached the server ~0.9s after "Build completed successfully", while the test asserted on the chunk index ~0.8s after the build returned. Forcing synchronous uploads makes both the chunk-layout assertion and the clean-rebuild cache-hit check deterministic. Verified by running the full script locally against a clean server: splice on upload, layout written, remote cache hit via SplitBlob on rebuild, SHA-identical output. * Pass NATIVELINK_DIR through sudo to docker compose in test harness The integration test harness exports NATIVELINK_DIR but launches the containers with `sudo env RUST_LOG=info docker compose up`, and sudo's env_reset strips the variable. Docker compose therefore fell back to mounting root's ~/.cache/nativelink instead of the per-run cache dir, so the per-test `find "$NATIVELINK_DIR" -delete` cleanup never touched real store state, and chunking_cache_test.sh asserted on a directory the CAS container never wrote to (its chunk layouts landed in /root/.cache/nativelink on the host). Pass the variable through sudo explicitly, and add diagnostics to the chunking test's failure path that list both candidate locations. * Use absolute store paths in docker-compose CAS config NativeLink expands only environment variables in config strings (shellexpand::env — the tilde feature is not enabled), so the "~/.cache/nativelink/..." paths in local-storage-cas.json5 were treated as literal relative paths: inside the container the CAS wrote to "/~/.cache/nativelink" under the process cwd, an unmounted ephemeral directory, never to the /root/.cache/nativelink bind mount. worker.json5 already uses absolute paths, which is why executor data was host-visible while CAS data was not. Use absolute paths to match, and drop the same misleading tilde idiom from the docs-site config snippet.

The documentation is generated from this commit.

2026-06-05

Add worker action memory telemetry (#2398)

2026-05-14

Generate precondition_failure (#2333)

2026-01-29

Add worker config option to limit maximum inflight tasks (#2125)

2025-10-17

Single worker stream (#1977) When there are multiple schedulers in high availability mode then the workers can end up communicating with the wrong scheduler and state can get confused. Modify the communications such that there is a single bi-directional gRPC stream between the client and the worker. This way they will always be one-to-one mapped even with a load balancer in front. Co-authored-by: Chris Staite <chris@yourdreamnet.co.uk>

Notify execution complete (#1975) When execution is complete, there's a large amount of IO still to be done. In the mean time a new action could be starting. Previously an attempt to implement this was quite complex and caused panics. In this implementation a very simple mechanism is used which only executes on success and keeps track of which operations have been notified on the scheduler. This massively simplifies things. Fixes #1903 Co-authored-by: Chris Staite <chris@yourdreamnet.co.uk>

2025-10-16

Single worker stream When there are multiple schedulers in high availability mode then the workers can end up communicating with the wrong scheduler and state can get confused. Modify the communications such that there is a single bi-directional gRPC stream between the client and the worker. This way they will always be one-to-one mapped even with a load balancer in front.

Notify execution complete When execution is complete, there's a large amount of IO still to be done. In the mean time a new action could be starting. Previously an attempt to implement this was quite complex and caused panics. In this implementation a very simple mechanism is used which only executes on success and keeps track of which operations have been notified on the scheduler. This massively simplifies things. Fixes #1903

2025-10-14

Single worker stream When there are multiple schedulers in high availability mode then the workers can end up communicating with the wrong scheduler and state can get confused. Modify the communications such that there is a single bi-directional gRPC stream between the client and the worker. This way they will always be one-to-one mapped even with a load balancer in front.

Notify execution complete When execution is complete, there's a large amount of IO still to be done. In the mean time a new action could be starting. Previously an attempt to implement this was quite complex and caused panics. In this implementation a very simple mechanism is used which only executes on success and keeps track of which operations have been notified on the scheduler. This massively simplifies things. Fixes #1903

2025-09-22

a license to protect the team (#1930) * a license to protect the team * fix nix. * update protos * update stranglers and header generator * Fix protos licenses * Fix non-executable scripts * Fix remaining other Apache licenses * tsan.sh was missing executable --------- Co-authored-by: Tom Parker-Shemilt <tom@tracemachina.com>

2025-09-18

Revert ExecutionComplete early scheduling optimization (#1929) * Revert "Early scheduling (#1904)" This reverts commit 85c279a4467c5322159c5f55bca05be6b3bf92c4. * Revert Early Scheduling to fix the Build Panics

2025-08-14

Early scheduling (#1904) Currently the scheduler waits for an action to complete before freeing the worker to be scheduled again. This is an inefficient use of CPU since uploading the result is IO bound. Add a new message to the worker API to notify the scheduler that the execution has completed and is now doing upload and cleanup. When this is received, the scheduler frees the resources from the worker. Fixes #1903 Co-authored-by: Marcus Eagan <marcuseagan@gmail.com>

2025-04-18

Initial Remote Asset support (#1646)

2025-03-28

Add RemoteAsset protobuf (#1647)

2025-02-06

Add ability to prefix worker_id in config (#1578) WorkerId can now be prefixed with a config. This will cause all internal WorkerIds to be prefixed with this config followed by a generated UUIDv6.

Add OriginEvent for scheduler scheduling action (#1574) * Add OriginEvent for scheduler scheduling action Adds an OriginEvent for when the scheduler schedules an action. This can be used for external systems to give visuals to what is happening inside the system or for debugging. * Expose OriginContextCollector to scheduler's matching engine Ensures OriginContext data is available to the matching engine and when dealing with an action, the first user to invoke the action will be associated with the creation of the operation.

2025-01-27

Add `Closed` stream event to OriginEvents (#1570) Adds an event for when a stream is closed.

2024-12-11

Add OriginEventPublisher (#1497) Adds ability for nativelink to publish events to a store/database that can be used by another service to aggregate and/or visualize what is happening related to requests, users or other related metadata. By default bazel's metadata will be published with every event if present giving the ability to combine with BEP data.

2024-07-28

Significantly reduce Bazel test time (#1210) This massive compile time improvement comes from a few evil platform tricks to fetch a precompiled protoc. This saves 713 compile actions. Observed speedups for clean builds are between 20% and 40%.

2024-07-26

Upgrade toolchains (#1191) - Bump flake - Regenerate LRE toolchains - Update native-cli pulumi build - Bump nightly Rust and fix exposed issues - Fix a misplaced aarch64-linux system configuration in flake (typo fix, doesn't add support for any new targets)

2024-07-17

[Refactor] Overhaul of scheduler component (#1169) This is a significant overhaul to Nativelink's scheduler component. This new scheduler design is to enable a distributed scheduling system. The new components & definitions: * AwaitedActionDb - An interface that is easier to work with when dealing with key-value storage systems. * MemoryAwaitedActionDb - An in-memory set of hashmaps & btrees used to satisfy the requirements of AwaitedActionDb interface. * ClientStateManager - A minimal interface required to satisfy the requirements of a client-facing scheduler. * WorkerStateManager - A minimal interface required to satisfy the requirements of a worker-facing scheduler. * MatchingEngineStateManager - A minimal interface required to satisfy a engine that matches queued jobs to workers. * SimpleSchedulerStateManager - An implements that satisfies ClientStateManager, WorkerStateManager & MatchingEngineStateManager with all the logic of the previous "SimpleScheduler" logic moved behind each interface. * ApiWorkerScheduler - A component that handles all knowledge about workers state and implmenets the WorkerScheduler interface and translates them into the WorkerStateManager interface. * SimpleScheduler - Translation calls of the ClientScheduler interface into ClientStateManager & MatchingEngineStateManager. This component is currently always forwards calls to SimpleSchedulerStateManager then to MemoryAwaitedActionDb. Future changes will make these inner components dynamic via config. In addition we have hardened the interactions of different kind of IDs in NativeLink. Most relevant is the separation & introduction of: * OperationId - Represents an individual operation being requested to be executed that is unique across all of time. * ClientOperationId - An ID issued to the client when the client requests to execute a job. This ID will point to an OperationId internally, but the client is never exposed to the OperationId. * AwaitedActionHashKey - A key used to uniquely identify an action that is not unique across time. This means that this key might have multiple OperationId's that have executed it across different points in time. This key is used as a "fingerprint" of an operation that the client wants to execute and the scheduler may decide to join the stream onto an existing operation if this key has a hit. Overall these changes pave the way for more robust scheduler implementations, most notably, distributed scheduler implementations will be easier to implement and will be introduced in followup PRs. This commit was developed on a side branch and consisted of the following commits with corresponding code reviews: 54ed73cc6fcaf9236a544e45a90b9071b5df6ad8 Add scheduler metrics back (#1171) 50fdbd73983dfdbd2620ac6bfcd5c7b642b5705e fix formatting (#1170) 89262366544d2310e0e9a5bd91c99297112d4e27 Merge in main and format (#1168) 9c2c7b9d3a2bfd71615fe46893ac937e6b23974c key as u64 (#1166) 0192051fca6e3509a6b2953ab53f77c12b4c7092 Cleanup unused code and comments (#1165) 080df5d0c559055d12ac53275024d32e35077675 Add versioning to AwaitedAction (#1163) 73c19c41e31fef55a3c6e81beb19a7233f0c2e8d Fix sequence bug in new memory store manager (#1162) 6e50d2c16fad92f2f8fb856b199f7798c30d5967 New AwaitedActionDb implementation (#1157) 18db991e3a414c3f95c2dd5a14bfafce80358de9 Fix test on running_actions_manager_test (#1141) e50ef3c27d16fc840fd63b917f39f1c317f48398 Rename workers to `worker_scheduler` 1fdd5056f357124d6aff63fe145821b795cf9780 SimpleScheduler now uses config for action pruning (#1137) eaaa872bc62a952fb926214a6fb11bd4ffd25bae Change encoding for items that are cachable (#1136) d64705627c045d108340546a3ea135edb984163a Errors are now properly handles in subscription (#1135) 7c3e730e30e96fb5d08e4d43cca8d27593bcea03 Restructure files to be more appropriate (#1131) 5e98ec9f67f56744376e060b6aa423f1ff401a47 ClientAwaitedAction now uses a channel to notify drops happened (#1130) 52beaf9c727cd396ce4b5b4095bc8dc7be8ce940 Cleanup unused structs (#1128) e86fe0826b3512e78804744a8f1db2ec4c5b21ea Remove all uses of salt and put under ActionUniqueQualifier (#1126) 3b860367ba131c918127b6426e791b9ca0ca8b6b Remove all need for workers to know about ActionId (#1125) 5482d7f5e5f84c2ea80d79c703624c73b6ee36cc Fix bazel build and test on dev (#1123) ba52c7fb76bba56ff5485b91cd516712f867dc64 Implement get_action_info to all ActionStateResult impls (#1118) 2fa4fee48374ed0cf12dcd88225d6d208c0eeeb2 Remove MatchingEngineStateManager::remove_operation (#1119) 34dea0633f8b5198d9c51904beee8f4c617fc822 Remove unused proto field (#1117) 3070a40935c1a490b530af8551329c44bb4500ff Remove metrics from new scheduler (#1116) e95adfcd6035d3cecac74e67829b83e44a598228 StateManager will now cleanup actions on client disconnect (#1107) 6f8c001d1a96a0dc74a5c22716ba2f54e4c9f1b1 Fix worker execution issues (#1114) d353c3003437609547c5dcc8738637b13151a9b5 rename set_priority to upgrade_priority (#1112) 0d93671ba51be41a1a6b0297260cfc5ca0766ebc StateManager can now be notified of noone listeneing (#1093) cfc0cf62c673dc399938247b0c3abe55863c2687 ActionScheduler will now use ActionListener instead of tokio::watch (#1091) d70d31d74cdc337861ebddd8d19201b088b8faa6 QA fixes for scheduler-v2 (#1092) f2cea0cc3fcc259b9a4b7257f2eae807d7e99d05 [Refactor] Complete rewrite of SimpleScheduler 34d93b7d1cda7c0f4a430364b7588002ab612494 [Refactor] Move worker notification in SimpleScheduler under Workers b9d970264b59201e13053844a277092164c8416a [Refactor] Moves worker logic back to SimpleScheduler 7a16e2e6043b17e7813e41450b4de9c40de435f4 [Refactor] Move scheduler state behind mute

Update copyright headers (#1172)

2024-06-08

Introduce basic rustdoc infrastructure (#980) This commit adds a new `docs` command to the flake which can be used to build rustdocs. The setup is compatible with Cargo and Bazel. Docs may also be tested via `bazel test docs`. This commit doesn't publish the docs and doesn't fully surface all configuration. It provides some basic infrastructure as a starting point for future enhancments to autogenerated documentation. Co-authored-by: Blake Hatch <blakewihatch@gmail.com> Co-Authored-By: Blake Hatch <blakewihatch@gmail.com>

2024-06-06

Add initial support for BEP (Build Event Protocol) (#961) Add proto definitions for BEP and create `BepServer` which implements the `publish_lifecycle_event` and `publish_build_tool_event_stream` rpcs Closes #925 Co-authored-by: Nathan (Blaise) Bruer <github.blaise@allada.com>

2024-05-18

[Breaking] Digest function now auto-detected from request (#899) The digest function the client requested is now properly implemented in the OriginContext. This allows us to inspect what digest-function the client requested from anywhere in the code. If you use `verify` store and use `hash_verification_function`, simply change it to `"verify_hash": true`. We will now auto-detect the hash function based on the request and if it is not set, we use the `default_digest_hash_function` in the `global` config.

2024-04-08

Implement worker api for killing running actions (#840) Implements worker api for requesting a currently running action to be killed. Allows for action cancellation requests to be sent from the scheduler during scenarios such as client disconnection. towards #338 Co-Authored-By: Zach Birenbaum <zach@tracemachina.com>

2024-02-09

Adjust all instances of Native Link in comments and metadata to NativeLink (#658)

2023-12-15

Move custom crates to nativelink dirs (#483)

2023-12-12

Renames the project to nativelink (#463)

2023-11-24

Fix the incorrect config path in the documentation (#416)

2023-11-23

Rewrite the build infrastructure (#394) Our previous build layout was fairly custom and hard to maintain. Migrate to a more standards-conform layout that is compatible with good Cargo and Bazel practices and feels natural to develop in. This commit swaps the previous "one-crate-per-file" layout to a layout where we have a few medium sized crates that are just fine-grained enough to allow almost the same amount of parallelization as the previous layout. The new layout enables dependency auto-updaters, standard cargo tooling and doc autogeneration. This should improve DX quite a bit and gives us some much needed compatibility with nix/cargo interop and should make most standard cargo extensions compatible with the repository. Some notable changes to the file layout: - The previous `error` crate is no longer part of `utils`. It is now a standalone module. This was necessary to keep macro exports compatible with the Cargo build. - The old `scheduler.rs` file is now split into `action_scheduler.rs` and `worker_scheduler.rs` since we don't want imports like `native_link_scheduler::scheduler`. - Some tests now use a slightly different import mechanism for test-only dependencies. - The store traits are now in `native_link_utils`, as it was used virtually everywhere. Moving this up in the dependency graph allows for higher parallelization of the other crates. Added documentation targets for every crate (except the `cas` binary itself, as this will require a dedicated commit). These doc targets can be viewed by pointing a browser to `bazel-bin/thetarget.rustdoc/thetarget/index.html`. Added rustfmt configuration to standardize imports. We now use `crate::` syntax which plays well with rusfmts's import formatting. The new layout makes it easier to spot crate/directory interdependencies so that we can remove them in future commits.

2023-11-19

Add default pre-commit hooks (#405) These have mostly no effect and just make sure that trailing newlines are used properly, linebreaks are linux-like and trailing whitespace is removed.

2023-11-18

Rename Turbo Cache to Native Link (#402) This project has expanded well beyond build caching.

2023-11-13

Rename proto organization to trace_machina (#389)

2023-11-01

[Breaking Change] Message field can now be populated (#361) Breaking Change: This is a breaking change because the worker config's action cache store is now inside the "upload_action_result" property. Only the config file should need to be changed. We now allow users to configure how the message field is populated for ActionResult and added HistoricalExecuteResponse that is compatible with bb_browswer. closes #333 closes #331

2023-09-12

Update remote_execution.proto to v2.3

2023-07-16

Removes name from action state and make Operation::name schema We already have a unique way to reference a task `ActionInfoHashKey`. This PR removes the need for name in the ActionState and adds `unique_qualifier` to this field instead. This PR also adds the utilities needed to convert to-from a string, so components can reference the task by a unique name. `instance_name` is also now encoded into the hash key. The Operation's name can now be decoded into a format that will make it easy to route the request to the proper action on a scheduler or worker for future PRs.

2023-06-29

Add timestamps to executor jobs.

2022-10-16

Change license to Apache 2 license

2022-06-21

Scheduler will retry on internal errors Internal errors from the worker (not exit code errors) are now properly forwarded up to the client, but after some retries by the scheduler.

2022-04-18

Adds ability in proto for worker to return internal errors Updates the worker_api.proto so it can return error codes for internal errors in the event that an internal error occurred.

2022-04-15

Implement execution_response() in WorkerApiServer Implements the execution_response() function. This will allow workers to let the scheduler and thus the client know a job has an updated state (usually completed or errord). This required a few API changes, but nothing dramatic.

2022-04-12

Add support for keep alive for workers The scheduler will now support ability to evict workers if they timeout. The WorkerApiServer will now refresh worker timeouts based on keep_alive request being sent from the worker. If the worker was evicted and was executing a job the job will now be rescheduled.

2022-04-11

[RE] Various changes to worker_api.proto * ConnectWorker is no longer stream<->stream, instead has function calls and returns a stream for server->client updates. * Added `reserved` keyword representing each `NextId`. * Added ConnectionResult so worker itself knows it's own ID.

2022-04-08

Rename project turbo-cache

2021-11-26

Add more basic scheduler support Introduces most of the basic primitives for the scheduler and some of the logic with matching actions to workers along with a few tests.

2020-12-28

Add bytestream server scaffolding

2020-12-25

CAS server moved to cas directory

Add dependent proto files for bazel cas

2020-12-24

Initial commit