Proto commits in TraceMachina/nativelink

These 38 commits are when the Protocol Buffers files have changed:

2025-04-18

Initial Remote Asset support (#1646)

The documentation is generated from this commit.

2025-03-28

Add RemoteAsset protobuf (#1647)

2025-02-06

Add ability to prefix worker_id in config (#1578) WorkerId can now be prefixed with a config. This will cause all internal WorkerIds to be prefixed with this config followed by a generated UUIDv6.

Add OriginEvent for scheduler scheduling action (#1574) * Add OriginEvent for scheduler scheduling action Adds an OriginEvent for when the scheduler schedules an action. This can be used for external systems to give visuals to what is happening inside the system or for debugging. * Expose OriginContextCollector to scheduler's matching engine Ensures OriginContext data is available to the matching engine and when dealing with an action, the first user to invoke the action will be associated with the creation of the operation.

2025-01-27

Add `Closed` stream event to OriginEvents (#1570) Adds an event for when a stream is closed.

2024-12-11

Add OriginEventPublisher (#1497) Adds ability for nativelink to publish events to a store/database that can be used by another service to aggregate and/or visualize what is happening related to requests, users or other related metadata. By default bazel's metadata will be published with every event if present giving the ability to combine with BEP data.

2024-07-28

Significantly reduce Bazel test time (#1210) This massive compile time improvement comes from a few evil platform tricks to fetch a precompiled protoc. This saves 713 compile actions. Observed speedups for clean builds are between 20% and 40%.

2024-07-26

Upgrade toolchains (#1191) - Bump flake - Regenerate LRE toolchains - Update native-cli pulumi build - Bump nightly Rust and fix exposed issues - Fix a misplaced aarch64-linux system configuration in flake (typo fix, doesn't add support for any new targets)

2024-07-17

[Refactor] Overhaul of scheduler component (#1169) This is a significant overhaul to Nativelink's scheduler component. This new scheduler design is to enable a distributed scheduling system. The new components & definitions: * AwaitedActionDb - An interface that is easier to work with when dealing with key-value storage systems. * MemoryAwaitedActionDb - An in-memory set of hashmaps & btrees used to satisfy the requirements of AwaitedActionDb interface. * ClientStateManager - A minimal interface required to satisfy the requirements of a client-facing scheduler. * WorkerStateManager - A minimal interface required to satisfy the requirements of a worker-facing scheduler. * MatchingEngineStateManager - A minimal interface required to satisfy a engine that matches queued jobs to workers. * SimpleSchedulerStateManager - An implements that satisfies ClientStateManager, WorkerStateManager & MatchingEngineStateManager with all the logic of the previous "SimpleScheduler" logic moved behind each interface. * ApiWorkerScheduler - A component that handles all knowledge about workers state and implmenets the WorkerScheduler interface and translates them into the WorkerStateManager interface. * SimpleScheduler - Translation calls of the ClientScheduler interface into ClientStateManager & MatchingEngineStateManager. This component is currently always forwards calls to SimpleSchedulerStateManager then to MemoryAwaitedActionDb. Future changes will make these inner components dynamic via config. In addition we have hardened the interactions of different kind of IDs in NativeLink. Most relevant is the separation & introduction of: * OperationId - Represents an individual operation being requested to be executed that is unique across all of time. * ClientOperationId - An ID issued to the client when the client requests to execute a job. This ID will point to an OperationId internally, but the client is never exposed to the OperationId. * AwaitedActionHashKey - A key used to uniquely identify an action that is not unique across time. This means that this key might have multiple OperationId's that have executed it across different points in time. This key is used as a "fingerprint" of an operation that the client wants to execute and the scheduler may decide to join the stream onto an existing operation if this key has a hit. Overall these changes pave the way for more robust scheduler implementations, most notably, distributed scheduler implementations will be easier to implement and will be introduced in followup PRs. This commit was developed on a side branch and consisted of the following commits with corresponding code reviews: 54ed73cc6fcaf9236a544e45a90b9071b5df6ad8 Add scheduler metrics back (#1171) 50fdbd73983dfdbd2620ac6bfcd5c7b642b5705e fix formatting (#1170) 89262366544d2310e0e9a5bd91c99297112d4e27 Merge in main and format (#1168) 9c2c7b9d3a2bfd71615fe46893ac937e6b23974c key as u64 (#1166) 0192051fca6e3509a6b2953ab53f77c12b4c7092 Cleanup unused code and comments (#1165) 080df5d0c559055d12ac53275024d32e35077675 Add versioning to AwaitedAction (#1163) 73c19c41e31fef55a3c6e81beb19a7233f0c2e8d Fix sequence bug in new memory store manager (#1162) 6e50d2c16fad92f2f8fb856b199f7798c30d5967 New AwaitedActionDb implementation (#1157) 18db991e3a414c3f95c2dd5a14bfafce80358de9 Fix test on running_actions_manager_test (#1141) e50ef3c27d16fc840fd63b917f39f1c317f48398 Rename workers to `worker_scheduler` 1fdd5056f357124d6aff63fe145821b795cf9780 SimpleScheduler now uses config for action pruning (#1137) eaaa872bc62a952fb926214a6fb11bd4ffd25bae Change encoding for items that are cachable (#1136) d64705627c045d108340546a3ea135edb984163a Errors are now properly handles in subscription (#1135) 7c3e730e30e96fb5d08e4d43cca8d27593bcea03 Restructure files to be more appropriate (#1131) 5e98ec9f67f56744376e060b6aa423f1ff401a47 ClientAwaitedAction now uses a channel to notify drops happened (#1130) 52beaf9c727cd396ce4b5b4095bc8dc7be8ce940 Cleanup unused structs (#1128) e86fe0826b3512e78804744a8f1db2ec4c5b21ea Remove all uses of salt and put under ActionUniqueQualifier (#1126) 3b860367ba131c918127b6426e791b9ca0ca8b6b Remove all need for workers to know about ActionId (#1125) 5482d7f5e5f84c2ea80d79c703624c73b6ee36cc Fix bazel build and test on dev (#1123) ba52c7fb76bba56ff5485b91cd516712f867dc64 Implement get_action_info to all ActionStateResult impls (#1118) 2fa4fee48374ed0cf12dcd88225d6d208c0eeeb2 Remove MatchingEngineStateManager::remove_operation (#1119) 34dea0633f8b5198d9c51904beee8f4c617fc822 Remove unused proto field (#1117) 3070a40935c1a490b530af8551329c44bb4500ff Remove metrics from new scheduler (#1116) e95adfcd6035d3cecac74e67829b83e44a598228 StateManager will now cleanup actions on client disconnect (#1107) 6f8c001d1a96a0dc74a5c22716ba2f54e4c9f1b1 Fix worker execution issues (#1114) d353c3003437609547c5dcc8738637b13151a9b5 rename set_priority to upgrade_priority (#1112) 0d93671ba51be41a1a6b0297260cfc5ca0766ebc StateManager can now be notified of noone listeneing (#1093) cfc0cf62c673dc399938247b0c3abe55863c2687 ActionScheduler will now use ActionListener instead of tokio::watch (#1091) d70d31d74cdc337861ebddd8d19201b088b8faa6 QA fixes for scheduler-v2 (#1092) f2cea0cc3fcc259b9a4b7257f2eae807d7e99d05 [Refactor] Complete rewrite of SimpleScheduler 34d93b7d1cda7c0f4a430364b7588002ab612494 [Refactor] Move worker notification in SimpleScheduler under Workers b9d970264b59201e13053844a277092164c8416a [Refactor] Moves worker logic back to SimpleScheduler 7a16e2e6043b17e7813e41450b4de9c40de435f4 [Refactor] Move scheduler state behind mute

Update copyright headers (#1172)

2024-06-08

Introduce basic rustdoc infrastructure (#980) This commit adds a new `docs` command to the flake which can be used to build rustdocs. The setup is compatible with Cargo and Bazel. Docs may also be tested via `bazel test docs`. This commit doesn't publish the docs and doesn't fully surface all configuration. It provides some basic infrastructure as a starting point for future enhancments to autogenerated documentation. Co-authored-by: Blake Hatch <blakewihatch@gmail.com> Co-Authored-By: Blake Hatch <blakewihatch@gmail.com>

2024-06-06

Add initial support for BEP (Build Event Protocol) (#961) Add proto definitions for BEP and create `BepServer` which implements the `publish_lifecycle_event` and `publish_build_tool_event_stream` rpcs Closes #925 Co-authored-by: Nathan (Blaise) Bruer <github.blaise@allada.com>

2024-05-18

[Breaking] Digest function now auto-detected from request (#899) The digest function the client requested is now properly implemented in the OriginContext. This allows us to inspect what digest-function the client requested from anywhere in the code. If you use `verify` store and use `hash_verification_function`, simply change it to `"verify_hash": true`. We will now auto-detect the hash function based on the request and if it is not set, we use the `default_digest_hash_function` in the `global` config.

2024-04-08

Implement worker api for killing running actions (#840) Implements worker api for requesting a currently running action to be killed. Allows for action cancellation requests to be sent from the scheduler during scenarios such as client disconnection. towards #338 Co-Authored-By: Zach Birenbaum <zach@tracemachina.com>

2024-02-09

Adjust all instances of Native Link in comments and metadata to NativeLink (#658)

2023-12-15

Move custom crates to nativelink dirs (#483)

2023-12-12

Renames the project to nativelink (#463)

2023-11-24

Fix the incorrect config path in the documentation (#416)

2023-11-23

Rewrite the build infrastructure (#394) Our previous build layout was fairly custom and hard to maintain. Migrate to a more standards-conform layout that is compatible with good Cargo and Bazel practices and feels natural to develop in. This commit swaps the previous "one-crate-per-file" layout to a layout where we have a few medium sized crates that are just fine-grained enough to allow almost the same amount of parallelization as the previous layout. The new layout enables dependency auto-updaters, standard cargo tooling and doc autogeneration. This should improve DX quite a bit and gives us some much needed compatibility with nix/cargo interop and should make most standard cargo extensions compatible with the repository. Some notable changes to the file layout: - The previous `error` crate is no longer part of `utils`. It is now a standalone module. This was necessary to keep macro exports compatible with the Cargo build. - The old `scheduler.rs` file is now split into `action_scheduler.rs` and `worker_scheduler.rs` since we don't want imports like `native_link_scheduler::scheduler`. - Some tests now use a slightly different import mechanism for test-only dependencies. - The store traits are now in `native_link_utils`, as it was used virtually everywhere. Moving this up in the dependency graph allows for higher parallelization of the other crates. Added documentation targets for every crate (except the `cas` binary itself, as this will require a dedicated commit). These doc targets can be viewed by pointing a browser to `bazel-bin/thetarget.rustdoc/thetarget/index.html`. Added rustfmt configuration to standardize imports. We now use `crate::` syntax which plays well with rusfmts's import formatting. The new layout makes it easier to spot crate/directory interdependencies so that we can remove them in future commits.

2023-11-19

Add default pre-commit hooks (#405) These have mostly no effect and just make sure that trailing newlines are used properly, linebreaks are linux-like and trailing whitespace is removed.

2023-11-18

Rename Turbo Cache to Native Link (#402) This project has expanded well beyond build caching.

2023-11-13

Rename proto organization to trace_machina (#389)

2023-11-01

[Breaking Change] Message field can now be populated (#361) Breaking Change: This is a breaking change because the worker config's action cache store is now inside the "upload_action_result" property. Only the config file should need to be changed. We now allow users to configure how the message field is populated for ActionResult and added HistoricalExecuteResponse that is compatible with bb_browswer. closes #333 closes #331

2023-09-12

Update remote_execution.proto to v2.3

2023-07-16

Removes name from action state and make Operation::name schema We already have a unique way to reference a task `ActionInfoHashKey`. This PR removes the need for name in the ActionState and adds `unique_qualifier` to this field instead. This PR also adds the utilities needed to convert to-from a string, so components can reference the task by a unique name. `instance_name` is also now encoded into the hash key. The Operation's name can now be decoded into a format that will make it easy to route the request to the proper action on a scheduler or worker for future PRs.

2023-06-29

Add timestamps to executor jobs.

2022-10-16

Change license to Apache 2 license

2022-06-21

Scheduler will retry on internal errors Internal errors from the worker (not exit code errors) are now properly forwarded up to the client, but after some retries by the scheduler.

2022-04-18

Adds ability in proto for worker to return internal errors Updates the worker_api.proto so it can return error codes for internal errors in the event that an internal error occurred.

2022-04-15

Implement execution_response() in WorkerApiServer Implements the execution_response() function. This will allow workers to let the scheduler and thus the client know a job has an updated state (usually completed or errord). This required a few API changes, but nothing dramatic.

2022-04-12

Add support for keep alive for workers The scheduler will now support ability to evict workers if they timeout. The WorkerApiServer will now refresh worker timeouts based on keep_alive request being sent from the worker. If the worker was evicted and was executing a job the job will now be rescheduled.

2022-04-11

[RE] Various changes to worker_api.proto * ConnectWorker is no longer stream<->stream, instead has function calls and returns a stream for server->client updates. * Added `reserved` keyword representing each `NextId`. * Added ConnectionResult so worker itself knows it's own ID.

2022-04-08

Rename project turbo-cache

2021-11-26

Add more basic scheduler support Introduces most of the basic primitives for the scheduler and some of the logic with matching actions to workers along with a few tests.

2020-12-28

Add bytestream server scaffolding

2020-12-25

CAS server moved to cas directory

Add dependent proto files for bazel cas

2020-12-24

Initial commit