Proto commits in vercel/turborepo

These 46 commits are when the Protocol Buffers files have changed:

2024-08-08

chore(repo): update readme and links (#8965) ### Description Update the README, and update repository links ### Testing Instructions <!-- Give a quick description of steps to test your changes. -->

The documentation is generated from this commit.

2024-04-19

feat(turborepo): Watch mode (#7613) ### Description Implements an MVP version of watch mode that takes the package changes stream and re-runs package tasks as needed. ### Testing Instructions <!-- Give a quick description of steps to test your changes. --> Closes TURBO-2519

2024-04-18

feat(Turborepo): wire file hashing to grpc server (#7868) ### Description - Wire up daemon-backed file hashing into the client ### Testing Instructions Existing integration test suite Closes TURBO-2736

2024-04-04

feat(turborepo): Package Change Watching (#7570) ### Description Added a watcher in `FileWatching` to watch for file changes and map them to packages. Currently you can run this via `turbo daemon watch` but eventually this will be used for watch mode. ### Testing Instructions Run the daemon and use `turbo daemon watch` to see the changed package events as you change files. Closes TURBO-2499

2024-03-29

fix(pnpm): support new pnpm9 default link-workspace-packages (#7865) ### Description With [PNPM 9](https://github.com/pnpm/pnpm/releases/tag/v9.0.0-alpha.0), [link-workspace-packages]() is now defaulting to `false` meaning that unless the `workspace` protocol is explicitly used, packages from the registry will be preferred over workspace packages. It's odd to have 3 variants of the PNPM package manager, but that is the current state of the world. Unsure of why we changed from a generic interface to a enum in the Go->Rust port, but we'll probably want to go back to that world to reduce match statement complexity. ### Testing Instructions Unit tests for package manager detection. Quick spot check in a `create-turbo` repo with workspace deps updates so they don't use `workspace` protocol. Closes TURBO-2733

2024-02-27

Completed first pass at package change events stream

2024-02-20

Audit daemon error paths and ensure we are doing everything we can to handle (#7334) ### Description We have a few integration points with the daemon, this PR ensures they are all nicely handled. - [x] `cache.rs`: we emit a debug warning and a telemetry event in the case we are unable to mark outputs correctly - [x] `lsp/lib.rs`: in the LSP, package discovery means we cannot fulfill the user's request, so we must return an internal error. the error message has been set. Additionally, the LSP is a case where waiting for an answer is actually desirable, so a new API to support this has been added. - [x] package discovery: if we have a bug in watching, we may yield packages that don't exist. we need to report a telemetry event, in this case so that we can be aware of when this happens - [x] daemon interaction in run codepath: when trying to use the daemon during run we want to debug log but not error - [x] daemon connection in daemon codepath: when calling the daemon we want errors to be explicitly reported. in this case, the errors bubbles all the way up and is printed out as expected ### Testing Instructions <!-- Give a quick description of steps to test your changes. --> Closes TURBO-2319 --------- Co-authored-by: Alexander Lyon <Alexander Lyon>

2024-02-08

chore(turborepo): Ensure fresh builds (#7321) ### Description This had been bugging me for a while. When building turbo, even with no changes, we'd still rebuild turborepo-lib and turbo. Turns out this is cause we were generating `file_descriptor_set.bin` into the same directory we were watching for protobuf changes. To avoid this, I put the protobuf files in a nested directory. ### Testing Instructions Compile and be amazed at the speed! Closes TURBO-2294

2024-01-30

refactor: Deleting Go code (#6956) ### Description LFG ### Testing Instructions <!-- Give a quick description of steps to test your changes. --> Closes TURBO-2011

Deleting Go code

2024-01-08

Working on gRPC API, converting watcher to use events instead of state

Starting on gRPC API

Updated proto

2023-11-29

feat: add package discovery trait and a few discovery strategies (#6275) This adds a trait for discovering packages (paths to package and turbo jsons) using a variety of strategies: - **local**: runs in-process and does a recursive walk of the FS - **watch**: runs a file watching thread maintains a list in memory - **daemon**: makes a request to a remote service over GRPC - **timeout**: attempts a primary strategy and falls back in the event of errors or a timeout

2023-11-22

refactor: switch daemon version to be independent of turbo version (#6528) This change is backwards compatible, since we set the starting version of the daemon protocol to be equal to the current turbo version meaning we will not get any clashes with old clients reporting a bad version. Even though the version is currently equal to the turbo version, it is not expected to always be the case. ### Description If we want to allow other tools to interact with the turbo daemon, we should not be as strict with our versioning. As it stands, a 3rd party tool such as an LSP would need to perfectly match the version of turbo used in the repo which is an unreasonable constraint. This also relaxes the version requirement for all _new_ versions to least the same minor version (ie. that all the features available to the client are implemented by the server). Old clients will still demand identical versions (and have that demand respected correctly). ### Testing Instructions Unit tests for the semver check included.

2023-09-15

feat(turborepo): support Bun (#5934) Enable using `bun` as both a package manager and a runtime. Closes #4762 Closes TURBO-1324

2023-06-23

Implement git-based hashing with inputs (#5321) Co-authored-by: Greg Soltis <Greg Soltis>

2023-06-13

feat(lockfiles): add rust implementation for yarn1 (#5255) Co-authored-by: Alexander Lyon <arlyon@me.com> Co-authored-by: Chris Olszewski <Chris Olszewski>

2023-06-09

feat(turborepo): port git-based hashing with inputs (#5251) Co-authored-by: Greg Soltis <Greg Soltis> Co-authored-by: Chris Olszewski <chris.olszewski@vercel.com>

feat(turborepo): Port Global Hash Env Var (#5166) ### Description Ports env.go to Rust and connects it to the global hash code. ### Testing Instructions I added `global_hash_test.go` and connected the `env_test.go` tests to the Rust code. --------- Co-authored-by: --global <Nicholas Yang> Co-authored-by: Mehul Kar <mehul.kar@vercel.com>

2023-06-08

feat(turborepo): Port Manual hashing (#5237) Co-authored-by: Greg Soltis <Greg Soltis>

2023-05-31

Include timeSaved metric when skipping cache check (#4952) After this change, the `timeSaved` metric will be included in: `--summarize` otutput file when the daemon does not need to restore cache and will also be sent to Spaces. Co-authored-by: Nicholas Yang <nicholas.yang@vercel.com> Co-authored-by: Chris Olszewski <chris.olszewski@vercel.com>

2023-05-26

feat: rust pnpm lockfile implementation (#4906) ### Description This PR is a fairly faithful Rust port of the Go implementation. There are two intentional differences: - A few fields in the lockfile are no longer serialized in a flow. This is a short coming of `serde_yaml` (https://github.com/dtolnay/serde-yaml/issues/303). Luckily pnpm doesn't care, but it does mean that the outputs between Go and Rust aren't byte identical - I didn't port [convertLockfileV6DepPathToV5DepPath](https://github.com/pnpm/pnpm/blob/e2293bd6e6033cd26a7e58e151b2770cfd1699d8/lockfile/lockfile-file/src/experiments/inlineSpecifiersLockfileConverters.ts#L162) to Rust and instead added functionality to the `DepPath` parser to handle both v5 and v6 paths. Reviewer notes: - Start with `dep_path.rs`, this is primarily the parsing of "dependency paths" which is pnpm's term for lockfile keys. They consist of a package name, version, and sometimes a suffix which can contain the resolved versions of peer dependencies and/or patches. - `data.rs` contains all of the structure for the lockfile as well as the logic for traversing the lockfile - `se.rs`/`de.rs` contain some custom logic for handling the `lockfileVersion` field - All helper methods of `PnpmLockfile` in Go got ported to Rust using snake case ### Testing Instructions Ported all existing unit tests. Existing integration and e2e tests pass. Manual testing that `turbo prune` output can be used with `pnpm install --frozen-lockfile`. --------- Co-authored-by: Chris Olszewski <Chris Olszewski>

fix(turborepo): rust implementation of file hashing via git index (#4967) Co-authored-by: Greg Soltis <Greg Soltis> Co-authored-by: Nicholas Yang <nicholas.yang@vercel.com>

2023-05-18

Revert "Partial revert of #4820. (#4938)" This reverts commit f51b85b5d5ea69731218c2da743fec23822302ce.

refactor(turborepo): Signature Authentication (2nd try) (#4980) ### Description Refactors the signature authentication to use a language-independent algorithm, as the previous implementation depended on Go's JSON serialization. NOTE: This is a global hash bump. ### Testing Instructions Testing is using fuzzing across the Go-Rust boundary to check that signatures are correctly generated on both sides. --------- Co-authored-by: --global <Nicholas Yang>

2023-05-16

Added testing to confirm that Go and Rust implementations work the same

2023-05-15

Partial revert of #4820. (#4938) This is a partial revert of 78aa3339bc21090c5c6c74f45bde776292ac8196 in order to correct non-`git` fallback behavior. 1. It does not revert changes made inside of Rust `turbopath` as those do not apply cleanly; but they are also not relevant to the issue introduced by #4820. 2. The changes to `Cargo.lock` are the consequences of the new dependency graph. Tested for correctness locally, this restores behavior to what we saw prior to the commit landing. It's likely a 1-2 line change to correct behavior but I didn't attempt to fix-forward. This will be straightforward to land again once we have additional testing in place. Co-authored-by: Nathan Hammond <Nathan Hammond>

2023-05-10

First pass at file hashing for a package (#4820) Co-authored-by: Greg Soltis <Greg Soltis> Co-authored-by: Chris Olszewski <chris.olszewski@vercel.com> Co-authored-by: Nicholas Yang <nicholas.yang@vercel.com>

2023-05-01

Port RecursiveCopy to Rust (#4719)

Revert "refactor(turborepo): cache signature authentication" (#4750) Reverts vercel/turbo#4686 so we can coordinate global hash bump

2023-04-28

refactor(turborepo): cache signature authentication (#4686) ### Description Refactors the cache signature authentication code to no longer rely on Go implementation details. This requires a cache bump as it is a **breaking** change to cache signatures. ### Testing Instructions Existing cache signature authentication now calls the Rust implementation and confirms that it produces the same tag. There are also new tests that randomly generate test cases and confirm that the Rust and Go implementation agree on the tag output.

2023-04-27

feat: Use Rust Berry lockfile impl (#4684) ### Description Built on top of #4629 Overview of changes: - ~~Change `subgraph` so we calculate workspace's closure in parallel in a similar manner to Go~~ This had to get reverted, it appears that rayon was causing [compilation issues on linux](https://github.com/vercel/turbo/actions/runs/4791974785/jobs/8522988391#step:4:648) - `npm_subgraph` -> `subgraph` makes this FFI call generic - Addition of an optional `resolutions` map to the `subgraph` and `transitive_closure`, this is only used by Berry. In the ideal world this would be all of the root `package.json`, but serializing all of that via protobuf is overkill since berry is the only lockfile that needs that info and we only use that field. - Addition of `patches` FFI function, this wasn't needed before as NPM doesn't have patch files encoded in the lockfile - Addition of `global_changes` FFI function, previously for NPM we were just doing this in Go since there were just two fields we needed to check and parsing JSON is cheap. - Switches lockfile usage of berry from the Go implementation to the Rust one introduced in #4589 Reviewer notes: - The passing of the optional resolutions map is very icky and one off, but this is the only lockfile the requires additional information to properly parse. Once the package graph is squarely in Rust land this should go back to being more generic. This will finally close out #2791 ### Testing Instructions Added a new integration test that uses resolutions. This integration test invokes the new `patches` FFI call. For a full test of the new resolution feature, clone and follow the instructions in the [repro](https://github.com/erj826/turbo-prune-bug-repro-11-21) The `lockfile_aware_caching` integration tests all hit the new `global_changes` FFI callsite and provide test coverage. --------- Co-authored-by: Alexander Lyon <arlyon@me.com>

2023-04-26

chore: move lockfile ffi boundary (#4629) ### Description In order to avoid Go holding onto memory that's been allocated by Rust we parse the lockfile each time we need to make a lockfile call. This worked fine for the npm implementation as we just needed parse the JSON and it was usable, but Berry requires a lot more work/allocations that makes this strategy unfeasible. A quick sketch of the changes in this PR: - Moves the traversal of the lockfile to be the final step of building the package graph - We now calculate all of the closures of all workspaces at once - Rust FFI now takes the package manager string to select the lockfile implementation. This is a quick prefactor to prepare for hooking up the berry lockfile Reviewer notes: - Each commit can be reviewed on it's own, the first commit ended up getting reverted down the stack so it can be skipped. ### Testing Instructions Existing unit tests/integration tests. Manual verification of correct pruning behavior for the npm and pnpm monorepos created by `create-turbo@latest`

2023-04-20

fix(turborepo): Switching back to git command instead of git2 (#4606) ### Description Once #4604 is merged and we release 1.9.2, we can merge this with additional tests ### Testing Instructions Added test for shallow cloned repo that fails on main

2023-04-17

Revert "fix(turborepo): SCM tests and renaming (#4462)" (#4604) ### Description <!-- ✍️ Write a short summary of your work. If necessary, include relevant screenshots. --> This reverts commit 1cbba8332911944489962bcde2155fd23f16d12a due to a bug in shallow cloned repositories ### Testing Instructions <!-- Give a quick description of steps to test your changes. -->

2023-04-11

fix(turborepo): SCM tests and renaming (#4462) ### Description Added more testing to SCM and renamed variables to be more understandable (repo_root -> git_root, monorepo_root -> turbo_root) ### Testing Instructions Added tests on Rust side for deleting files and adding file to index.

2023-03-31

Vc<T>

2023-03-22

feat: port npm lockfile to Rust (#3632) Adds a Rust port of the npm lockfile implementation only used when `GO_TAG=rust` is set. Notes: - The crawling of the lockfile graph is done in the language of the lockfile impl to avoid incurring ffi overhead for every transitive dependency. This change requires moving this function into the lockfile package instead of context in Go (this is probably where it should've belonged all along) - We explicitly make an ffi function in Rust for each lockfile implementation. This adds boilerplate, but as this boundary should be short lived (only around until package graph gets moved into Rust) this is acceptable instead of coming up with a more generic solution. - If we stick with passing the lockfile contents for other implementations then we will be able to share the protobuf message definitions --------- Co-authored-by: Alexander Lyon <arlyon@me.com>

2023-03-17

refactor: Removed includeUntracked because it's always set to true (#4243) ### Description I realized that we only call `ChangedFiles` in one place and in that place `includeUntracked` is always set to true. So I just removed it as a parameter. ### Testing Instructions <!-- Give a quick description of steps to test your changes. -->

2023-03-14

fix: SCM relative paths in nested turborepo (#4184) ### Description <!-- ✍️ Write a short summary of your work. If necessary, include relevant screenshots. --> ### Testing Instructions <!-- Give a quick description of steps to test your changes. -->

2023-02-21

port(turborepo): SCM package (#3775) Co-authored-by: Chris Olszewski <chrisdolszewski@gmail.com>

2023-02-06

add first version of rgr sandwich (#3306)

2022-10-31

Correct all domain and repository references. (#2508) * Correct all domain references. * Update URL paths. * Adjust repository name. * Update external links. * Correct internal anchor links. * No more relative links. * Update infoLink URLs to not redirect. * Remove outdated links. * Update hashes.

2022-10-03

fix: Glob negation in outputs (#2031) Fixes glob negation in outputs, basically patterns that exclude directories like `!.next/cache/**`. Involves changes to protobuf types for `OutputWatcher`. Existing tests from `turbo_json_test.go` and `globby_test.go` should cover the cases.

2022-06-27

Watch task outputs to avoid cache restorations (#1269) Start a daemon, notify it of output globs when a task completes, watch those outputs, check if we can skip cache restores on subsequent runs. Uses `grpc` and `protobuf` for communication. Before shipping: - [x] Wire up `--no-daemon` flag to `run` - [x] Add command for `daemon status` - [x] Add command for `daemon stop` - [x] Add command for `daemon start` - [x] Add command for `daemon restart` - [x] Proper daemon logfile management - [x] Ensure that the client tries to start the server after maybe killing a previous incarnation - [x] Investigate golang internal lockfile - [ ] Verify ordering guarantees on file-watching - [ ] Sort out proper syntax for `protos` in `Makefile` - [x] Shut down if repo root is deleted - [x] Proper signal handling - [ ] Handle watching too many files - [ ] Consider using [panicwrap](https://github.com/mitchellh/panicwrap)