Proto commits in enfabrica/enkit

These 73 commits are when the Protocol Buffers files have changed:

2025-06-19

[Allocation Manager] Encapsulate topo requests (#1242) First phase of Allocation Manager, able to allocate by topology name, and with foundational support for further criteria such as num_hosts, etc --------- Co-authored-by: Kevin Wang <kjw@enfabrica.net>

The documentation is generated from this commit.

2025-06-04

Add tar support to muk (#1238) Adds support for untarring files during a docker create operation. Jira: ENGPROD-1198 Tested: build container, worked

2025-06-03

Add setting environment variables to muk image builder (#1235) Added new command to muk.py, muk.proto to set environment variables. Jira: ENGPROD-1198 Tested: locally

2025-06-02

ENGPROD-1197: test affected targets with `bzlmod`, (#1234) - make some hard failures just to print warning - update `workspace_log.proto` to latest version

2025-03-28

In progress: inviter prototype.

2025-01-11

auth/server: introduce more verbose logging (... for counters). (#1150) Look at the individual commits, especially at the last one. Tl;Dr: once the PR is merged, the auth server will start logging authentication related events in a format suitable to define log based metrics on gcp and app engine. This is in preparation for the next PR, that will improve the logic in auth.go to make it more resilient to misbehaving clients/applications, and less susceptible to connection exhaustion by avoiding long lasting connections. - astore, auth: consistently propagate and use logger. - auth/server/auth/factory.go: run of go fmt. - auth/server/auth/auth.go: make GetChannel private. - auth: add minimal documentation on auth.proto and server/auth/auth.go. - auth/server: start logging authentication requests and events.

2024-09-09

allocation manager first look (#1097) - At this point, the server compiles, and the client seems to reserve a dummy entry and release without crashing. Much more left to do, however. but commit'ing to checkpoint progress.

2024-06-14

Support Rocky Linux in muk (#1070) This change updates the `muk` tool to handle container builds based on the Rocky Linux distro. Tested: - `bazel run --override_repository=enkit=$HOME/enkit //infra/dev_container/mojave:mojave_base_image_builder -- --noclean_build_check` builds a new container from https://github.com/enfabrica/internal/pull/41200 JIRA: INFRA-8859

2024-04-18

Migrate muk and container rules from internal to enkit (#1050) This change migrates `muk` and container build/push rules from the internal repo to the enkit repo because there are some containers that need to be built in the enkit repo. The enkit repo should not import any bazel rules or libraries from the internal repo because this will cause a circular dependency. Tested: - `bazel test ...:all` - `bazel build ...:all` JIRA: INFRA-10368

2024-03-20

Copybara update from github.com/buildbuddy-io/buildbuddy (#1044) This change runs copybara on the current copybara definition to import old versions of buildbuddy protos. The BUILD files are manually changed to use the old protos but the new protobuf rules, rather than doing a straight rollback, as the new rules will serve us better moving forward. This won't be merged until: * [x] buildbuddy in staging is rolled back * [x] we've tested against the rolled-back buildbuddy in staging Tested: `bazel test //...` succeeds Jira: INFRA-10068 GitOrigin-RevId: 372b95b41a6ed82da632ee44ff0c48071c849c21 --------- Co-authored-by: Copybara <noreply@enfabrica.net>

2024-03-18

Copybara update from github.com/buildbuddy-io/buildbuddy to //third_party/buildbuddy (#1036) See //third_party/buildbuddy/README.md for more details. The proto files changed in this PR are done automatically via Copybara. BUILD files are updated manually to get this PR to build correctly. GitOrigin-RevId: aedaeafdc7b6bc9faaf77494a4598d750bafdf8c Jira: INFRA-9988 Co-authored-by: Copybara <noreply@enfabrica.net>

2024-03-14

Copybara update from github.com/buildbuddy-io/buildbuddy to //third_party/buildbuddy See //third_party/buildbuddy/README.md for more details. GitOrigin-RevId: 08bf678c4530eb5a367486f520dc802e219e33b6

2024-01-16

license manager: Delete unused code (#1014) `flextape` has replaced `manager`, but the latter is still present in the source tree. This change removes it so that it no longer needs to be maintained. The astore upload of the `manager` client has been replaced with one of `flextape_client`, to finish the migration from one to the other. Tested: `bazel test //...` still passes as much as it is expected to

2023-10-15

postsubmit: Add option to specify local disk space (#982) Some builds require extra local storage in order to run correctly. This change adds a field that allows for the amount of space required to be captured. Tested: N/A Jira: INFRA-8419

2023-10-09

postsubmit: Add support for debugging steps (#978) This change adds a mechanism for adding user-friendly debugging steps for determining why a postsubmit isn't behaving properly. The first step added is a "large file printer", which will print out the largest N files in a particular directory; this will be useful for debugging INFRA-8302. Tested: N/A Jira: INFRA-8302

2023-09-19

postsubmits: Add a container_image field to spec (#973) This change adds a field so that the container image + version can be manually overridden in postsubmit definitions. This will help us test an image in a specific postsubmit before pushing it to all postsubmits. Tested: N/A Jira: INFRA-7115

2023-08-30

auth: Move server code out of astore dir (#961) Moves auth server implementation, along with its dependencies out of the astore directory and places it in its own root 'auth' directory. This is in an attempt to make it a bit more easy to spin up a local astore server for development testing. This does not yet change the astore server from including the auth server. This is just to move the files out, and point imports to the new location. A followup PR will be created to move the auth server out of the astore binary. No code was altered except for import paths and accompanying build file import path changes as well as `deps` renames. Tested by building the entire repo (bazel build //...)

2023-05-12

ghinviter: gRPC working.

2023-05-11

lib/khttp/kgrpc: add new library for grpc server code. This new library takes some of the code we added in lib/server and plumbs it with khttp/server.go. The end result is: - An http server supporting gRPC in all the known flavors (HTTP2, H2C, HTTPS - with plain gRPC or gRPC web). - An http server that can be fully configured with all the khttp modifiers. For example, see the added tests for now.

2023-02-20

lib/bazel: implement an included_tags, to limit to a subset of targets. (#802) In this PR: - add support to only execute tests *including* at least one of the specified tags.

2023-01-23

bazel: Nuke all web frontend code, rules from orbit (#820) This change removes all frontend code, including: * the `//ui` dir * associated bazel rules/macros * loading of npm packages, frontend-related rules from WORKSPACE The catalyst for this is INFRA-4192 - use of these rules uses a `managed_directory` feature of WORKSPACE that is removed in bazel 6.0 and is thus blocking bazel updates. Fixing the UI code is: * probably time-consuming, to understand the nuances of JS/TS compilation, by looking at https://docs.aspect.build/guides/rules_js_migration/ * of little value, since none of this code is actually used in anything The only dependency that actually exists is on some font and CSS assets; these are inlined into the repo directly, with a note on how to update them if needed. This also has ancillary benefits: * reduced build times, since node modules don't need to be fetched * no build-out-of-sync-type errors that necessitated a `bazel clean --expunge && rm -rf ui/node_modules to resolve Tested: no Jira: INFRA-4192

2022-09-12

Libraries to write our own CODEOWNERS/OWNERS bot. Specifically, this PR includes: - an initial library to parse OWNERS/CODEOWNERS files as well as protocol buffers in our repository as owners files, and some tests. The thinking here is that we'd have backward compatibility with the existing CODEOWNERS file, and some simple syntax to support OWNERS file, as well as supporting a basic protocol buffer that allows full flexibility. The library - albeit "code complete/usable" - is unfinished, as I have not written the remaining tests, and there are some known bugs/missing features.

2022-08-31

enkit: Remove FUSE P2P FS commands (#745) These commands are not used or maintained, and require us to keep flaky tests alive. This change removes these features.

2022-07-11

astore: Move protos into package dirs (#702) This is part of a larger effort to make the repository `go get`-clean. This change moves protos around such that their import paths match their actual package paths in the repository. Their BUILD files are cleaned up such that `bazel run //:gazelle` creates no diffs, and gazelle doesn't complain about ambiguous imports. Tested: * `bazel run //:gazelle` creates no diffs * `bazel test //...` still works * `go get -u ./...` no longer complains about astore/rpc/* packages

2022-06-21

postsubmit: Add ability to configure buildbarn instance (#673) This change adds a configuration setting to postsubmits that will allow users to configure which buildbarn instance builds should go to. This will be used in the short-term to migrate builds to buildbarn on gcp01 while leaving the majority of builds running on buildbarn in MTV. Jira: INFRA-1284

postsubmits: Add method of disabling build in config (#660) This change adds a "disable" trigger that creates the build but prevents it from ever running. This can be used to disable builds on a temporary basis. Jira: INFRA-1232

2022-04-22

flextape: Raise and parameterize timeouts (#545) All durations that the service uses are moved to a "server" config section, so that they are no longer hard-coded. At the same time, timeouts are greatly raised to reduce the number of expirations that happen because the client is too slow to refresh an active allocation. The client is modified to attempt a refresh earlier into the server-specified duration, but given the raised timeouts its minimum refresh duration is raised from 1s to 5s. Jira: INFRA-607

2022-04-11

postsubmits: Add more necessary fields (#537) This change adds fields to the Build message that are necessary in order to migrate the two existing HW nightly regressions: * `fpga-fullchip` passes a bunch of arguments to `bazel test` that we need to preserve * `fpv-nightly` makes use of positive tags in `build_tag_filters`/`test_tag_filters`, and currently the proto message only handles negative tags. Jira: INFRA-875

2022-04-08

postsubmit: Add timeout field (#536) This change adds a timeout field to the Build spec, which is the only missing piece to onboard the FPGA nightly regression. Tested: None Jira: INFRA-876

2022-02-18

bestie: Add py_proto_library target for test_metrics.proto (#471) The enkit repo owns the definition of test_metrics.proto and needs to produce each of the language-specific protobuf libraries being used, currently Go and Python. The latter is @enkit referenced by the internal repo for writing metrics to its test.metrics.pb file. Summary of changes: - Added py_proto_library() target in bestie/proto/BUILD.bazel. - Renamed KeyValuePair submessage to Tag in test_metrics.proto file. Tested: Using a local instance of BES Endpoint, sent simulated test metrics in a test.metrics.pb file for processing by the endpoint. Required use of the --override_repository command line arg by the client 'bazel test' command to reference the changes in the enkit repo. Jira: INFRA-593

2022-02-16

flextape: introduce prioritization support. (#458) Background: Before this PR, licenses were always allocated in FIFO order. This addresses INFRA-501. In this PR: - allow to specify a prioritization strategy (per license). - define an interface to provide different prioritization strategies, refactor the code to allow for the plumbing. - create a strategy that spreads licenses evenly across users. - added extensive tests. Note that this PR still does not allow to allocate multiple licenses at once. The rough plan to allow for that is to expand the concept of "invocation", so it links to all the types it is trying to allocate, and extend the prioritizer to account for that. For now, I just wanted to allow better license spreading, and figure out a reasonable interface for queue reordering. Tested: added significant unit tests, all passing.

2022-02-04

bestie: Add test metrics upload to BigQuery (#423) (#429) * bestie: Add test metrics upload to BigQuery (#423) This continues the work of #394 and #406 by adding support for uploading test metrics to BigQuery. The changes in this PR do the following: - Read and process metrics contained in *.metrics.pb file(s) produced during bazel test runs. Format and push these test metrics to the BigQuery 'bestie-builds.<dataset>.testmetrics' table, where <dataset> is specified by the --dataset command line parameter. - Supports both file:// and bytestream:// schemes for the outputs.zip file in the TestResult event, which correspond to local and cluster operation of the BES Endpoint, respectively, using APIs provided by enkit/lib/kbuildbarn. A new --baseurl command line parameter allows a deployment-specific base URL to be specified. - Adds several service metrics related to BigQuery interactions for monitoring the BES Endpoint operation. Tested: Ran BES Endpoint service both locally and in the staging cluster, where it detected the test.metrics.pb file produced by a standalone test app. Authentication issues prevented the staging cluster service from communicating with BigQuery, but that will be resolved in a separate PR by creating a new service account. Jira: INFRA-423 * bestie: Addressed code review comments Many changes in this commit to respond to the review comments in this PR. The most noteworthy of these are: - No longer creating non-existent datasets or tables. - Replaced all console print messages with a logger. - Added --debug support for additional log messages. - Revamped service stats implementation using counter IDs. - Using a goroutine to update Prometheus metrics.

2022-01-27

third_party: Import BuildBuddy protos (#411) This change uses Copybara to selectively import a few BuildBuddy protos needed to make HTTP RPC calls to fetch BES logs by invocation ID. Copybara is used to easily modify the protos as they are imported. This is necessary because even though BuildBuddy is itself a bazel workspace, it includes copied protos from the bazel repository that conflict with the same protos already imported under //third_party/bazel. Using Copybara allows for the import paths to be rewritten as they are imported. This change squashes the first import into the same change that adds the Copybara script, BUILD files, and README. Jira: INFRA-468

2022-01-26

Copybara update from github.com/buildbuddy-io/buildbuddy to //third_party/buildbuddy See //third_party/buildbuddy/README.md for more details. GitOrigin-RevId: 1a389a55e562cc3f49bb95ef1984831664ffc443

2022-01-19

bestie: Add test metrics processing to BEP server (#394) * bestie: Add test metrics processing to BEP server This change allows the BEP server to process TestResult events containing test metrics presented in the test_metrics.pb file that is incorporated in the outputs.zip file. The test_metrics.proto defines the protobuf message format, which reflects the data format needed by BigQuery for Prometheus consumption. A follow on PR is needed to continue this work, taking the test metric data and storing it in BigQuery. For now, the BEP server continues to display the Bazel event info same as before, and additionally displays the relevant metric information it finds in the TestResult events. Tested: Started server, ran 'bazel build //bestie/... --bes_backend=grpc://localhost:6433' and saw messages printed on stdout; bazel exited with no error. Jira: INFRA-326 * bestie: Improve error handling and propagation Changed the way the handler functions treat errors per Go best practices. Removed newline formatting from error messages that are propagated through fmt.Errorf(). Added comment for error handled within function (not propagated). Defined code owners of /bestie/ subtree for future reviewer notifications in GitHub. * bestie: Unzip test metrics directly into variable Instead of extracting the contents of outputs.zip into an exact replica of the directory and file structure, read the contents of test_metrics.pb and test_summary.pb into a byte slice variable and unmarshall the protobuf message from there. This eliminates contention for the temporary file name and is more efficient. Changed the file name search through outputs.zip to use regular expressions, accepting and processing all files named "test_metrics*.pb" and "test_summary*.pb", with the restriction that the * part begin with _ or -. For example, "test_metrics_foo.pb" or "test_summary-bar.pb". This is in anticipation that a given 'bazel test' command may need to output multiple metric protobuf messages, one per output file. * bestie: Protobuf message changes Changed test_metrics.proto to define the metrics tags field as repeated key/value pairs. Retained the BigQueryTable sub-message, but demoted it to an optional element, primarily for development testing. Cleaned up some protobuf API usage and defined a stream identity struct to carry information that uniquely identifies a stream. * bestie: Change metrics output filename regex Relaxed the regex used to find metrics files to process from the outputs.zip file. The general pattern is *.metrics.pb, with minor restrictions. Addressed another review comment related to passing a byte slice directly, instead of by pointer, when only reading from it.

2022-01-04

postsubmit: Define build configuration proto (#384) This change adds a documented protobuf that lays out the configuration for postsubmit builds. Each configuration will be an `infra.postsubmit.Build` message in its own configuration file; Terraform and other configuration will be generated from this high-level config and applied to create the builds in GCP. Tested: Builds Jira: INFRA-361

2021-12-21

Better support for working with React and Javascript (#316) * working example * huzzah it works * get ptunnel working * add in babel * remove commit * add in jest testing directly

2021-12-13

flextape: Return queue details in LicensesStatus (#366) This change extends the LicensesStatus RPC to show details on each invocaition that has been allocated, and each invocation that is queued. This should help us answer questions about where particular builds are queued if asked. Tested: Unit tests, ran server plus clients with ``` for i in $(seq 1 10); do (LICENSE_SERVER_IMPL=flextape bazel-bin/manager/client/client_/client localhost 6433 xilinx vivado sleep 60 &); done ``` and then `LicensesStatus` with: ``` grpc_cli call localhost:6433 flextape.proto.Flextape.LicensesStatus '' ``` and verified that `allocated_invocations` and `queued_invocations` were filled as expected. Jira: INFRA-287

2021-12-10

bestie: Implement logging BEP server (#344) This change implements the Build Event Protocol server in a minimal fashion, printing out all messages received on stdout in textproto format and acking them. To make this happen, protos are imported from the bazel repo via Copybara. This makes the protos easier to find during development, makes maintaining our custom BUILD files easier, and is faster during build as the bazel repo archive does not need to be fetched. This change does the following: * Adds a Copybara script for importing protos from the bazel repository * Imports protos under //third_party/bazel from the bazel repository using Copybara, modifying the import paths and adding go_package options that are required for our environment * Adds a README.md to //third_party/bazel that shows how it can be updated via Copybara * Adds me as a codeowner for //third_party to help with Copybara imports in the short-term * Replaces the "hello world" service with BEP in bestie, using handlers that log messages received to stdout. Tested: Started server, ran `bazel build //bestie/... --bes_backend=grpc://localhost:6433` and saw messages printed on stdout; bazel exited with no error. Jira: INFRA-193

2021-12-03

Copybara update from github.com/bazelbuild/bazel to //third_party/bazel See //third_party/bazel/README.md for more details. GitOrigin-RevId: 75a16b7107479441274fa983c04264cdf8556479

2021-11-23

bestie: Add initial skeleton (#331) This change adds enough of a gRPC and HTTP application to: * get something running on Google Cloud Run * develop Bazel rules to deploy a container to Google Cloud Run The initial gRPC service is a hello-world type service that is expected to be deleted quickly; this is simply the quickest path to a working RPC service for testing Cloud Run. The binary also exports a /metrics endpoint, which we can use to test HTTP fetches as a surrogate for an actual UI. Tested: `bazel run //bestie/server:bestie` and successfully: * did `grpc_cli` call of `Greet` RPC: `grpc_cli call localhost:6433 bestie.proto.Greeter.Greet 'name: "Scott"'` * fetched metrics: `curl localhost:6433/metrics` Jira: INFRA-193

Revert "Add working example of react dashboard with grpc (#312)" (#324) This reverts commit cd303d79d42248acf8308f3f25af000cbd8025f3. Fixes a `yarn install` error when running `bazel build //...`. Jira: INFRA-189

2021-11-17

flextape: Add config definition and Xilinx config (#309) This change adds: * A protobuf config definition, for a config file passed to the server on startup * A flag on the server to provide the config file at runtime * Server code to parse the config Jira: INFRA-159

2021-11-16

Add working example of react dashboard with grpc (#312)

2021-11-10

flextape: Return queue position to client (#304) The client should log to the user the status of the license acquisition progress when queued. Currently, there's not much progress the client can communicate, other than that it's queued. This change modifies the server to return the queue position so that clients can communicate this to the user, providing user visibility that in fact progress is being made (or the queue appears to be stalled). Tested: Unit tests Jira: INFRA-139

2021-11-08

INFRA-107: Implement flextape support in existing client (#294) This change creates: * a client library for Flextape communication * unit tests for said library * a codepath in the existing license manager client that uses Flextape communication, triggered via environment variable Tested: In addition to the above: ``` # Ran flextape server bazel run //flextape/server:flextape # Monitor license status watch -x -n 1 grpc_cli call localhost:6433 Flextape.LicensesStatus '' # Run a bunch of client instances wrapping a `sleep` call for i in $(seq 1 10); do (LICENSE_SERVER_IMPL=flextape bazel-bin/manager/client/client_/client localhost 6433 xilinx bar sleep 5 &); sleep 1; done # Watch licenses get allocated, invocations get queued ``` Jira: INFRA-107

2021-10-28

INFRA-105: Add configuration file, filter options (#276) This change adds filter options to the presubmit driven by a configuration file. Filter options include: * including targets by patterns such as `//foo/bar:baz` or `//foo/bar/...` * excluding targets by patterns * excluding targets by tags The above should give us enough control to: * incrementally add directories to the presubmit * exclude broken/expensive tests by either exact label or tag To achieve this, the change contains the following: * a protobuf definition of the config file. * a first config file in the repository root: `presubmit.textproto`. * logic to parse the config file and parse patterns from the config file * logic to filter targets based on patterns and tags Tested: Unit tests, run locally to confirm both `exclude_patterns` and `exclude_tags` filtering functions as expected Jira: INFRA-105

2021-10-27

INFRA-115: Rename license_manager to flextape (#287) This change renames our license manager server to flextape by: * changing the top-level directory name * changing the proto and service names A README is added to explain what this is since it is no longer clear from the file name. Jira: INFRA-115

2021-10-25

Fuse Over RPC (#222) * commiting to switch * basics working * remove added flag * added in wrong ui * change back go version * reset build * update gazelle and run Close() on file * gazelle * fix recursive test from enkit //... * changes for PR, renaming to be cleaner * add in rename to fusepb * net.JoinHostPort * separate types * change package name * rename interfaces * remove allowother in test

2021-10-20

bazel: Parse workspace_log during query (#255) This change modifies Query to return a struct of query results. These results include targets that match the query, as well as other optional information. The first useful such piece of information is the workspace log, which details when: * WORKSPACE caused a download to happen, including the SHA256 of the download * WORKSPACE caused a command execution to happen among other events. These events will be useful in reducing external dependencies down to the SHA256s of their corresponding downloads, rather than needing to resort to hashing individual files of the external dependencies. Depends-On: #252

bazel: Add workspace_log proto (#252) The workspace_log proto is from the bazel repository and is required for parsing workspace logs; these logs will be used to determine SHA hashes for third-party dependencies, allowing for the detection of changes without hashing each file individually. This file is copied over integrating directly the WORKSPACE because: * it saves fetch time (the bazel repository is large) * BUILD files fetched in the bazel repository do not grant us visibility to the proto, necessitating patches

2021-10-08

license_manager: Implement server handlers (#244) This change contains the bulk of the server handler implementation, along with associated unit tests. The service proto is tweaked: * More fields are added to `RefreshRequest` to better handle the case where the server is adopting existing allocations after a restart. The client must pass all the information (license type, owner, build_tag, etc.) during RefreshRequests in case the server has restarted; otherwise, the server won't know which license to allocate, who it is allocated to, etc. * Common fields between RefreshRequest and AllocateRequest are deduplicated to an Invocation message. * Repeated fields are pluralized * Queued license count added to LicensesStats response Jira: INFRA-90

2021-10-05

license_manager: Add proto definition file (#229) This change adds a gRPC definition for the license_manager service, intended to supplant the existing license service under `//manager` with a protocol that uses multiple unary RPCs instead of a bidirectional streaming RPC, in an effort to clarify the implementation of both the client and the server. Requirements and more info can be found in: https://docs.google.com/document/d/1TNqbBprpcNU9tTHVCFzRwaQoHlGFdjkw221C5p9UsAw/edit Tested: All targets build: ``` bazel build //license_manager/... ``` Jira: INFRA-90

2021-07-22

Machinist DNS Polling (#91) * Adds in polling for Registering DNS entries * Adds in registering and keepalives * Adds in e2e testing

2021-04-29

Work in Progress: License Manager (#55) First draft of generic license manager for EDA tools especially for Xilinx. Not necessary for Cadence tools because -licqueue argument automatically queues Cadence jobs and polls for a license.

2021-04-19

add periods

2021-04-13

revert proto numbering

initial commit

2021-04-12

initial commit

2021-04-09

change for pr

2021-04-08

update for PR

2021-04-07

add rpc for host certificates

2021-03-29

Change auth response proto (#53) Changes the shape of the general auth response to include signed certificates.

2021-03-05

Revert "checking in before friday nap" This reverts commit 6640c8cbaecf23212b0b122df1f4bb39abc5819f.

2021-02-27

checking in before friday nap

2021-02-26

initial skeleton of client and server for machinist. Basic .proto file defining the structure of the communication protocol, and an initial implementation of a ping message.

[WIP]: initial skeleton of client and server for machinist. Basic .proto file defining the structure of the communication protocol, and an initial implementation of a ping message.

2021-01-06

[WIP] initial draft of adding astore deletion (#6) * initial draft commit * changes for PR, adding skip confirm flag and fmt

changes for PR, adding skip confirm flag and fmt

2021-01-05

initial draft commit

2020-11-03

Placeholder for machinist, a daemon to manage machines. machinist will be able to: - register machines by name, provide name resolution. - dynamically allocate and free machines. - run tests on those machines.

2020-09-18

astore: refactor download command. Before this change: - download command did not support tags - did not work correctly when uids were used - had a set of defaults used for every downloaded file After this change: - update download command to use the same style as other commands - no longer defaults, options are per file - support for tags and archs now working correctly

2020-08-26

Initial commit of utilities.