Proto commits in golang/build

These 69 commits are when the Protocol Buffers files have changed:

2024-10-02

internal/gomote: add ListDirectoryStreaming RPC ListDirectory responses can overflow the maximum RPC message size. Add a streaming version which returns its results in chunks. For golang/go#69732 Change-Id: I3f7d50a3899f0cbc9954da89a26906f1f5850a93 Reviewed-on: https://go-review.googlesource.com/c/build/+/617160 LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Carlos Amedee <carlos@golang.org>

The documentation is generated from this commit.

all: regenerate .pb.go files, unify generation process Add a internal/cmd/update-protos command which regenerates all protobuf files within the repo, using the versions of the code generators listed in the go.mod file. Replace the per-directory //go:generate commands with a single top-level one. Regenerate all .pb.go files. For golang/go#69732 Change-Id: I337badfb49fc29f638d71eee609d5995262ccd75 Reviewed-on: https://go-review.googlesource.com/c/build/+/617057 Reviewed-by: Carlos Amedee <carlos@golang.org> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>

2024-04-05

internal/relui/sign: add binary artifact signing modes For golang/vscode-go#3186 Change-Id: I684e222cc018b18ad0c173522dfd70cb3b34e0ca Reviewed-on: https://go-review.googlesource.com/c/build/+/571575 Reviewed-by: Dmitri Shuralyov <dmitshur@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Auto-Submit: Hyang-Ah Hana Kim <hyangah@gmail.com> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org> Reviewed-by: Carlos Amedee <carlos@golang.org>

2024-02-20

internal/gomote/protos: add experiment option to gomote proto This adds the ability to pass an experimental option to the gomote server. Change-Id: I86dd35c6e16212f18d91bea58d36c06109c7ab11 Reviewed-on: https://go-review.googlesource.com/c/build/+/565417 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org> Reviewed-by: Dmitri Shuralyov <dmitshur@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>

2023-12-15

internal/relui/{sign,protos}: add construct-installer-only build types These are similar to the existing macOS and Windows signing build types, but will be used specifically for installer construction only. For golang/go#63147. Change-Id: If835cb66024f3119b69e35593c323dbba6e5f5b2 Reviewed-on: https://go-review.googlesource.com/c/build/+/550320 Auto-Submit: Dmitri Shuralyov <dmitshur@google.com> Reviewed-by: Carlos Amedee <carlos@golang.org> Reviewed-by: Dmitri Shuralyov <dmitshur@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>

2023-09-07

internal/gomote/protos: rearrange function order Change-Id: Ia35fe8f4e63b3891993a91122e6e6fe4daecd757 Reviewed-on: https://go-review.googlesource.com/c/build/+/526258 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org> Reviewed-by: Dmitri Shuralyov <dmitshur@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>

2023-08-14

internal/gomote/protos: add a list swarming builders endpoint This change adds a GRPC endpint to the gomote server which lists all of the LUCI swarming builders for the project. The builders names will be used to request the types of gomote instance requested. Updates golang/go#61773 For golang/go#61772 Change-Id: I2c6bfb3d251967b8008ba64ee28f2e1c8472df5b Reviewed-on: https://go-review.googlesource.com/c/build/+/518796 Reviewed-by: Dmitri Shuralyov <dmitshur@google.com> Run-TryBot: Carlos Amedee <carlos@golang.org> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> TryBot-Result: Gopher Robot <gobot@golang.org> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>

2023-04-27

all: remove repeated definite articles Change-Id: Ide10bbe72c9fdcafb064ff0af552e279eafa180f Reviewed-on: https://go-review.googlesource.com/c/build/+/489675 Auto-Submit: Heschi Kreinick <heschi@google.com> Reviewed-by: Heschi Kreinick <heschi@google.com> Reviewed-by: Carlos Amedee <carlos@golang.org> Run-TryBot: shuang cui <imcusg@gmail.com> TryBot-Result: Gopher Robot <gobot@golang.org>

2022-09-02

internal/relui/protos: update wire protocol Development and testing of the release signing workflow helped us find ways to adjust the wire protocol to be a better fit for our needs. The responsibility to come up with a job ID for future requests is moved to the initial sign request creation transaction, replacing the need for taskName and retryCount parameters. Have just one build type for macOS signing because we no longer need to keep the AMD64 and ARM64 versions separate. Add a message and endpoint for marking a previously started signing job as obsolete, so it'll be possible to do this automatically when a relui release workflow happens to be stopped by a release coordinator. Update the sign server and its tests accordingly. For golang/go#53632. Change-Id: I99081b0a0f8140b68e1a4b20bc5e422516052e28 Reviewed-on: https://go-review.googlesource.com/c/build/+/428057 Auto-Submit: Dmitri Shuralyov <dmitshur@golang.org> Reviewed-by: Dmitri Shuralyov <dmitshur@google.com> TryBot-Result: Gopher Robot <gobot@golang.org> Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org> Reviewed-by: Carlos Amedee <carlos@golang.org>

internal/relui/protos: reorder protos This change is a no-op other than moving messages and fields around to make them easier to make sense of and to navigate when reading from top to bottom. The next CL makes logical changes. For golang/go#53632. Change-Id: I03bf5e31f1ee2845b6e6014f610b3e07540b0159 Reviewed-on: https://go-review.googlesource.com/c/build/+/428056 TryBot-Result: Gopher Robot <gobot@golang.org> Reviewed-by: Carlos Amedee <carlos@golang.org> Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org> Reviewed-by: Dmitri Shuralyov <dmitshur@google.com> Auto-Submit: Dmitri Shuralyov <dmitshur@golang.org>

2022-08-18

internal/relui/sign: add signing server This change adds the signing server used to request artifact signing. An interface has been added to the sign package in order to create different implementations of the signing service if needed. With the GRPC signing service, the client creates a bidirectional connection with the server. The server then sends well specified signing requests to the client. Updates golang/go#54303 Updates golang/go#53632 Change-Id: I062f7db716edba810b6c1dab122b2a10c1c18923 Reviewed-on: https://go-review.googlesource.com/c/build/+/422598 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org> Reviewed-by: Jenny Rakoczy <jenny@golang.org> Run-TryBot: Carlos Amedee <carlos@golang.org> Reviewed-by: Dmitri Shuralyov <dmitshur@google.com> TryBot-Result: Gopher Robot <gobot@golang.org>

2022-08-09

internal/relui/protos: add protos to relui This change adds protos for release artifact signing. Updates golang/go#54303 Change-Id: I9ff6645177a4b300023f842874fc8c2acb754c72 Reviewed-on: https://go-review.googlesource.com/c/build/+/420801 Reviewed-by: Heschi Kreinick <heschi@google.com> Reviewed-by: Dmitri Shuralyov <dmitshur@google.com> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org> Reviewed-by: Jenny Rakoczy <jenny@golang.org>

buildlet: create gRPC client As a first step to port relui over to the gRPC API, create a gRPC client implementation. This should be suitable for use in the gomote command, but I don't want to conflict with the giant stack in flight. Also add a GRPCClient function to iapclient for convenience. The gomote service was missing working directory information; added it. I also did some refactoring/cleanup. - The Client interface is used internally to the coordinator, and as such it has a lot of internal functions. The set used by remote clients is much more reasonable. Separate it out into a new RemoteClient interface. - AddCloseFn was only used in one place. Delete it. - DestroyVM was completely unused. Delete it. For golang/go#54344. Change-Id: I77344a8c27076e53f565e9af56f8d365ac89b487 Reviewed-on: https://go-review.googlesource.com/c/build/+/422095 Auto-Submit: Heschi Kreinick <heschi@google.com> Reviewed-by: Carlos Amedee <carlos@golang.org> TryBot-Result: Gopher Robot <gobot@golang.org> Run-TryBot: Heschi Kreinick <heschi@google.com>

2022-07-28

cmd/coordinator,internal,maintner: prioritize relui builds Populates a new User field on SchedItem which will keep track of either the gomote user or the commit author email for trybots. Sets relui related builds to the highest priority, PriorityUrgent. Fixes golang/go#48857 Change-Id: I8930a0e8c6bdb1becd454e953779f42f3885fbef Reviewed-on: https://go-review.googlesource.com/c/build/+/419428 Run-TryBot: Jenny Rakoczy <jenny@golang.org> TryBot-Result: Gopher Robot <gobot@golang.org> Reviewed-by: Heschi Kreinick <heschi@google.com>

2022-07-22

internal/gomote, cmd/gomote: execute command returns bytes The ExecuteCommand endpoint should return bytes instead of strings. The caller should be able to choose how to manipulate the output. Fixes golang/go#54004 Change-Id: I4c0f9f2aaf816494cf2c07fc567f2686d50a52a2 Reviewed-on: https://go-review.googlesource.com/c/build/+/419078 Run-TryBot: Carlos Amedee <carlos@golang.org> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org> TryBot-Result: Gopher Robot <gobot@golang.org> Reviewed-by: Than McIntosh <thanm@google.com> Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>

2022-06-07

internal/gomote,cmd/gomote: implements GRPC add bootstrap This change adds the implementation for GRPC putbootstrap command to the gomote client. It also adds the gomote server implementation of the AddBootstrap endpoint. This endpoint adds the bootstrap Go version to an existing client. Updates golang/go#48737 Updates golang/go#48742 For golang/go#47521 Change-Id: Ib0807a13e85a0e350485c8300ac2e180456bd0fc Reviewed-on: https://go-review.googlesource.com/c/build/+/410818 TryBot-Result: Gopher Robot <gobot@golang.org> Run-TryBot: Carlos Amedee <carlos@golang.org> Reviewed-by: Carlos Amedee <carlos@golang.org> Auto-Submit: Carlos Amedee <carlos@golang.org> Reviewed-by: Alex Rakoczy <alex@golang.org>

2022-05-12

internal/gomote: moves logic to server from client These changes are being made in an attempt to move logic from the client to the server. - Changes the function which extracts the object name from a URL. - Moves setting the environment variables to the server in ExecuteCommand endpoint. For golang/go#47521 Updates golang/go#48742 Change-Id: I4fa370a1b3c949bd5913491d1650d131577ff30f Reviewed-on: https://go-review.googlesource.com/c/build/+/406014 Reviewed-by: Heschi Kreinick <heschi@google.com> Run-TryBot: Carlos Amedee <carlos@golang.org> Auto-Submit: Carlos Amedee <carlos@golang.org> Reviewed-by: Carlos Amedee <carlos@golang.org> TryBot-Result: Gopher Robot <gobot@golang.org>

2022-05-11

internal/gomote: add object name to UploadFile endpoint This change adds the object name as one of the fields returned to the caller. The object name is not included in the URL. The object name is required for any subsequent calls needed to pass the file along to the gomote instances. For golang/go#47521 Updates golang/go#48742 Change-Id: I5437ab2029907460806f2f3b6a19f6dec22115da Reviewed-on: https://go-review.googlesource.com/c/build/+/405535 TryBot-Result: Gopher Robot <gobot@golang.org> Auto-Submit: Carlos Amedee <carlos@golang.org> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org> Reviewed-by: Dmitri Shuralyov <dmitshur@google.com> Reviewed-by: Carlos Amedee <carlos@golang.org> Run-TryBot: Carlos Amedee <carlos@golang.org>

2022-04-12

internal/gomote, internal/gomote/protos: add ReadTGZToURL endpoint This change adds the ReadTGZToURL endpoint implementation. This tars and zips the directory requested on the gomote instance and uploads it to GCS. A signed URL is returned which the caller can use to download the file. For golang/go#47521 Updates golang/go#48742 Change-Id: I5e9574994810b804acb4b9ed9e6bdda68ea26713 Reviewed-on: https://go-review.googlesource.com/c/build/+/397598 Run-TryBot: Carlos Amedee <carlos@golang.org> Auto-Submit: Carlos Amedee <carlos@golang.org> TryBot-Result: Gopher Robot <gobot@golang.org> Reviewed-by: Carlos Amedee <carlos@golang.org> Reviewed-by: Heschi Kreinick <heschi@google.com>

2022-04-08

internal/gomote, internal/gomote/protos: implement write files from URL This change adds the implementation for the WriteFileFromURL endpoint. The caller will be able to add the contents of an HTTP get call to a file on the gomote instance. They must set the permissions on the file. Files located on the gomote transfer GCS bucket will be retrieved using the GCS storage package and authentication vs a vanilla HTTP call. For golang/go#47521 Updates golang/go#48742 Change-Id: If9ac24654352433c7a073de08017213223cf9020 Reviewed-on: https://go-review.googlesource.com/c/build/+/397596 Reviewed-by: Heschi Kreinick <heschi@google.com> Trust: Carlos Amedee <carlos@golang.org> Run-TryBot: Carlos Amedee <carlos@golang.org> TryBot-Result: Gopher Robot <gobot@golang.org>

internal/gomote, internal/gomote/protos: add the upload file endpoint This change adds the upload file endpoint which will be used by the gomote clients to upload files to GCS before they are retrieved by a gomote instance. The endpoint generates a signed URL and associated fields which must be used in the upload. For golang/go#47521 Updates golang/go#48742 Change-Id: Id85a55b41b8211b3aae8c2e30245a0b71ecfa238 Reviewed-on: https://go-review.googlesource.com/c/build/+/397595 Trust: Carlos Amedee <carlos@golang.org> Reviewed-by: Heschi Kreinick <heschi@google.com>

2022-03-21

internal/gomote, internal/coordinator/remote: add the sign SSH key endpoint implementation This change adds the implementation for the sign SSH key endpoint. The endpoint accepts a public SSH key and signs it with the gomote server's certificate authority. The certificate added to the public key will be used to validate if the certificate authority was used to sign the certificate. It will also be used to determine if the requestor has rights to initiate and SSH session with the gomote instance being requested. This is part of a shift to OpenSSH certificate authentication in the gomote SSH server. For golang/go#47521 Updates golang/go#48742 Change-Id: I427b34c7f006ae20f5643322dc0754bf7a82e5f1 Reviewed-on: https://go-review.googlesource.com/c/build/+/391516 Trust: Carlos Amedee <carlos@golang.org> Run-TryBot: Carlos Amedee <carlos@golang.org> TryBot-Result: Gopher Robot <gobot@golang.org> Reviewed-by: Heschi Kreinick <heschi@google.com>

internal/gomote, buildlet: add execute command implementation This change adds the implementation for the execute command endpoint. This enables the caller to execute a command on the gomote instance. The output from the command will stream back to the caller if requested. This change also adds helper functions that retrieve the session from the session pool and the buildlet client. For golang/go#47521 Updates golang/go#48742 Change-Id: Iec1853667992b5674b07be5e972ac145a6349fca Reviewed-on: https://go-review.googlesource.com/c/build/+/382494 Trust: Carlos Amedee <carlos@golang.org> Run-TryBot: Carlos Amedee <carlos@golang.org> TryBot-Result: Gopher Robot <gobot@golang.org> Reviewed-by: Alex Rakoczy <alex@golang.org>

2022-03-17

all: converge GRPC usage to a single version The build repository currently uses two diffrent GRPC implementations. The go4.org one and the official one. This change attempts to converge on a single version of GRPC for the build repository. Change-Id: I3c19c1159646cbbff8fa28c8f1738cb468b6be8c Reviewed-on: https://go-review.googlesource.com/c/build/+/361454 Reviewed-by: Alex Rakoczy <alex@golang.org> Reviewed-by: Heschi Kreinick <heschi@google.com> Trust: Carlos Amedee <carlos@golang.org> Run-TryBot: Carlos Amedee <carlos@golang.org> TryBot-Result: Gopher Robot <gobot@golang.org>

2022-03-16

internal/gomote/protos: fix spelling errors This change fixes some spelling and grammatical errors. Updates golang/go#48742 Change-Id: I9e6da866bc077ca59e0ff1dba42a4eea637b3c13 Reviewed-on: https://go-review.googlesource.com/c/build/+/393094 Trust: Carlos Amedee <carlos@golang.org> Run-TryBot: Carlos Amedee <carlos@golang.org> TryBot-Result: Gopher Robot <gobot@golang.org> Reviewed-by: Alex Rakoczy <alex@golang.org>

2022-01-31

internal/gomote, buildlet: add the list directory endpoint implementation This change adds the implementation for the list directory endpoint. The list directory endpoint lists the contents of a directory on an gomote instance. The ListDir function on the fake buildlet client has also been modified to behave more like the actual buildlet client implementation. Updates golang/go#48742 Change-Id: If5a1fe9aa62c0dfee56392390a0f4d6d6df7af3d Reviewed-on: https://go-review.googlesource.com/c/build/+/381738 Trust: Carlos Amedee <carlos@golang.org> Run-TryBot: Carlos Amedee <carlos@golang.org> Reviewed-by: Alex Rakoczy <alex@golang.org> TryBot-Result: Gopher Robot <gobot@golang.org>

2022-01-14

internal/gomote, internal/gomote/protos: add remove files This changes the remove directory endpoint to remove files. This change was made because the endpoint should describe the removal of files or directories from the gomote instances. It also adds the implementation of remove files to the gomote server. Updates golang/go#48742 Change-Id: Icaf11afd760bf9837b8c536703774c9d76e588f6 Reviewed-on: https://go-review.googlesource.com/c/build/+/374155 Trust: Carlos Amedee <carlos@golang.org> Run-TryBot: Carlos Amedee <carlos@golang.org> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org> Reviewed-by: Alex Rakoczy <alex@golang.org> TryBot-Result: Gopher Robot <gobot@golang.org>

2022-01-11

internal/gomote, internal/gomote/protos: add write tgz from URL This change adds the WriteTGZFromURL implementation to the gomote server. This RPC allows the caller to instruct the gomote instance to download the tar.gz file from the provided URL and extract it to the work directory. Updates golang/go#48742 Change-Id: I9aefb5dd5fed52dbfc1fba102c38234219e5d56b Reviewed-on: https://go-review.googlesource.com/c/build/+/377694 Trust: Carlos Amedee <carlos@golang.org> Run-TryBot: Carlos Amedee <carlos@golang.org> TryBot-Result: Gopher Robot <gobot@golang.org> Reviewed-by: Alex Rakoczy <alex@golang.org>

2021-12-22

internal/coordinator/remote, internal/gomote: add instance alive This change adds the implementation for gomote instance alive. This endpoint enables the caller to check if an instance is alive. If the instance is alive, it will extend the gomote timeout time. A renew timeout method has been added to the session pool. This differs from the existing keep alive method in that it is an single call to renew the timeout for an instance instead of a continuous renewal of the timeout tied to the lifetime of a context. Updated golang/go#48742 Change-Id: I3b3462407d9f4a02c4e2cea0f14950c8c9f21060 Reviewed-on: https://go-review.googlesource.com/c/build/+/374115 Trust: Carlos Amedee <carlos@golang.org> Run-TryBot: Carlos Amedee <carlos@golang.org> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org> TryBot-Result: Gopher Robot <gobot@golang.org>

internal/gomote, internal/gomote/protos: add destroy instance This change adds the implementation for the gomote destroy instances endpoint. Updates golang/go#48742 Change-Id: I8168c6f10583cb6609bce9daafd69e599cd1a349 Reviewed-on: https://go-review.googlesource.com/c/build/+/371814 Trust: Carlos Amedee <carlos@golang.org> Run-TryBot: Carlos Amedee <carlos@golang.org> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org> TryBot-Result: Gopher Robot <gobot@golang.org> Reviewed-by: Alex Rakoczy <alex@golang.org>

internal/gomote, internal/gomote/protos: add list instances This change implements the list instances endpoint for the gomote service. Updates golang/go#48742 Change-Id: Ib59b9f70a399b4795aa33b9234227b50d08b12f5 Reviewed-on: https://go-review.googlesource.com/c/build/+/371817 Trust: Carlos Amedee <carlos@golang.org> Run-TryBot: Carlos Amedee <carlos@golang.org> Reviewed-by: Alex Rakoczy <alex@golang.org> TryBot-Result: Gopher Robot <gobot@golang.org>

internal/gomote, internal/gomote/protos: add create instance implementation This change implements the endpoint to create gomote instances for the gomote GRPC service. In the process of implementing creates, various other changes were needed: - Refactoring the remote session pool. - Extending the fake schedule used for testing. Updates golang/go#48742 Change-Id: I0c74e38539428d028917200ccd6bd0c58fa14801 Reviewed-on: https://go-review.googlesource.com/c/build/+/370662 Trust: Carlos Amedee <carlos@golang.org> Run-TryBot: Carlos Amedee <carlos@golang.org> Trust: Dmitri Shuralyov <dmitshur@golang.org> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org> TryBot-Result: Gopher Robot <gobot@golang.org>

2021-10-21

internal/gomote/protos: add a skeleton for a gomote api This change adds a skeleton for a new GRPC gomote API. This work is part of a reworking of the security model around gomotes. Updates golang/go#47521 Updates golang/go#48742 Change-Id: I4b0ae84bf58fe6e999fb34c17e670a6f638055f0 Reviewed-on: https://go-review.googlesource.com/c/build/+/356589 Trust: Carlos Amedee <carlos@golang.org> Trust: Alexander Rakoczy <alex@golang.org> Run-TryBot: Carlos Amedee <carlos@golang.org> TryBot-Result: Go Bot <gobot@golang.org> Reviewed-by: Alexander Rakoczy <alex@golang.org> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>

2021-09-02

cmd/relui: remove original implementation This is in preparation for a move to using internal/workflow. For golang/go#47401 Change-Id: Ie58224fff4fb73d006f2ce56f17329193140a5e6 Reviewed-on: https://go-review.googlesource.com/c/build/+/347291 Trust: Alexander Rakoczy <alex@golang.org> Run-TryBot: Alexander Rakoczy <alex@golang.org> TryBot-Result: Go Bot <gobot@golang.org> Reviewed-by: Heschi Kreinick <heschi@google.com>

2021-02-11

cmd/relui: add datastore support This change replaces the file storage layer with Google Cloud Datastore. It adds a fake implementation of the datastore client, supporting only the features we use so far. Slightly simplifies Workflow configuration. Updates golang/go#40279 Change-Id: I55228f6540fbcdf5f803203ff7309232cebf6a20 Reviewed-on: https://go-review.googlesource.com/c/build/+/275237 Run-TryBot: Alexander Rakoczy <alex@golang.org> TryBot-Result: Go Bot <gobot@golang.org> Trust: Alexander Rakoczy <alex@golang.org> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org> Reviewed-by: Carlos Amedee <carlos@golang.org>

2020-09-25

cmd/relui: publish task start message to pubsub This change adds a new handler for starting a task by sending a message to pubsub. A test pubsub server is used in unit testing. The exact message used to start a task, as well as updating a task to mark it as started, is still under design and likely to change dramatically. This change is intended to unblock worker development. For golang/go#40279 Co-authored-by: Carlos Amedee <carlos@golang.org> Change-Id: I59bb5c5261a9a5d17e52597c1835a2a980cf91f8 Reviewed-on: https://go-review.googlesource.com/c/build/+/257239 Trust: Alexander Rakoczy <alex@golang.org> Run-TryBot: Alexander Rakoczy <alex@golang.org> TryBot-Result: Go Bot <gobot@golang.org> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>

cmd/relui: generate uuid for workflow and task instances Sets workflow and tasks IDs when a workflow is created in the UI. These IDs will be used in the future when operating on workflows and tasks via the UI or API. This ID will likely change to a datastore ID after datastore integration is added. For golang/go#40279 Co-authored-by: Carlos Amedee <carlos@golang.org> Change-Id: Ib75c00c234d156cc1bbdab8c658281f04914165b Reviewed-on: https://go-review.googlesource.com/c/build/+/257238 Trust: Alexander Rakoczy <alex@golang.org> Run-TryBot: Alexander Rakoczy <alex@golang.org> TryBot-Result: Go Bot <gobot@golang.org> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>

2020-08-05

cmd/relui: add persistence to development database This change renames memoryStore to fileStore, and persists data created in the UI. This will help make local development less painful when testing changes. Data stored is intended to be loaded at start-up, which will be implemented in a future change. For golang/go#40279 Co-authored-by: Carlos Amedee <carlos@golang.org> Change-Id: Id2390c35b8e1d1d368fbf7ac13b3cdef0776ad87 Reviewed-on: https://go-review.googlesource.com/c/build/+/246298 Run-TryBot: Alexander Rakoczy <alex@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org> Reviewed-by: Andrew Bonventre <andybons@golang.org>

2020-07-29

cmd/relui: add proto definition for workflows Using configuration for our workflows will help separate concerns between implementation and workflow configuration. Eventually, similar tasks can be re-used in different workflows, such as fetching from Git, updating the VERSION file, or publishing a tag. The current configuration definition is mainly illustrative, and is expected to change as we build out a prototype. For golang/go#40279 Change-Id: I5c6f8a18571ab819de0b1d026c86050735efeed9 Reviewed-on: https://go-review.googlesource.com/c/build/+/243340 Run-TryBot: Alexander Rakoczy <alex@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Andrew Bonventre <andybons@golang.org>

2020-07-08

maintner/maintnerd/maintapi: look for TRY= in inline comments The Gerrit UI has changed its default behavior when leaving top-level comments on a CL. Previously, that text was a part of the CL-scoped messages, but now it is located in a patchset-level inline comment. This change updates the logic in maintapi to look for TRY= comments in the new place. This requires making an additional Gerrit API call to the new ListChangeComments endpoint, because the inline comment data is not tracked by the maintner corpus, and it can't be fetched via the existing QueryChanges API call. Start using the patch set number as returned by the Gerrit API, instead of parsing it out of the message prefix with a regexp, as this is simpler and should be more robust. Document and simplify tryWorkItem slightly by removing support for nil *gerrit.ChangeInfo. This was only needed for tests, but it's easy to adjust tests to not depend on this. That makes the code in the helper less indented and less complicated. During these changes, the unintentional pitfall where the TRY= comment had to be preceded by a blank link has also been removed. It only needs to be on its own line now, as it was likely originally intended. Fixes golang/go#40106. Fixes golang/go#38747. Change-Id: I64e85c18fefc23e1796de61dd8ba7025ea411706 Reviewed-on: https://go-review.googlesource.com/c/build/+/241323 Reviewed-by: Filippo Valsorda <filippo@golang.org>

2020-02-20

cmd/coordinator,cmd/retrybuilds: add wipe API to coordinator As part of our migration to combine codebases of the Build Dashboard and the Coordinator, the first step is to start calling a Coordinator API for wiping release status of failed builds. This adds a gRPC API to the coordinator, listening on the same port as the HTTPS listeners. The Coordinator API in this implementation simply validates and forwards a request to the dashboard API. This change also updates cmd/retrybuilds to optionally use the Coordinator gRPC API for wiping. Tested locally using the live Dashboard API. Updates golang/go#34744 Change-Id: I4b34b064625193eb11a280565d701605064a8443 Reviewed-on: https://go-review.googlesource.com/c/build/+/219120 Run-TryBot: Alexander Rakoczy <alex@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Carlos Amedee <carlos@golang.org>

2019-12-06

app, gitmirror, maintner: use maintner for dashboard, not datastore Historically, the build.golang.org was the entire build system, and it maintained a parallel copy of that git history in its datastore. It was always buggy and incomplete and things like force pushes were scary because the datastore mirror could get out of sync. It was also a lot of code to support that sync. This changes build.golang.org to instead get the git history from maintnerd, and then we can remove all the HTTP handlers around updating it, and can remove all the gitmirror code to call it to maintain it. Now build.golang.org only keeps build results, keyed on the commit hash. It's much less code, but is still an App Engine app for now. (but it's getting small enough, that porting it to cloud.google.com/go/datastore is looking very simple) This also adds a new "repos" package to unify the configuration of the various Go repos. There were incomplete & redundant copies all over the place. Updates golang/go#34744 Fixes golang/go#35828 Fixes golang/go#31236 (New branch=mixed support adds this when desired) Fixes golang/go#35944 Change-Id: Ifb39417287df3dea052ba8510566d80b4bc75d51 Reviewed-on: https://go-review.googlesource.com/c/build/+/208697 Reviewed-by: Bryan C. Mills <bcmills@google.com>

2019-12-05

maintner: don't generate no-op github issue mutations on empty bodies Fixes golang/go#35985 Change-Id: I45a10ce99bf047c600d72db2dc1428235103322f Reviewed-on: https://go-review.googlesource.com/c/build/+/209967 Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Alexander Rakoczy <alex@golang.org>

2019-12-03

maintner: record, note deleted Gerrit refs When run in --config=devgo mode (which forks the main mutation log and runs it locally, adding new mutations as needed), I now see: 2019/11/27 13:48:41 gerrit go.googlesource.com/go: gerrit ref "refs/heads/dev.link" = "27c0aeee3fc7ed35f9b2eac4725b9147902537da" 2019/11/27 13:48:41 gerrit go.googlesource.com/go: gerrit ref "refs/heads/release-branch.go1.12-security" now deleted 2019/11/27 13:48:41 gerrit go.googlesource.com/go: gerrit ref "refs/heads/release-branch.go1.11-security" now deleted 2019/11/27 13:48:41 gerrit go.googlesource.com/go: gerrit ref "refs/tags/andybons/goroot.mailed" now deleted 2019/11/27 13:48:41 gerrit go.googlesource.com/go: gerrit ref "refs/heads/release-branch.go1.10-security" now deleted 2019/11/27 13:48:41 gerrit go.googlesource.com/go: gerrit ref "refs/tags/1.10beta1.mailed" now deleted 2019/11/27 13:48:41 gerrit go.googlesource.com/go: gerrit ref "refs/tags/andybons/blog.mailed" now deleted 2019/11/27 13:48:41 gerrit go.googlesource.com/go: gerrit ref "refs/tags/andybons/cachedir.mailed" now deleted 2019/11/27 13:48:41 gerrit go.googlesource.com/go: gerrit ref "refs/heads/release-branch.go1.13-security" now deleted 2019/11/27 13:48:41 gerrit go.googlesource.com/go: gerrit ref "refs/heads/tmp.shadams" now deleted Change-Id: I0da673a3de7064f50406de9b369e2f1b9f246730 Reviewed-on: https://go-review.googlesource.com/c/build/+/209117 Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>

2019-10-16

maintner/maintnerd: return TRY= comment directives in Run-TryBot votes For slowbots support. Updates golang/go#34501 Change-Id: I6f8a8b187c3ce4531498f106d62e29e1d1f421d8 Reviewed-on: https://go-review.googlesource.com/c/build/+/201204 Reviewed-by: Ian Lance Taylor <iant@golang.org>

2019-07-03

maintner/maintnerd: improve API documentation Rearrange the maintner API documentation, and reflect the guarantee of at least two Go releases on success. Updates golang/go#32606 Change-Id: I90d5017f280254cc6482e747ffc1534f8a51bc20 Reviewed-on: https://go-review.googlesource.com/c/build/+/182981 Run-TryBot: Alexander Rakoczy <alex@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>

2019-01-25

dashboard, cmd/coordinator, maintner/maintnerd: add support for BuildConfig.MinimumGoVersion field The new BuildConfig.MinimumGoVersion field specifies the minimum Go version the builder is allowed to use. It's useful when some of the builders are too new, and do not support all of the supported Go releases (e.g., openbsd-amd64-64 and freebsd-amd64-12_0 currently require Go 1.11 and don't work on Go 1.10). It only needs to be set when a builder doesn't support all supported Go releases, since we don't typically test unsupported Go releases. To allow cmd/coordinator to use this field and filter out work it receives from maintner/maintnerd's GoFindTryWork RPC call, we add a GoVersion slice to apipb.GerritTryWorkItem, and populate it in maintapi.apiService.GoFindTryWork method. For trybots on the Go repo, the GoVersion field is determined from the branch name. For "release-branch.goX.Y" branches, it parses out the major-minor Go version from the branch name. For master and other branches, it assumes the latest Go release. For trybots on subrepos, we already have the Go version information available, so use it directly. Afterwards, all that's left is to modify newTrySet in cmd/coordinator to make use of BuildConfig.MinimumGoVersion and work.GoVersion to skip builders that are too new for the Go version that needs to be tested. Fixes golang/go#29265 Change-Id: I50b01830647e33e37e9eb8b89e0f2518812fa44f Reviewed-on: https://go-review.googlesource.com/c/155463 Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>

2018-12-06

maintner: extend GitHubIssue to track Code Review Events Previously, maintner would track only the Events on an Issue/PullRequest, which missed out on the discussions and approvals that the review process generates. This adds a new struct GitHubReviewEvent to represent the results from the GitHub "pulls/reviews" API. This struct follows the same pattern as the GitHubIssueEvent struct. To accomplish this, a new call has been added to the sync() function: syncReviews, which follows the pattern established with syncEvents(). Finally, this adds an additonal method: ForeachReview which iterates over the GitHubReviewEvents in the given GitHubIssue serially and in chronological order. GitHub API Reference: https://developer.github.com/v3/pulls/reviews/ Updates golang/go#21086 Change-Id: I4f59e154a4e4a8a4b8f2676dea932cf44354c288 Reviewed-on: https://go-review.googlesource.com/c/144699 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>

2018-11-02

maintner/maintnerd: change definition of Go release to require release branch Previously, the list-Go-releases RPC endpoint was more flexible and considered a Go release to exist as long as there's a git tag for it, regardless of whether a corresponding release branch existed. Such a situation is very unlikely to come up, and we don't want to cause all users of this API to filter out releases without a release branch. So just re-define a Go release to require a release branch, and make that the API promise. In practice, this should not have any effect because all go tags have corresponding release branches. Updates golang/go#17626 Change-Id: Ia7a8354000483c969e123f0f3605fd360846c40b Reviewed-on: https://go-review.googlesource.com/c/147200 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>

2018-10-31

maintner: add RPC endpoint for listing Go releases This change adds an RPC endpoint to maintnerd API server to list the supported Go releases. This is needed and will be used by cmd/coordinator to run subrepo trybots on previous Go release (in addition to current). It's implemented on top of the Gerrit project ref data that the maintner corpus already tracks. A release is considered to be exist for each git tag named "goX", "goX.Y", or "goX.Y.Z". This functionality is also implemented in some other places using data that is further away from the source of truth. Such places can start to use this endpoint instead. Add a subcommand to maintq to invoke the new list Go releases endpoint. Its current output (against a local development maintnerd server): $ go run .../maintner/maintq -server=localhost:6344 list-releases major:1 minor:11 patch:1 tag_name:"go1.11.1" tag_commit:"26957168c4c0cdcc7ca4f0b19d0eb19474d224ac" branch_name:"release-branch.go1.11" branch_commit:"97781d2ed116d2cd9cb870d0b84fc0ec598c9abc" major:1 minor:10 patch:4 tag_name:"go1.10.4" tag_commit:"2191fce26a7fd1cd5b4975e7bd44ab44b1d9dd78" branch_name:"release-branch.go1.10" branch_commit:"e97b7d68f107ff60152f5bd5701e0286f221ee93" Updates golang/go#17626 Change-Id: Ia9ea7f49d421ce0c7a9e85c423aba31572cea52b Reviewed-on: https://go-review.googlesource.com/c/146137 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>

2018-05-01

maintner: add support for team assignment GitHub reviews can be assigned to teams. Change-Id: I675944c40619635e554d083cfa4fd6e332315f9b Reviewed-on: https://go-review.googlesource.com/101955 Reviewed-by: Chris Broadfoot <cbro@golang.org>

2018-02-23

maintner: update DismissedReviewEvent "state" field from int to string GitHub has docs on valid values. On many repos in the GoogleCloudPlatform org, I observed that this field is indeed a string. Change-Id: I19fd4871f741c6e44968ec7a2dda9f1ef93ae128 Reviewed-on: https://go-review.googlesource.com/96835 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>

2017-12-20

maintner: add 3 new fields to GithubIssueEvent GithubIssueEvent added three new fields: Requested Reviewer, Review Requester, and Dismissed Review. Parse these fields from the API JSON and also handle them in the Protobuf file. Fixes golang/go#23151. Change-Id: I07f2004f300223045636c3e32027468b29dd6329 Reviewed-on: https://go-review.googlesource.com/84375 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>

2017-10-10

maintner: add PullRequest field to GitHubIssue All GitHub Pull Requests are Issues, but not all Issues are PRs. Add a boolean field to track whether a GitHub issue is a Pull Request. Update golang/go#18517 Change-Id: I5e61254f956ca90459ef0aafc4211fd3ee33f337 Reviewed-on: https://go-review.googlesource.com/69590 Reviewed-by: Sarah Adams <shadams@google.com>

2017-07-28

maintner/maintnerd, cmd/coordinator: use maintnerd to find TryBot work Instead of polling Gerrit every 60 seconds, poll maintnerd every second. This should improve TryBot-requested to TryBot-running latency by 30 seconds on average. (Ideally it'd long poll for changes and not even have ~500ms delay, but this is an improvement.) Also, in the process this does most of the work for golang/go#17626. And this cleans up the repo head tracking complication in the coordinator and just asks maintner for it instead. Running benchmarks in the coordinator has been disabled since 2017-06-23 but this CL disables it further, removing some code it'd need to work. But considering that benchmarking would need code repairs to get working again anyway, I consider this acceptable in the short term. I left TODO notes. Updates golang/go#17626 Updates golang/go#19871 Change-Id: Idab43f65a9cca861bffb37748b036a5c9baee864 Reviewed-on: https://go-review.googlesource.com/51590 Reviewed-by: Andrew Bonventre <andybons@golang.org>

2017-05-25

maintner: parse Gerrit reply bodies The contents of a Gerrit reply are stored in the body of the commit message for a meta commit, but we currently ignore them. Parse the message text when we walk the tree of meta commits, and return it when we return a CL. The API returns these in very raw fashion - what you see in a reply in the UI is what you get in the git commit. We don't do any parsing either. We could add helpers on the GerritMessage object to try to return you better formatted information. To see what the API returns for these messages, visit this URL: https://go-review.googlesource.com/changes/42180?o=MESSAGES. The messages attached to a specific line and file are stored a different way and returned via a different API. It's trickier to get those so I'll add them in a separate commit. Change-Id: Ibd56fb828c225b57650157d08b27034ece75848a Reviewed-on: https://go-review.googlesource.com/42452 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>

2017-05-18

maintnerd: start of gRPC service for maintner Also, update docs on how to re-generate. Switches to using go4.org/grpc and go4.org/grpc-codegen/protoc-gen-go4grpc for now. Updates golang/go#19866 Updates golang/go#20222 Change-Id: Ifa8a123fca2a30f17270c3c558b7395a02064eae Reviewed-on: https://go-review.googlesource.com/43560 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>

2017-03-24

maintner: initial gerrit support Pass --watch-gerrit on the command line to watch a Gerrit project. We initialize a Git repository in $HOME/var/maintnerd/<gerrit-url>, then fetch all metadata related to that project, and store it in the Corpus. Change-Id: I47a8b5e5af2ae1b99c97cc756790c8e21465e8ee Reviewed-on: https://go-review.googlesource.com/38421 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>

2017-03-20

maintner: wire up event sync Also, handle "renamed" events. Change-Id: I08f20281a87713c792ad496c4d04e6adeb7323c0 Reviewed-on: https://go-review.googlesource.com/38371 Reviewed-by: Kevin Burke <kev@inburke.com> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>

maintner: add GithubIssueEvent proto, githubEvent.Proto method, tests More work on syncing issue events. This adds a new GithubIssueEvent proto type, adds ways to go both from and to the in-memory githubEvent type, and cleans up the event tests added in 8927fdda9. The next commit will wire it up to the syncing. Change-Id: Ibdd9759dab432c6eaa4c37becfbb31f1ebf8f5b9 Reviewed-on: https://go-review.googlesource.com/38370 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>

2017-03-18

maintner: finishing implementing label and milestone sync Change-Id: I9c8f68ea0ca52dfd7e04becd5263d5efbf17b704 Reviewed-on: https://go-review.googlesource.com/38334 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>

2017-03-17

maintner: sync all github issue attributes, start of Events, Labels, Milestones More remains, but this is as good of a checkpoint as any. Change-Id: I63f6b13508562143159e92d78fc4ef60070342ae Reviewed-on: https://go-review.googlesource.com/38163 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>

2017-03-14

maintner: sync Github issue comments Change-Id: I2f4e5ac38e940461a9a3d694ce9268a5c73a4736 Reviewed-on: https://go-review.googlesource.com/38137 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>

2017-03-11

maintner: make github issue importer reliably resume anywhere and finish Including tombstomb records. Change-Id: I026b1a26d6867bee59705b5200dde8f42a7ba6f6 Reviewed-on: https://go-review.googlesource.com/38004 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>

2017-03-08

maintner: start of git polling (just proto and maintner plumbing, no git) This is just the start of git commit metadata importing, but without any of the git-specific bits. This is just the proto changes and the changes to wire up the guts with maintner. The guts will come in a following CL. Change-Id: I321590dc30ba5c62346911526fed647fc0d0b3ad Reviewed-on: https://go-review.googlesource.com/37938 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>

2017-03-07

maintner: add github ID field Change-Id: Ia497734b786c4e013a5dc3b0982ad068de23138c Reviewed-on: https://go-review.googlesource.com/37888 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>

2017-03-06

maintner: add more fields This should comprise most of the fields that are needed by the existing godash/devapp app. Will wire up the Milestone code in another commit. Change-Id: I83c399afebce5865dd03c00e295be5a19e1e36b2 Reviewed-on: https://go-review.googlesource.com/37753 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>

2017-02-12

maintner/maintpb: use google.protobuf.Timestamp Add $GOPATH/src to the --proto_path so we can import the google Timestamp definition instead of redefining a copy in our tree. Change-Id: I09c3554fb025a32c9db4892b702ee2bedb3c0e53 Reviewed-on: https://go-review.googlesource.com/36899 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>

maintner: start of proto file Change-Id: I94f1bdc212ebb5565690df61deebc57dfcefa58e Reviewed-on: https://go-review.googlesource.com/36895 Reviewed-by: Kevin Burke <kev@inburke.com> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>