Proto commits in ServiceWeaver/weaver

These 60 commits are when the Protocol Buffers files have changed:

2024-06-25

Manage grpc services with Service Weaver

2024-05-30

status: map weavelet id to pids for `multi`, `single` and `ssh` deployment. (#767) * status, multi: add weavelet id with pids As of now weaver logs shows the weavelet id and `weaver multi status` shows the respective `pid` but its hard to map weavelet id from the logs to the running process, following commit adds weavelet id with the pids of the components so its easier to debug. Signed-off-by: flouthoc <flouthoc.git@gmail.com> * status,dashboard: render weavelet id with pids Extends work in PR https://github.com/ServiceWeaver/weaver/pull/767 to refect weavelet id on status dashboard as well. Thanks to @rgrandl for diff provided here: https://github.com/ServiceWeaver/weaver/pull/767#pullrequestreview-2088787919 Signed-off-by: flouthoc <flouthoc.git@gmail.com> * ssh/manager: remove mutex lock from Status Context: https://github.com/ServiceWeaver/weaver/pull/767#discussion_r1621146209 Signed-off-by: flouthoc <flouthoc.git@gmail.com> * dev/build: run tidy Run `./dev/build_and_test tidy generate build vet test testrace` on source. Signed-off-by: flouthoc <flouthoc.git@gmail.com> --------- Signed-off-by: flouthoc <flouthoc.git@gmail.com>

The documentation is generated from this commit.

2024-05-21

DO NOT SUBMIT grpc demo impl

2024-05-09

Check the ready status of component before it becomes reachable. (#761) This fixes an situation where the component takes a while to initialize: previously, we were declaring the component as activated/healthy even during the pending init, resulting in subsequent method calls failing. Thanks to rgrandl@ for debugging the issue. Other changes: * Don't add a component to the routing info until it has been initialized. This fixes the health-checking situation where one component replica becomes healthy, but we subsequently send method calls to other replicas which fail. * Fix go vet in a couple of places.

2024-01-12

Move bulky data from EnvelopeInfo to InitWeaveletRequest. (#714) EnvelopeInfo is sent in an environment variable, so it should not be too big. Some fields in it (Sections in particular) have unbounded length, so they are moved to InitWeaveletRequest. NewRemoteWeavelet now waits for InitWeavelet() before returning. This ensures that all information the weavelet needs is available after NewRemoteWeavelet returns. Renamed EnvelopeInfo to WeaveletArgs.

Rename WeaveletInfo to InitWeaveletReply. (#713) Fix a TODO by renaming WeaveletInfo to InitWeaveletReply. This makes the type name consistent with the types of other method return value types. Reintroduce a version verification check that was mistakenly dropped in an earlier change. Envelope API provides just the weavelet properties that make sense for a deployer author to worry about (like DialAddr) instead of the entire InitWeaveletReply proto, which may contain internal details.

Remove envelope<->weavelet pipes. (#712) Initialization info sent by the envelope to the weavelet is now sent in an environment variable. Initialization info sent by the weavelet to the envelope is now sent as the response to a control.WeaveletControl.InitWeavelet() call. This call is made by NewEnvelope() as an initial hand-shake right after creating the weavelet. Deleted all the envelope/weavlet connection management code. Deleted low-level tests that were testing details of the connection management protocol. Dropped unnecessary code that carried a Bootstrap object in a context. We had switched away from our use of a Bootstrap context value a long time ago.

2024-01-10

Remove HandleLogEntry, the old log delivery method. (#710) An earlier change transitioned to using LogBatch() for sending log entries from the weavelet to the envelope. Remove the old HandleLogEntry method that is no longer used.

Move trace span propagation from pipe to component method. (#709) Move a low-level trace test to traceio directory. End-to-end span propagation under the multi deployer was verified by temporarily changing example/examples_test to issue a large sequence of HTTP requests and checking that some of these requests showed up in the multi trace database.

2024-01-09

Switch certificate RPCs from pipe to component calls (#708) Also change the MTLS test to only run for collatz since it can be very slow.

Move listener RPCs from pipe to deployer component. (#706)

Move ActivateComponents RPC from pipe to component method call. (#705) Continuing the work to remove pipe-based RPCs, the ActivateComponents call to the deployer is now a component method call. Also made NewEnvelope() responsible for serving calls to the deployer component. (Callers that are directly using NewEnvelopeConn() still have to manage such calls themselves.)

2024-01-03

Move GetLoad from a pipe RPC to a controller component method. (#702)

Move GetMetrics from a pipe RPC to a controller component method. (#701)

2024-01-02

Move GetHealth from a pipe RPC to a controller component method. (#699)

2023-12-19

Make profile collection a component method. (#691) Also, stop tracking whether or not CPU profiling is already on explicitly; let the Go runtime do it. * Update godeps * Fix unused function lint error

Move UpdateRoutingInfo RPC from pipe to Controller component. (#689)

2023-12-18

Drop UpdateComponentsRPC. (#687) An earlier changed switched to using a controller component method call to update the set of components served by a weavelet. This change finishes that by dropping the UpdateComponentsRPC from the pipe protocol. The weavertest multi deployer code had to be refactored a little bit to give it access to the controller component (either local for components hosted in the main process, or remote for components hosted in other processes). Also dropped the GetHealth method from the controller component since it is currently not used. (The only user of GetHealth is an out of tree deployer that has not yet been switched to using the controller component.)

2023-12-15

Use component method calls for some of the envelope->weavelet calls. (#686) * Use component method calls for some of the envelope->weavelet calls. Previously, all envelope initiated communicaton to the weavelet was carried over a pair of pipes. This change switches some of that communication to be based on component method calls. In particular, the signal to start components is now carried by a method call (UpdateComponents) to a component hosted by the weavelet. Main changes ------------ 1. Introduce a controller component that is hosted by every remoteweavelet. 2. The envelope picks the name of a Unix domain socket and passes it to the weavelet it creates. 3. The envelope creates a stub pointing at this socket and makes calls to it to control the weavelet. (Currently only UpdateComponents is handled this way.) 4. remoteweavelet creates a controller implementation to handle method calls received over this Unix domain socket. Other changes ------------- 1. envelope.NewEnvelope() now takes an Options struct as an argument. This struct can contain an optional Logger and an optional Tracer. The supplied Logger and Tracer are used for calls made to the controller component. 2. envelope.NewEnvelope() creates a temporary directory and allocates a Unix domain socket inside the directory. It also arranges to remove the directory when cleaning up (by catching termination signals for examples). 3. The socket name is passed in the EnvelopeInfo proto sent to the weavelet. 4. RemoteWeavelet creates a controller component and listens for calls to it on the supplied socket. 5. Added deployers.NewUnixSocketPath(), which allocates a new socket name inside a supplied directory. 6. weavertest multi deployer manages its own socket creation when it bypasses envelope.NewEnvelope(). 7. website/blog/deployers/deployers_test.go kills child processes using SIGTERM instead of SIGKILL. This allows the child process a chance to clean up its temporary directories. 8. deployers_test.go logs the stderr and stdout lines it receives instead of dropping them on the floor. This can be very helpful when things aren't working as expected. 9. The deployer examples in the blog import the weaver package so they have access to the controller component registration.

2023-10-19

Add optional names for colocation groups We've discussed to enable the user to specify group names. Right now we compute automatically the group names as the name of the first component in each group. This PR enables the user to specify group names (optionally). This is becoming more and more important given that we allow the user to configure various knobs in the deployer, and many knobs make sense for a particular colocation group. E.g., secrets/config maps make sense for colocation group. Maybe we will allow people to configure resource requirements per colocation group at some point, etc.

2023-10-06

Remove usages of runtime.Deployment in weaver (#653) We use runtime.Deployment in very few places. This PR get rid of protos.Deployment usages in the weaver repo.

2023-09-21

Use component method calls for multi deployer logging. (#630) We plan to make it easier to create deployers by providing pluggable system facilities (logging for example). We will achieve this by using component method calls to communicate between a weavelet and its envelope. This change starts us on this path by adding a logging component and changing the multi deployer to collect log entries via method calls made by the weavelet: 1. The weavelet contains a Logger component that logs to stderr. 2. The multi deployer has a different component multiLogger that has the same set of methods as the Logger component, but writes log entries to the multi deployer's database. 3. This multiLogger component is made available over a unix domain socket. 4. The multi deployer passes configuration information to the weavelet asking it to redirect calls made to the Logger component to the multiLogger component. 5. The weavelet calls a Logger method to write log entries (though for the benefit of other deployers, it falls back to using the envelope pipe protocol if a Logger redirect is not available). Other related improvements -------------------------- The multi deplopyer could previously lose log messages since the process could exit before the log messages written to the logs database could be read and printed to stderr. We now write log messages to stderr as soon as they arrive instead of routing them through the logs database. Other Details ------------- * Fix generator so it can handle components defined in the weaver package. * multi deployer creates a temporary directory and a unix domain socket inside it. * The Logger component method accepts a batch of log entries, which should help efficiency when the rate of logging is high. * Fixed comment with outdated example of logging output. * Added a list of redirects to `EnvelopeInfo` proto. Added helpers ------------- * `call.NewHandlerMap()` provides a single place to register ready method. * `FixedListener` provides a call.Listener that serves a fixed set of components that are not subject to any communication graph constraints. * `reflection.ComponentName[T]` returns the name of component T. * `codegen.Find` returns registration by name. * `dirs.NewTempDir` makes a provate temporary directory. * `logger.IsSystemGenerated` determines whether a log entry is system generated.

2023-08-23

Removed colocation group from status server. Recall that the single, multi, and ssh deployers all run status servers which report the status of their applications via "weaver status" and "weaver dashboard". This PR removes the outdated use of co-location groups in the reported status.

2023-08-14

Replaced InternalPort with InternalAddress. Recall that a weavelet listens on an internal listener to serve remote procedure calls from other weavelets. This PR simplifies how a weavelet picks the address for this internal listener. Previously, the address was dictated by two fields inside `EnvelopeInfo`: `SingleMachine` and `InternalPort`. The weavelet would listen on `$HOSTNAME:InternalPort` if `SingleMachine` was false and `localhost:InternalPort` otherwise. This PR removes the `SingleMachine` field and replaces `InternalPort` with `InternalAddress`. A weavelet listens on the `InternalAddress` provided by the envelope.

2023-08-11

Removed Pid field from WeaveletInfo proto. This PR removes the mostly unused Pid field from the WeaveletInfo proto. The field is unnecessary, as an envelope already knows the pid of the weavelet its running.

Removed SingleProcess field from EnvelopeInfo. The SingleProcess field was no longer being used. Since this is a change to the deployer API, I also bumped the deployer API version.

2023-07-17

Store trace spans in a relational DB format. (#459) * Store trace spans in a relational DB format. This format allows for traces to be queries. Here are the changes to the trace storage format: * Traces are stored in encoded proto format. This format is more efficient than the earlier Perfetto JSON format. * Some trace fields are also available as table columns, for fast query matching. * Traces are converted to the Perfetto JSON format during trace retrieval and display. As discussed offline with sanjay@, traces are no longer displayed in Perfetto in bulk. Instead, each trace has a separate link and the Perfetto UI always displays a single trace. This makes it easier to visualize individual trace executions, as they are no longer interleaved on the same timeline. Other changes: * Fold Span proto messages into the `Span` type, as their names are quite generic (e.g., `Attribute`). * Pass trace data between weavelet and envelope in Proto format, since the Proto format can be efficiently serialized, if needed. (The OTel format doesn't support efficient serialization.) * Minor code re-organization. COMPATIBILITY NOTE This change requires a deployer version bump, as it changes the trace wire format. * Addressed final comments.

2023-06-20

Fix the reading of the SSH config. (#408) In particular, read the config once and with all the right fields specified.

Small tweaks to the listener configuration. (#404) Namely: * Listener names are case sensitive. * Deployers validate listener names in the config.

2023-06-14

Modify the listener format for local listeners. (#394) This fixes the weirdness of the "local_address" field, which is only used by multi/single/ssh deployers, but not others (e.g., GKE, kube). The plan is to move the listener options into the deployer config sections, since they are largely different. Based on discussions with @rgrandl and @mwhittaker. Other changes: * Simplify the parsing of the listener config, by reading into a `map[string]ListenerOptions`. This obviates the need for using the `toml.Primitive` type. TODO: Change the format of the GKE/kube deployers to be similar.

2023-06-12

Declare listeners as fields in the component implementation. (#380) This will allow us to build a static map of component -> listeners, which a deployer may read from a binary to initialize the listeners ahead of time (e.g., a Kube deployer). Other changes: * Listeners are customized in the application config file. * The weavelet no longer knows about listener options. * Slight simplifying changes to the envelope protocol.

2023-06-08

Add instrumentation scope to the read span. (#378) The ReadOnlySpan has a new field `InstrumentationScope` that is supposed to replace the `InstrumentationLibrary` that is deprecated. However, our internal implementation of the span doesn't override this field, hence the exported spans have it nil. When using the jaeger exporter, it relies on the instrumentation scope to get some info about the spans. This PR overrides the instrumentation scope field as well, in our internal span implementation.

2023-06-01

Propagate the internal listener port (#371) In the current implementation, every weavelet will listen for internal traffic at an arbitrary port. However, for some deployers (e.g., kubernetes based deployer), we will prestart the listeners hence the port has to be propagated from the deployer. This PR adds a new field `internal_port` that contains the port number for the internal listener. By default this is 0, which preserves the current behavior. However, in a future PR, the `internal_port` value will be used by the kubernetes deployer.

2023-05-30

Remove local_address from GetListenerAddressRequest (#358) When the weavelet sends a `GetListenerRequest` to the deployer, it also propagates the local address. However, the local address is not used by any deployer on this path. I found passing local address on `GetListenerRequest` very confusing, but then I realized that this field is never used.

Remove local_address from GetListenerAddressRequest When the weavelet sends a `GetListenerRequest` to the deployer, it also propagates the local address. However, the local address is not used by any deployer on this path. I found passing local address on `GetListenerRequest` very confusing, but then I realized that this field is never used.

2023-05-18

Request fresh certificates on every connection establishment. (#347) This is required as certificates in general have a short validity and typically get rotated. Previously, we would pass the certificate to the weavelet and use it indefinitely. We now request a new (i.e., latest) certificate from the weavelet each time a connection is established, ensuring that the freshest certificate is always used.

2023-05-17

weavertest tests can now run in a single process mode with RPCs. (#340) The "SingleProcess bool" option has been replaced by a mode that can be either Local or Multi or RPC. The RPC mode places every component in the testing process, but uses RPCs for method calls. This mode is typically useful when collecting profiles or coverage data that should include RPC costs. (Multi runs will also contain RPC costs, but are harder to profile since their execution is split across multiple processes.) The mode argument has also been moved out of weavertest.Options{} so it no longer has a default value and the caller is forced to specify whether they want a Local or Multi run.

2023-05-10

Use the call graph to authorize peers in the multi deployer. (#313) In this change: 1. The multi deployer, when constructing colocation groups, also computes the set of components each colocation group is allowed to call methods on. 2. When the server receives a new client connection, it calls `VerifyClientCertificate` on the multi deployer. The multi deployer extracts the colocation group name from the client certificate and returns the pre-computed set of components (see step 1 above) the client's colocation group is allowed to access. 3. (Already implemented) The weavelet makes sure that a given connection only admits method calls on the allowlisted components (see step 2 above). Other changes: * Verify WeaveletInfo inside envelope_conn. This removes a bunch of redundant code.

2023-05-01

weaver multi signs and verifies certificates (#310) Change the `weaver multi` deployer to: * Mint a new CA certificate for each new deployment. * Mint a signed certificate for each colocation group. * Verify peer certificates to ensure that: 1. their identities were assigned by the deployer, and 2. (TODO) the identities match the call graph. Adds the certs package used for minting certificates in-process.

2023-04-26

Send PKI certificates between weavelets. (#289) In particular, the envelope now has the ability to pass a certificate to its weavelet. In such cases, the weavelet will use the certificate to authenticate itself to its network peers, and to authenticate and authorize those peers. The intended protocol is as follows: * Envelope optionally passes a certificate to the weavelet. * (TODO) The weavelet passes the call graph to the envelope. * The weavelet establishes a TLS connection with its peers. * On every network connection, the weavelet calls an envelope method to verify the peer's certificate. * (TODO) If the peer is a server end of a connection, the envelope verifies that the peer has provided the correct identity for the target component. * (TODO) If the peer is a client end of a connection, the envelope verifies that the peer is supposed to talk to the weavelet and returns all of the components the peer is allowed to access. Other minor changes: * fold `componentStub` into `component` * fold `client` into `component` * de-dup `dialAddr` between weavelet and weaveletconn. * small cleanups

2023-04-05

Added prototype versioning system. > Version Issues Service Weaver is currently at v0, which means we're making a lot of breaking changes to the code. This has the problem that a deployer and the app its trying to deploy may be at incompatible versions. For example, imagine someone is trying to learn Service Weaver. They run `go install github.com/ServiceWeaver/weaver@latest`. A couple weeks later, they start writing a new Service Weaver application and run `go mod tidy`. `go mod tidy` picks up the latest version of Service Weaver which is likely incompatible with the stale version of the `weaver` tool they downloaded a couple weeks back. As a result, the `weaver` tool will likely fail to deploy the application, and it won't produce any meaningful error messages explaining why it failed. > Proposed Solution This PR introduces a proposed solution. It's not perfect, but hopefully it kickstarts some ideas on how to solve this annoying versioning issue. With this PR, we version the deployer API. A weavelet reports to the envelope the deployer API version it was built with. The envelope can then check whether it supports the reported version, aborting if not. See runtime/version.go for more details.

2023-04-03

Removed requirement that ids are RFC 4122 UUIDs. Before this PR, we required that both deployment ids and weavelet ids were [RFC 4122 formatted UUIDs][uuid]. This PR lifts that requirement. Ids must be globally unique, but we don't impose any specific format on them. For some deployers, using more readable ids is preferable. [uuid]: https://github.com/google/uuid

2023-03-30

Cleaned up runtime.proto. This PR cleans up the protobufs in the deployer API. This PR shouldn't introduce any behavior changes. I mostly renamed things and moved things around to be clearer. For example, StartComponent was renamed to ActivateComponent since it doesn't always start a component. I also added documentation explaining the pipe API.

2023-03-27

Removed groups entirely from deployer API! Recently, we've been doing a lot of refactoring to remove co-location groups from the deployer API: #184, #182, #181. This PR caps this effort off by removing them entirely. Now, the weavelet is completely unaware of groups. It starts whatever components the deployer tells it to start. To enable this change, I added a boolean `Local` field to the `RoutingInfo` proto. When this field is set, a weavelet routes requests locally. Otherwise, it routes requests over the network. Note that this is orthogonal to whether or not a component is running locally. A weavelet may want to call a remote instance of a component even if it has a local instance running. Routed objects, for example, do this to respect sharding. This also changes the flow of weaver.Get. Before, when a weavelet called weaver.Get[T], if T was local and unrouted, the weavelet would create a local instance and route to it immediately. Now, Get[T] blocks until it receives a RoutingInfo from the deployer to learn whether it should be routing locally or remote. Right now, the decision to route locally is rigid. For a given component, you either always route locally or always don't. In the future, we could make this more flexible.

Made envelopes push components and routing info. Before this PR, a weavelet issued blocking gets [1] to the envelope to learn which components to run and to get updated routing information. This version-based long polling API was complicated to explain. People had to understand what long polling was and learn the details of our version semantics. I also felt that it was a bit tricky to implement a deployer using this API. This PR replaces the long polling with envelope-initiated RPCs: UpdateComponents and UpdateRoutingInfo. Now, whenever the set of components changes or a component's routing info changes, the deployer pushes an update to the weavelet. I think this API is simpler, but a deployer implementer does have to be careful not to forget to call UpdateComponents or UpdateRoutingInfo. This can be tricky when events happen in unexpected orders. For example, a weavelet may subscribe to a component before the component exists. With this API, our weavelet also has to be careful about how it handles UpdateComponents and UpdateRoutingInfo. In general, a weavelet handler should never block or touch the pipe. Doing so makes it easy for a deployer to deadlock. For example, imagine an envelope receives a StartComponent request. It grabs a lock and then issues an UpdateComponents request. If the weavelet then issues another RPC over the pipe, and the envelope handles this RPC by grabbing the same lock, there is deadlock. === Future Work === Note that there are still lingering artifacts in the pipe API. There are still co-location groups here and there despite the work in PR #181. The weavelet still assumes placement based on groups. I'll continue to fix these issues in future PRs. === Details === I removed versioned.Versioned from the weavertest and weaver multi deployers. The SSH deployer still uses it, but we can remove that in the future. [1]: https://javascript.info/long-polling

2023-03-23

Remove RegisterReplica method from the envelope <-> weavelet protocol. (#184) Instead, we get the weavelet's runtime information immediately after creating the weavelet. Suggested by mwhittaker@. Other changes: * Minor Envelope cleanup.

2023-03-22

Removed co-location groups from bulk of pipe API. Recall that, before this PR, the deployer API had four methods related to weavelets and components: RegisterReplica, StartComponent, GetComponentsToStart, and GetRoutingInfo. The notion of a co-location group was baked into these APIs. A weavelet would ask for a group's routing info, not a component's, for example. This PR removes the idea of a co-location group from the APIs and makes them component-centric instead. This is desirable because co-location groups are implementation details and should not be forced into the API. Deployers are still free to use the notion of a co-location group internally---and right now, all our deployers do---but they do not have to. Note that this change does not eliminate co-location groups from the API entirely. There are still some message that reference groups, and the weavelet still uses the idea of a group. Future changes to the pipe API should eliminate these entirely.

Change load report from push to pull

2023-03-20

Cleaned up SSH deployer's manager. This PR cleans up the SSH deployer's manager. Specifically, I ported the changes we made to the 'weaver multi' babysitter in #172 and #174. These cleanups will make it easier to make some of the deployer API changes we've been working on. I noticed that the SSH babysitter re-uses a lot of the same code as the 'weaver multi' babysitter. I don't think we should merge the two into one mega babysitter. In fact, I recently split the 'weaver multi' babysitter from the weavertest babysitter, so I think there's value in having things separate. But, maybe we can pull out some of the common code without having a frankenstein babysitter API.

2023-03-17

Cleaned up the multi/weavertest babysitter. This PR cleans up the babysitter used by weaver multi and weavertest. The changes don't impact the behavior of the babysitter; they simply clean up the code. I made these changes in preparation for some bigger changes I'm making in an effort to simplify the deployer API. I made the following changes: - I replaced the babysitter's use of versioned.Map with a new versioned.Versioned abstraction (first introduced in #162). Previously, a babysitter was storing a single piece of state in a map, which was a bit confusing. The new Versioned abstraction allows for the next change. - I introduced a group struct to store all the information about a group. Previously, the state was spread across a couple different fields, and in some versioned.Maps. - I removed babysitter.proto. I realized that there was no need to be using protos. Now, everything is vanilla Go. - I moved the code in profile.go to babysitter.go, so the babysitter is now a single file. Ironically, a lot of the changes introduced in this PR will be deleted in future PRs as we transition to the deployer API, but these changes will help that transition.

2023-03-16

Ran clang-format on runtime.proto. I ran clang-format on runtime.proto to fix up the formatting. I reworded a couple of comments to avoid some awkward line wrapping.

2023-03-13

Fixed UseLocalhost bug introduced in PR #115. Before PR #115, the WeaveletInfo proto had a UseLocalhost field. When true, weavelets would listen on localhost for external listeners (the kind returned by the Instance.Listener method). In PR #115, I removed the UseLocalhost field and replaced it with a GetAddress API. However, I didn't actually remove the field, and the field was being used for more than I initially thought. The UseLocalhost field was still in WeaveletInfo and was being used for internal listeners (the kind that a weavelet uses to execute RPCs). The PR changed 'weaver multi' to not set the UseLocalhost field which led to weavelets listening on $HOSTNAME rather than localhost. This PR fixes the bug. I replace UseLocalhost with SingleMachine since it parallels the existing SingleProcess field better and I think it's a bit clearer (use localhost for what?). I also removed some related unused fields from the WeaveletInfo proto. Fixes #154.

Removed StartColocationGroup from the pipe API. Consider a component A that calls weaver.Get on a different component B. Before this PR, the weavelet hosting A would send two messages over the pipe to the envelope. 1. It would send a StartComponent message indicating that the B component should be started by B's colocation group. 2. It would send a StartColocationGroup message indicating that B's colocation group should be started, if it hasn't already. This PR deletes the unneeded StartColocationGroup message. When a weavelet sends a StartComponent message, the envelope now registers that the component should be started *and* starts its colocation group, if it hasn't already. This simplifies the pipe API and makes implementing deployers a little bit easier. I also removed some unneeded fields in the StartComponent proto.

2023-03-10

Don't treat colocation group replica ids as UUIDs. (#141) GKE deployer uses pod names as replica ids, for debugging and to be able to quickly determine the set of replicas, by examining the set of all pods.

2023-03-09

Remove references to "process" from our runtime types. (#134) The term process is a leftover from when we used to have colocation groups hosting multiple processes. Other changes: * Get rid of the colocation-group internal transport. * Small drive-by changes.

2023-03-07

Improved ExportListener API. Before this PR, the flow for exporting a listener was a little complicated. A weavelet would receive two listener related options via a protos.Weavelet: use_localhost and process_picks_ports. If use_localhost was true, a weavelet would listen on localhost. Otherwise, it would listen on $HOSTNAME. If process_picks_ports was true, the weavelet would listen on port 0. Otherwise, it would call ExportListener to get a port. Thus, ExportListener sometimes exported a listener and sometimes didn't (and returned a port). This PR simplifies the API by splitting removing use_localhost and process_picks_ports options completeley and by splitting ExportListener in two. First, a weavelet calls GetAddress to get the address it should listen on for a listener. For example, the single process deployer may return "localhost:0", while the SSH deployer may return "$HOSTNAME:0". Second, the weavelet listens on this address and calls ExportListener to export the listener. This flow is much simpler, but it does require an additional RPC over the pipe. However, exporting listeners is not on critical path, plus there is already significant traffic over the pipe (for every log entry, to export metrics, etc.). Overall, I think the performance hit is negligible.

Replaced Weavelet proto with concise WeaveletInfo. (#114) Before this PR, we had a Weavelet proto with an embedded Deployment proto, which in turn had an embedded AppConfig proto. When you launched a weavelet, you passed it a Weavelet proto over the pipe. However, the Weavelet proto contained a lot of unnecessary fields that the weavelet never actually used. For example, does a weavelet need to know its binary? No. This made it hard to know which fields were needed and which weren't. This PR replaces the Weavelet proto with a simpler WeaveletInfo proto that only contains the fields a weavelet actually uses. With this change, the Deployment proto becomes more of a deployer implementation detail. We could probably remove it from the runtime package if we wanted to, though I'll defer that. In future CLs, I think we can make the WeaveletInfo proto even simpler. Co-authored-by: Srdjan Petrovic <spetrovic@google.com>

2023-02-22

Store perfetto traces in a database. (#50) This is necessary in order to allow GKE-local babysitters to store traces directly, i.e., without sending them through the manager/distributor/controller so that the controller can store them. While at it, moved the trace-server code to the dashboard.

2023-02-15

Renamed log fields to align with slog. This PR renames the following log fields in the log query API: - severity -> level - file & line -> source - payload -> msg This makes the field names align with the soon to be standard slog [1]. Thanks @ghemawat for the suggestion. === Attributes === Consider the following debug log. c.Logger().Debug("a debug log", "foo", "bar") Currently, we query the "foo" attribute like this. attr["foo"] == "bar" Sanjay also suggested we re-work the log query language to make it more convenient to query attributes. Specifically, we hoist the attributes out of the attrs map: foo == "bar" Then, we might move the baked in fields (e.g., app, deployment source) to a field like this: weaver.app == "todo" && foo == "bar" I tried implementing this, but it was hard, given our current implementation. The challenge is that we don't know attribute names ahead of time. Users can log whatever attributes they want. I couldn't figure out how to easily form CEL queries without knowing the variable names ahead of time. I'm guessing we can make it work, but I'll have to try harder. There's also an API design question. How do users write a query testing if an attribute exists at all? Today, they can write : "foo" in attrs If we hoist attribute names, this breaks. We can maybe replace it with something like: has(foo) Overall, hoisting attributes will require some more thinking. [1]: https://pkg.go.dev/golang.org/x/exp/slog [2]: https://github.com/google/cel-spec/blob/master/doc/langdef.md

2023-02-06

Initial commit.