Proto commits in kelda/blimp

These 53 commits are when the Protocol Buffers files have changed:

2020-10-28

blimp: Add backend This commit copies the backend code that was previously closed source into this repo.

The documentation is generated from this commit.

auth: Remove auth0 authentication backend This makes self hosted clusters completely isolated -- they don't depend on any central services run by Kelda. Authentication is no longer done through auth0. The "token" field in protobufs are now just a way for clients to provide a unique identifier for their namespace. The only authentication is at the cluster scope via the shared cluster token.

The documentation is generated from this commit.

Fix imports The backend code lived at github.com/kelda-inc/blimp before.

2020-10-26

Add backend

2020-10-09

auth: Auto-refresh auth tokens Users now only have to login once -- the CLI will automatically fresh new access tokens when they expire.

2020-10-06

auth: Add support for cluster authorization This updates the protobuf to use a generic auth message which includes an optional cluster_auth field. The "old_token" field is preserved in the protobuf, so that the backend can still read the token from old clients that send that field instead of the new auth field. To use the new cluster_auth token, simply set the "cluster_token" field in ~/.blimp.yaml. This also generalizes some areas of the code that rely on auth being a single string, such as the registry credential for the docker registry.

2020-09-23

expose: Update wording to reflect multi-service support Also, update comments in the protobuf to reflect changes in how it is used for ExposedTunnel.

2020-09-11

proto: Make BlimpUpPreview return a stream of the CLI output

proto: Make BlimpUpPreview take environment variables

2020-09-09

proto: Add RPCs for booting preview This adds the BlimpUpPreview RPC, which will be implemented by the server.

2020-09-08

proto: Add compose files option to BlimpUpPreview This makes it possible to work with projects that don't have their compose files at `docker-compose.yml` in the root of the repo.

2020-09-06

proto: Add RPCs for booting preview This adds the BlimpUpPreview RPC, which will be implemented by the server.

2020-09-03

proto: Add RPCs for interacting with buildkit

2020-08-01

proto: Add ExposedTunnel to node RPCs This will be used by internal Blimp components to facilitate exposed ports.

cli: Add `blimp expose` command This allows you to expose an HTTP service over at a publicly available domain, protected with HTTPS.

2020-07-29

status: Add "Preparing" phase to sandbox status This allows us to report when a sandbox has been created but service have not yet been deployed.

2020-07-05

proto: Add method for attaching to sandbox This allows clients to get the connection information necessary to connect to a sandbox, without doing a full deployment.

2020-06-29

down: Add --volumes flag for deleting volumes A recent change in the backend made it so volumes are persisted across `blimp down`s. This commit makes it possible to clear the volumes with `blimp down -v`.

2020-06-17

status: Report unschedulable pods This gives a better error message in edge cases where some pods in a sandbox may be unschedulable.

build: Pre-push base images Before, image pushes could be very slow, especially on bad internet connections. Now, blimp asks the cluster-controller to pre-push the base image as defined in the Dockerfile for any custom imges. Then, when the custom image is pushed, Docker will recognize that the layers from the base image are already present on the registry, and will not push these layers. Co-authored-by: Kevin Lin <kevin@kelda.io>

2020-06-16

cli: Add `blimp restart` This will let you restart the pod for an individual service.

2020-06-11

cli: Remove from repo The CLI has been pulled into a separate repo.

Fork code from old repo

2020-06-10

wait: Convert DependsOn to a slice This allows us to sort the values, which makes sure that the resulting marshalled wait spec is deterministic. Changes due to go's random iteration order in maps were causing unnecessary pod restart. Co-authored-by: Kevin Lin <kevin@kelda.io>

2020-06-09

sync: Force the remote volumes to match the local when starting We now initialize Syncthing in two phases: 1) The Syncthing daemon on the local CLI starts in Sendonly mode, and sends Override requests until the remote Syncthing matches the files in the CLI. 2) The CLI Syncthing switches to Sendreceive mode, and does two way sync as normal. This commit also removes the hash-based approach of blocking boot to avoid scenarios where the container starts before Syncthing switches into Sendreceive mode. If we didn't, then any files created by the container while Syncthing was still in Sendonly would get deleted.

2020-06-05

volume: Block boot until volumes are initialized Before, if you booted a compose file where a volume was shared with multiple services, the services would boot independently, so there was no guarantee that the volume would be fully initialized by the time all services started. Compose deals with this by separating boot into two parts -- creating the containers (which includes initializing any attached volumes), and starting them. The containers are only started once _all_ the containers have been created. This commit does something similar, but only blocks boot for services that reference the same volume. Only blocking for services that reference the same volume, rather than all services, should make the boot feel a bit faster for users.

2020-05-21

node: Replace sandbox controller with node controller Before, each sandbox had its own public IP that the CLI would connect to in order to send bind volume hash update, and tunnel traffic. However, allocating IPs is serialized by GKE, so that would bottleneck sandbox creations. Plus, there was no way to clean up sandbox state after the namespace is deleted, since the sandbox controller lived in the same namespace as the customer pods. This commit makes it so each Kubernetes node has a controller that handles all the inbound connections from the CLI for all sandboxes running on the node. This controller is also responsible for cleaning up volumes after namespaces are deleted, and ordering boots for customer containers. Note that we still run a per-sandbox DNS server, since our DNS library doesn't allow us to tell what IP made a DNS request.

2020-05-19

down: Wait until sandbox terminates before exiting This is helpful for scripts that run `blimp down` followed by `blimp up`.

2020-05-14

compose: Implement healthchecks This commit makes it so we run any healthchecks specified in the Docker Compose file, makes the CLI show the status of the checks, and updates the depends_on logic to respect dependencies on service_healthy.

2020-05-11

manager: Fix protobuf numbering for version check We need it to be backwards compatible so that clients know to upgrade.

blimp: Add FriendlyError type This commit adds a new error type that lets us specify a helpful error message to users. When printing errors, blimp automatically looks for any errors that are in the "helpful" format, and print out the message, rather than printing out the full error tree.

2020-05-10

status: Handle image pull states better We now tell the user when a pod is stuck in pending because the image failed to pull, and when the image is pulling. To achieve this, the status API now has a Phase field, which has a limited set of valid options, and a Message field, which can contain arbitrary additional information.

2020-05-08

volumes: Support mounting individual files This commit rewrites our syncthing approach to allow mounting individual files. Before, we only supported mounting directories. The syncthing pod now syncs all of the files into a single Kubernetes volume. The structure of the volume mirrors the structure of the user's filesystem, and the cluster controller just projects the appropriate pieces of the volume into pods via VolumeMounts. The CLI is responsible for ensuring the files referenced by volumes are synced. For example, when mounting an individual file, the CLI actually syncs the parent directory, and uses a stignore file to ensure that only the file that's being mounted is synced.

2020-05-05

login: Rewrite flow for getting credentials Before, the CLI obtained the login token after logging in from the browser by spinning up a local web server, and setting the OAuth redirect URL to localhost. This was brittle, since it depended so heavily on the configuration of people's local system. For example, we ran into people who had other processes listening on the same port locally, and people with odd DNS setups. This commit makes it so the OAuth token is retrieved server-side, and pushed back to the CLI.

2020-04-12

wait: Properly block container boot on initial volume sync The old code that blocked the container boot didn't work because of a number of incorrect assumptions about the Syncthing API. Because Syncthing doesn't have a concept of "an initial sync", this patch makes it so we determine when the sync is complete by having the CLI publish a hash representing the expected contents of the volume. We then delay starting the user's container until the remote hash matches the expected hash.

build: Remove vendor

2020-04-02

cli: Start collecting CLI analytics They now get proxied to the cluster manager, which posts them to DataDog.

dockercompose: Use compose-spec/compose-go

vendor: Add vendor directory

2020-03-31

cli: Print a friendlier message if we can't parse Docker Compose file

cli: Check Version This patch causes the cli to send the manager its version and ask it for instructions how to proceed. Currently the manager always allows the cli to go forward, but this patch lays the groundwork to make different decisions later as needed.

2020-03-30

volumes: Make volume identifier unique for relative bind volumes Before, if the user switched to a Docker Compose file that was in a different directory, but had a bind volume with the same relative path (e.g. `.`), Blimp would use the same host directory for the volume. This would lead to the other volume's files getting synced back to the local filesystem.

2020-03-29

volumes: Block booting container until initial volume sync completes Before, it was possible for the volume to start before syncthing synced the local files into the bind volumes, which would cause applications to crash.

2020-03-27

up: Warn if Compose file references unsupported fields

tunnel: Authenticate calls to the tunnel API We now require that the client provide an Auth0 token associated with the target namespace.

boot-waiter: Rewrite DependsOn logic to talk to sandbox controller Before, the init container just pinged the containers it depended on. This commit makes it so the init container polls the sandbox controller instead. This lays the foundation for blocking container boot until bind volumes are fully synced.

registry: Forward local registry credentials to the cluster This lets users run private images as long as they're logged in to the registry on their local Docker daemon.

cluster: Make the Docker Compose file available to CreateSandbox Ethan needs it for the Syncthing code.

2020-03-26

blimp: Use TLS for all gRPC communication This commit makes it so we use TLS when communicating between the CLI and the cluster manager, and the CLI and the sandbox manager.

2020-03-19

tunnel: Implement localhost tunneling This patch creates an initial implementation of localhost tunneling. It's likely buggy and a bit fragile, but it's a reasonable starting point.

down: Add command to delete development sandbox

ps: Add support for getting sandbox status It currently just displays the services and pod phase, but we can now easily add other attributes like uptime.

2020-03-17

Basic scaffolding