Proto commits in e2b-dev/infra

These commits are when the Protocol Buffers files have changed: (only the last 100 relevant commits are shown)

Commit:be43b9a
Author:Jiri Sveceny
Committer:e2b-bot[bot]

refactor(scheduling-api): generate scheduling protos as contracts GitOrigin-RevId: 6f95478318f0f8c8ea36b045fc6ee30eb226c664

Commit:eb31633
Author:Jiri Sveceny
Committer:e2b-bot[bot]

feat(scheduling-api): add removal and transition RPCs GitOrigin-RevId: a432b9478950bbdc06950741e455c7b5c9e8039b

Commit:48bc3b5
Author:Jiri Sveceny
Committer:e2b-bot[bot]

feat(scheduling-api): add update and lifecycle hold RPCs GitOrigin-RevId: 66bd6daa1c2c1a7cab429774d7d7d87b579ee62e

Commit:e75c5d4
Author:Babis Chalios
Committer:e2b-bot[bot]

feat(orchestrator): start sandboxes with a requested memory size through hotplug Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> GitOrigin-RevId: e501782f9295d981ba36197cab2d581d6713d617

Commit:82bc7bf
Author:Jiri Sveceny
Committer:e2b-bot[bot]

feat(scheduling-api): add sandbox record RPC handlers GitOrigin-RevId: 18226181dea893e8f8660addd362b0b30c211776

Commit:db8dcf0
Author:Jiri Sveceny
Committer:e2b-bot[bot]

feat(scheduling-api): add reservation RPC contract and handlers GitOrigin-RevId: ea4d77c4ab83dfc3f75f262497504d675cbfecda

Commit:289e9a8
Author:Jiri Sveceny
Committer:e2b-bot[bot]

refactor(scheduling-api): remove v1 RPCs and PostgreSQL store GitOrigin-RevId: 78e02ca490778937d870d678f40e05868feb359d

Commit:94411b4
Author:Jiri Sveceny
Committer:e2b-bot[bot]

feat(scheduling-api): add sandbox lifecycle RPCs GitOrigin-RevId: f718fe4d93b2829d6cb1a6d6e13262d873b12158

Commit:05cf7c2
Author:Babis Chalios
Committer:e2b-bot[bot]

feat(orchestrator): build templates with a memory maximum for hotplug Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> GitOrigin-RevId: aa1a09f0643be6eeedf2d079cc77aa264ed4ad9c

Commit:4d088fd
Author:Jiri Sveceny
Committer:e2b-bot[bot]

feat(scheduling-api): add sandbox state service and start RPCs Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> GitOrigin-RevId: e2e3dcdc92acc18bc0efb28bfa44d87b19f68525

Commit:d69d6ac
Author:Shan Valleru
Committer:e2b-bot[bot]

feat(orchestrator,api): carry standby owners and capabilities on nodes and skip a status-refusing node Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> GitOrigin-RevId: 2654c7db932ec6179ff68f9a389e606c8d9ed142

Commit:ae316c1
Author:Petr Vaněk
Committer:e2b-bot[bot]

refactor(orchestrator): stop sending sandbox config in the running sandbox list GitOrigin-RevId: 21675470fe6168e4c9d45db7cf86da83a5653a6b

Commit:7d9932e
Author:Shan Valleru
Committer:e2b-bot[bot]

feat(orchestrator,api): owner, force and capabilities on the node status contracts GitOrigin-RevId: a6f67425012c29adb56b13cd0a46299cf3f49580

Commit:c71b736
Author:Petr Vaněk
Committer:e2b-bot[bot]

refactor(api+orchestrator): always use huge pages for sandboxes and template builds GitOrigin-RevId: 60af8b5d286677c10d70b039af30b0d355f9d7fe

Commit:207cf90
Author:Tomas Srnka
Committer:e2b-bot[bot]

feat(orchestrator): gate the sandbox lifecycle on the guest profile GitOrigin-RevId: 02c3c8bee2a8ff053beb59ea883d2057b727c95e

Commit:e85b9f5
Author:Babis Chalios
Committer:e2b-bot[bot]

feat(orchestrator,api): record a pause taken over an unresponsive guest agent and refuse its memory resume Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> GitOrigin-RevId: edc7e25713caf66d1cdafc8ecb842ea8304fe5ea

Commit:0c68693
Author:gm-e2b
Committer:e2b-bot[bot]

fix(orchestrator): report NotReady until a starting node can serve GitOrigin-RevId: ce8050271b63c14a225e2e01ba0670396b493470

Commit:70ffc05
Author:Babis Chalios
Committer:e2b-bot[bot]

feat(api,orchestrator): after a grace period the eviction sweep takes the memory pause without the envd probe Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> GitOrigin-RevId: d58875705992041064ba3c9eb7d99848bbc8872c

Commit:9a9db27
Author:Babis Chalios
Committer:e2b-bot[bot]

feat(orchestrator,api): probe envd before a memory pause and refuse with a typed reason when it does not answer Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> GitOrigin-RevId: 6f828a72c219e22ac3e4dd0891f37562b8b8ce6d

Commit:13faa4d
Author:Babis Chalios
Committer:e2b-bot[bot]

fix(api,orchestrator): name the checkpoint-in-progress refusal and answer it as a retryable 503 Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> GitOrigin-RevId: 3358f3cd56fd6ef5017ba1ae24d7659abf0c04d1

Commit:c3c992c
Author:gm-e2b
Committer:e2b-bot[bot]

feat(api): return saved network placement from sandbox resume GitOrigin-RevId: a9828362abfd8777d20d8cde67f06d5cc65bab75

Commit:6b82caf
Author:gm-e2b
Committer:e2b-bot[bot]

feat(orchestrator): commit network updates through the network-assign hook and record checkpoint placements GitOrigin-RevId: 117df024da6237d311479b3b459faa9e6cf188e5

Commit:331b3b0
Author:Babis Chalios
Committer:e2b-bot[bot]

fix(api): score the hugepage pool against the base pool, not HugePages_Total Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> GitOrigin-RevId: f487ec76e9c051e9e8024afbd41f7d6ae7aff2eb

Commit:b100be0
Author:gm-e2b
Committer:e2b-bot[bot]

build(orchestrator): compile the orchestrator protos from the contracts tree GitOrigin-RevId: 31be9fb081d09b307c6af26cffad2e21953221c5

Commit:d5f3532
Author:Alex Sardan
Committer:e2b-bot[bot]

feat(orchestrator): build templates with spare vCPUs Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> GitOrigin-RevId: 11df54d4b9124f69460aa1c663ff7e309f1c41f4

Commit:16ec5e6
Author:Ryan Sana
Committer:e2b-bot[bot]

feat(api): refuse placements that would overfill a node's hugepage pool GitOrigin-RevId: 27d7ff4dbe4954b2ed0f84c39319dbddcb29ba0b

Commit:90f3409
Author:Tomas Srnka
Committer:e2b-bot[bot]

feat(client-proxy): one resume flight per sandbox, detached from its leader GitOrigin-RevId: d63044418c82aede5423dccd1fa36fa410eef4e5

Commit:37407f3
Author:Alex Sardan
Committer:e2b-bot[bot]

feat(orchestrator): vCPU hotplug at sandbox start and resume Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> GitOrigin-RevId: 41ed62c3bc428abbcc499bc74ee7bd8e16a8473b

Commit:ba0e76c
Author:gm-e2b
Committer:e2b-bot[bot]

feat(network): add portable routing contracts GitOrigin-RevId: bb8bc1ef3be45309f2fe43dc37e657df69cc7ca0

Commit:e7b859b
Author:Alex Sardan
Committer:e2b-bot[bot]

feat(envd): bring guest CPUs online to the count requested on /init Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> GitOrigin-RevId: 5fc4907aaf8cc4e67a6c9717aaa77a307e0af13a

Commit:1599d6e
Author:Petr Vaněk
Committer:e2b-bot[bot]

feat(orchestrator): optional TLS on the hop to the egress proxy GitOrigin-RevId: 6bbb0090f8060abaab6ed0099b1248e9e470e67c

Commit:bfd9179
Author:Babis Chalios
Committer:e2b-bot[bot]

feat(orchestrator): filesystem-only in-place checkpoint Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> GitOrigin-RevId: 6d975d1ebc8751690bf4a4181f314eb27f7f8e01

Commit:90367fa
Author:Ryan Sana
Committer:e2b-bot[bot]

feat(envd): report whether a filesystem entry is a symlink Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> GitOrigin-RevId: 2d11e88233540195778587a74bcf4df149e10e24

Commit:81cb3b2
Author:Jakub Rojko
Committer:e2b-bot[bot]

refactor(api): require node sandbox limits GitOrigin-RevId: eab8e72dcb26e72882182839fe4c84cd44d182d7

Commit:44a8a75
Author:Jakub Rojko
Committer:e2b-bot[bot]

refactor(api): require node outstanding work GitOrigin-RevId: df717eaf0c0d60ae75f4b3f1a985353b69bcaa2d

Commit:756512c
Author:michalsuba-e2b
Committer:e2b-bot[bot]

feat(storage): sign Azure layer-file uploads with a SAS and return upload headers Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> GitOrigin-RevId: 870d5caecee5bc9cfced61609051f0a54b0c286e

Commit:8cd25be
Author:Michal Suba

feat(storage): sign Azure layer-file uploads with a SAS and return upload headers Azure's Put Blob requires the request header "x-ms-blob-type: BlockBlob". A SAS can only pin response headers, so no signed URL alone can satisfy it and the provider refused to issue one at all — every template build with a COPY instruction failed on Azure at the get-signed-URL step. UploadSignedURL now returns {URL, Headers} instead of a bare URL, and the header travels with the URL through the gRPC and public API responses for the upload client to apply. Azure mints a create+write blob SAS with whichever credential the provider already authenticated with: a user delegation key for a token credential (the managed identity path), or the shared key when one is configured. A SAS-only connection string can sign neither and now says so at construction and at the call. GCS, S3 and the filesystem provider return nil headers and byte-identical URLs. The proto and OpenAPI additions are additive — url stays optional, and a client that ignores the new field parses the response unchanged. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Commit:5c7f835
Author:Jakub Rojko
Committer:e2b-bot[bot]

feat(api): report node sandbox limits GitOrigin-RevId: a50b0d751603667d568b6562771704a0e4e52745

Commit:fd9602b
Author:Jakub Rojko
Committer:e2b-bot[bot]

feat(orchestrator): report outstanding work GitOrigin-RevId: b51e1160df674e10f607d7c55c5e6e3d6527d4ce

Commit:c13dc2d
Author:Jakub Rojko
Committer:e2b-bot[bot]

feat(orchestrator): add shutting-down status GitOrigin-RevId: c7bb7eb11e4c2d3abf88d0ced6a65a36d869242c

Commit:972e9d3
Author:Nikita Kalyazin
Committer:e2b-bot[bot]

fix(orch): re-send a sandbox's default user so a live envd upgrade cannot lose it GitOrigin-RevId: 366ea8f9a6e5fb6dd5aa6884da31fdd79a047e3f

Commit:8c8d8d7
Author:gm-e2b
Committer:e2b-bot[bot]

feat(templates): separate build and free disk targets GitOrigin-RevId: 432ab9a88c21055d3524eb4458a32b385b8b2750

Commit:d69867b
Author:Jakub Rojko
Committer:e2b-bot[bot]

feat(orchestrator): let the sandbox proxy dial HTTPS backends GitOrigin-RevId: 7434ff96d0e82bc03557e54dde13f0cf76e3e801

Commit:49eaa7a
Author:Babis Chalios
Committer:e2b-bot[bot]

feat(snapshots): gate in-place checkpoint and fs-only snapshots on FC release >= 0.2.0 Co-authored-by: Claude Fable 5 <noreply@anthropic.com> GitOrigin-RevId: c7b5dff70b1e8a311ff3211a5150e80547102a4d

Commit:f06e9bc
Author:Nikita Kalyazin
Committer:e2b-bot[bot]

feat(envd): freeze the customer's cgroups before a pause, not only our own GitOrigin-RevId: e117443ac949de0810e8eaa9bec9f5d0126d1fa4

Commit:8614ec4
Author:Babis Chalios
Committer:e2b-bot[bot]

feat: filesystem-only resume of memory-inclusive snapshots GitOrigin-RevId: 60617431dda40837e8530508f2ed29ac077c2319

Commit:cbcb146
Author:Jakub Rojko
Committer:e2b-bot[bot]

refactor(api+orchestrator): narrow the running sandbox list GitOrigin-RevId: 24f431d2a2555138e913dbc722f6e7c593e5561e

Commit:988ab87
Author:Tomas Valenta
Committer:e2b-bot[bot]

refactor: remove unused ListCachedBuilds RPC GitOrigin-RevId: 2db50dfba5aeca0d06409f3e4f04045692b6911f

Commit:0baa6ac
Author:Jakub Novak

refactor(api): give orphan reconciliation its own narrow sandbox type GetOrphanCandidates returned a sandbox.Sandbox with 8 of its ~25 fields populated, which invites callers to read a field that is always zero. Introduce sandboxtypes.NodeSandbox, holding exactly what reconciliation and the kill path use, and thread it through Storage.Reconcile, Store.Reconcile and killSandboxOnNode. Sandbox.ToNodeSandbox converts at the regular-removal call sites, so both paths share one kill signature. Rename the Callbacks field to KillOrphanSandbox to match what it does and what it now receives. Also restore proto and doc comments that were dropped from the previous commit, and record why vcpu/ram_mb are on the wire: killSandboxOnNode feeds them to Node.OptimisticRemove.

Commit:a42041a
Author:Jakub Novak

refactor(api): give orphan reconciliation its own narrow sandbox type GetOrphanCandidates returned a sandbox.Sandbox with 8 of its ~25 fields populated, which invites callers to read a field that is always zero. Introduce sandboxtypes.NodeSandbox, holding exactly what reconciliation and the kill path use, and thread it through Storage.Reconcile, Store.Reconcile and killSandboxOnNode. Sandbox.ToNodeSandbox converts at the regular-removal call sites, so both paths share one kill signature. Rename the Callbacks field to KillOrphanSandbox to match what it does and what it now receives. Also restore proto and doc comments that were dropped from the previous commit, and record why vcpu/ram_mb are on the wire: killSandboxOnNode feeds them to Node.OptimisticRemove.

Commit:1115ced
Author:Jakub Novak

refactor(api): narrow the orchestrator sandbox list to orphan detection Redis became the source of truth for sandbox state, and Store.Reconcile now only compares the node's running sandboxes against the store and kills the ones the store does not know about. It never writes anything back. The API was still rebuilding a full Sandbox from the deprecated SandboxConfig in every RunningSandbox -- network config, volume mounts, auto-resume, IAM tokens, alias, disk size, kernel/firecracker/envd versions, auto-pause flags -- and then discarding all of it. Add the fields that decision actually needs directly to RunningSandbox (sandbox_id, team_id, execution_id, vcpu, ram_mb) and mark config deprecated. The orchestrator still sends config so API instances that predate these fields keep working during a rollout; the API reads the scalar fields and falls back to config, and no longer errors when config is absent. That unblocks dropping config and Sandbox.APIStoredConfig in a follow-up, once every API instance is on this build. Rename GetSandboxes to GetOrphanCandidates: the sandboxes it returns are deliberately partial and must not be used as a general lookup. An unparseable team ID now skips that entry instead of failing the call. Such a sandbox has no store key, so it can be neither confirmed nor killed either way, and aborting here failed the whole node sync and eventually marked the node unhealthy. Drop TestGetSandboxes_RestoresAutoPauseFilesystemOnly and TestGetSandboxes_RestoresIamTokens. Both asserted that config survives an API re-sync, which stopped being true when Reconcile became orphan-only.

Commit:c47cf49
Author:Jakub Novak

refactor(api): narrow the orchestrator sandbox list to orphan detection Redis became the source of truth for sandbox state, and Store.Reconcile now only compares the node's running sandboxes against the store and kills the ones the store does not know about. It never writes anything back. The API was still rebuilding a full Sandbox from the deprecated SandboxConfig in every RunningSandbox -- network config, volume mounts, auto-resume, IAM tokens, alias, disk size, kernel/firecracker/envd versions, auto-pause flags -- and then discarding all of it. Add the fields that decision actually needs directly to RunningSandbox (sandbox_id, team_id, execution_id, vcpu, ram_mb) and mark config deprecated. The orchestrator still sends config so API instances that predate these fields keep working during a rollout; the API reads the scalar fields and falls back to config, and no longer errors when config is absent. That unblocks dropping config and Sandbox.APIStoredConfig in a follow-up, once every API instance is on this build. Rename GetSandboxes to GetOrphanCandidates: the sandboxes it returns are deliberately partial and must not be used as a general lookup. An unparseable team ID now skips that entry instead of failing the call. Such a sandbox has no store key, so it can be neither confirmed nor killed either way, and aborting here failed the whole node sync and eventually marked the node unhealthy. Drop TestGetSandboxes_RestoresAutoPauseFilesystemOnly and TestGetSandboxes_RestoresIamTokens. Both asserted that config survives an API re-sync, which stopped being true when Reconcile became orphan-only.

Commit:e5f26b4
Author:Babis Chalios
Committer:Babis Chalios

feat(checkpoint): allow fs-only in-place checkpoints in orchestrator Wire a filesystem_only flag through the Checkpoint gRPC handler. Unlike the earlier destroy-and-resume-fresh design, the in-place checkpoint needs no new mode: fs-only checkpoint is just Pause with WithFilesystemSnapshot() + WithMaintainSandbox() together, which the current Pause already handles — it fsfreeze-quiesces the guest, skips the memory snapshot (mem stays NoDiff, so no memfile is produced or uploaded), seals the rootfs in the background, resumes the same VM in place, and thaws it (bestEffortFsthaw is already gated on envd fsfreeze support, so no 404 teardown on old envd). The persisted template is marked filesystem-only and cold-boots (reboots) on launch; the live sandbox keeps its memory and running processes. Only the handler flag mapping changes (was hardcoded false); prefetch is already omitted for in-place checkpoints. No SnapshotMode enum, no build_upload guards. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Babis Chalios <babis.chalios@e2b.dev>

Commit:f06dd17
Author:Joe Lombrozo
Committer:GitHub

pass placeholder for byoc clusters (#3433)

Commit:8858ff6
Author:Joe Lombrozo
Committer:GitHub

pass placeholder for byoc clusters (#3433)

Commit:8f81c9c
Author:Joe Lombrozo

only return the volume type when creating

Commit:69b7358
Author:Joe Lombrozo
Committer:Joe Lombrozo

create the volume first, which gives us (potentially) the volume type

Commit:f89e4fd
Author:Nikita Kalyazin
Committer:GitHub

feat(orch): live-upgrade envd inside a running sandbox at resume (#3333) # feat(orch): live-upgrade envd inside a running sandbox at resume ## Why envd is the in-VM daemon every sandbox depends on, but its version is baked into the snapshot at build time. A long-lived paused sandbox therefore resumes on whatever envd it was built with, potentially many releases behind — there is no way to ship an envd fix or behavioral change to the existing fleet of snapshots short of rebuilding every template. We want to roll a new envd to running sandboxes the same way we roll Firecracker versions: gated by a LaunchDarkly flag, ramped by cohort, and without disturbing the customer's workload. The hard part is doing it without a visible interruption: the swap has to keep arbitrary customer processes — and their stdio/PTYs, timeouts, filesystem watchers, and recently-observed exit codes — alive across the binary change. ## What Two halves: an in-guest handover mechanism in envd, and an orchestrator-side resume-time trigger driven by a flag. They are independent — the envd half is inert until something calls `POST /upgrade`. ### In-guest handover (envd) A new authenticated `POST /upgrade` endpoint performs a **same-PID `syscall.Exec`** into a new envd binary and re-adopts the running workload, so customer processes never notice the daemon changed underneath them. - **Delivery + trigger in one call.** The new binary is streamed in the request body, written inside the guest (default `/usr/bin/envd.next`), then swapped. This uses the token-authenticated `/upgrade` path rather than the unauthenticated `/files` copy, which a live post-`/init` sandbox rejects. - **The swap.** envd freezes the workload cgroups, serializes the process table to a tmpfs handover blob, relocates the carried fds and `syscall.Exec`s under `ForkLock`. On success it never responds — the process image is replaced. The freeze lock is **held across the whole handover** (freeze → serialize → execve, via `FreezeHold`), not just the freeze sweep, so a concurrent `/init` or `/unfreeze` thaw can't slip in and unfreeze the workload mid-serialization; on `execve` it evaporates with the process image, and on any failure it's released before the thaw. The port scanner is left running: holding `ForkLock` across the fd relocation already serializes it against the scanner's fd/socat creation, so there's no CLOEXEC race to quiesce. - **Guarded fd relocation.** The carried fds are `dup3`'d to fixed high numbers (so the fresh runtime's early startup can't have grabbed them) under `ForkLock`, and each target is checked free first — a collision aborts the upgrade (closing what it already relocated) instead of `dup3` silently clobbering a live fd, leaving the old envd running. - **What the handover carries** (tmpfs `/run/e2b/envd-handover.json`, so it never touches the rootfs diff): per-process pid / tag / cgroup / config, the stdio + PTY fds (dup'd with CLOEXEC cleared so they survive execve), the terminal-event **retention cache** (exit codes of processes that ended just before the swap), the filesystem **watchers**, and each process's **remaining timeout**. - **The incoming side** re-adopts each process from its inherited fds, resumes streaming its output, and re-arms its timeout — recording a fresh deadline so a *further* chained upgrade re-carries the remaining timeout. It restores the retention cache (skipping any PID already re-adopted as live, so a reused PID resolves to the live process, not a stale exit) and re-arms the filesystem watchers **before** thawing the workload, so the workload stays frozen until the watches are live and no event is dropped in the gap. Re-adopted processes are reaped via **pidfd + a pid-specific `wait4`** — never `wait4(-1)`, which would steal `os/exec`'s post-upgrade children — and the reaper retains a terminal event **synchronously, before** closing its event channel, so a late reconnect always finds the exit in the cache rather than racing the retain. - **Failure handling — the sandbox is never left worse off.** On an execve failure the outgoing envd closes the fds it relocated (CLOEXEC-cleared — else they'd leak into the still-running old envd and its future children), thaws the workload it froze, and keeps running the old binary. The incoming resume path always thaws via a top-level `defer` (success, error, or panic) and is recover-wrapped, so a malformed blob can't crash envd into a systemd restart that orphans the workload. Degraded at worst, never hung or dead. - **One shared freeze lock.** The freeze/unfreeze sweep and its serializing lock are factored into a single `cgroups.WorkloadFreezer` shared by the HTTP API (`/freeze`, `/unfreeze`, the `/init` deferred thaw) and the upgrade, so the upgrade's freeze can no longer interleave with the resume thaw and strand the workload frozen. envd is bumped to **0.6.11**, the first upgrade-aware version. ### Resume-time trigger + gate (orchestrator + shared) On resume (`Create` with `snapshot=true`, and the checkpoint-resume path), `maybeUpgradeEnvd` decides against the **live running envd version**, delivers, and confirms — best-effort, recover-wrapped, and bounded so it can never disrupt resume. - **Live version, not built-with.** envd advertises its running version via an `X-Envd-Version` response header on `/init`; the orchestrator captures it off the resume-path `/init` it already makes (no extra round-trip) and keys the decision, gate, `from_version`, and success check on it. The template built-with never changes across a live upgrade, so keying on it would re-trigger the handover on every resume. - **Flag-driven target resolution.** `ResolveEnvdUpgrade` (shared, the resume analog of `ResolveFirecrackerVersion`) reads the `envd-upgrade-target` flag and returns the target binary's local path + version, or `""` when the target is not strictly newer than the live version — so an already-upgraded sandbox (live == target) is a true no-op, skipped with no delivery or handover, and a downgrade is refused. Target binaries live on the node-local read-only `/fc-envd` gcsfuse mount. - **Version gate.** The *running* envd must already speak `/upgrade`, so the trigger skips a live version below `MinEnvdVersionForUpgrade` (0.6.11) and counts it (`gated{reason=old_envd}`). - **Delivery + confirmation.** `Sandbox.CallEnvdUpgrade` streams the binary over the authenticated `/upgrade` body to a fixed guest path; envd reading the body then exec'ing without replying (transport drop) is the expected path. After `WaitForEnvd` (which re-reads the version off the new envd's `/init`), success is confirmed by the running version actually equalling the target — a transport quirk can't mislabel the outcome. Only unambiguous never-reached-envd errors (connection refused / dial) are treated as delivery failures. Flags / knobs: | Flag | Values | Default | Meaning | |------|--------|---------|---------| | `envd-upgrade-target` (LD string) | `off` \| `promoted` \| `<git-sha>` | `off` | `off`: no upgrade. `promoted`: track node-local `HOST_ENVD_PATH`, upgrade when strictly newer than the live version. `<sha>`: pin `/fc-envd/envd.<sha>`. | | `ENVD_UPGRADE_TARGET` (env fallback) | same | `off` | Overrides the flag fallback where there is no LD (dev), mirroring `DEFAULT_FIRECRACKER_VERSION`. | | `MinEnvdVersionForUpgrade` (const) | — | `0.6.11` | Below this the running envd lacks `/upgrade`; trigger skips. | ### Rollout metrics envd has no metrics pipeline, so its side reports via Loki summary logs (`handover_resumed{procs,retained}`, `watchers_rearmed{...}`). The orchestrator side adds: - `orchestrator.envd.upgrade.attempts{result,from_version,to_version}` — `result ∈ {success, delivery_failed, not_ready, version_mismatch, panic}`; `from_version` is the **live** running version. `success` is recorded only when the running version actually flips to `to_version` (not inferred from the transport outcome). `success/total` is the rollout success rate; the common per-resume no-op (flag off / already on target) resolves to `""` and is deliberately **not** counted. - `orchestrator.envd.upgrade.duration{result}` — wall-time of delivery + trigger + `WaitForEnvd`, i.e. the overhead added to the resume. - `orchestrator.envd.upgrade.gated{reason=old_envd}` — targeted resumes skipped by the version gate (watch during a ramp). - an `envd.upgraded` bool label on `orchestrator.sandbox.create.duration`, and an `envd-upgrade` Tempo child span. The post-upgrade readiness re-check re-runs `/init` (to re-read the running version) but is guarded to record the per-start resume KPIs — the envd-init duration histogram + call counter and `StartedAt` — **once per start**, so a successful upgrade doesn't double-count the init or push `StartedAt` (and thus `execution_time`) to a later, wrong timestamp. ```promql # rollout success rate sum(rate(orchestrator_envd_upgrade_attempts_total{result="success"}[5m])) / sum(rate(orchestrator_envd_upgrade_attempts_total[5m])) ``` ## Validation - **Unit tests (root-free, in CI):** a `Connect` after a gap-exit returns the retained Start+End by pid/tag (unknown → NotFound); a live PID is never served a stale cached exit and a reused PID keeps its successor; a stale retention timer can't evict a newer entry; a re-adopted process whose `pidfd_open` fails still emits a terminal event (no orphan); a watcher exported then imported keeps its id, delivers post-handover events, and preserves pending events; `ResumeFromHandover` always thaws on a malformed blob and on no blob; `Upgrade` refuses a target other than the fixed path; the resolver is table-driven over off/unset/promoted/sha × newer/same/older/missing (downgrade refused) without a LaunchDarkly client; `/init` advertises `X-Envd-Version`; the delivery-error classifier flags only never-reached-envd; the three metrics have description+unit entries and construct. - **Dev cluster, end-to-end (exact branch binaries; re-validated 2026-07-23 on the final code):** this branch's envd was rebuilt at 0.6.11 (baked into the template) and 0.6.12 (staged as the `promoted` target), and this branch's orchestrator (`0.2.0-c21705aeb`) deployed to the dev client nodes. Validated against a live workload — a stdout-writing + tmpfs-appending background counter, so a broken fd-carry would SIGPIPE-kill it and line growth proves it kept running: - **upgrade fires + workload survives:** flag → target ⇒ resume auto-upgrades 0.6.11→0.6.12; the running envd becomes `/usr/bin/envd.next --resume-handover` and the counter process **keeps the same PID and keeps writing across the swap** (7→20 lines) — exercising stdio fd inheritance and pidfd reaping end to end. The orchestrator confirms by ground truth: `envd_upgrade_attempts_total{result="success",from_version="0.6.11",to_version="0.6.12"} = 1`. - **flag off ⇒ no upgrade:** plain resume, workload preserved on 0.6.11, no `self-upgrade` in the journal. - **gap-exit retention:** a process that exits during the handover gap has its exit code (42) recovered by a late `Connect` via the carried retention cache. The rollout metrics are emitted on the same path (`attempts{result,from,to}`, `duration`, `gated`, `create.duration{envd_upgraded}`), observed on dev Prometheus with a per-upgrade `duration` of ≈ 345–440 ms. - **Note:** the 2026-07-23 re-run above exercised the **final** binaries (all the review-driven changes: live-version decision, version-confirmed success, the shared freeze lock held across the handover, the freeze/serialize/exec path, and the once-per-start resume-KPI guard on the post-upgrade readiness re-check). The remaining review changes that only manifest on failure/edge paths — reaper retaining before closing its channel, watcher re-arm before thaw, chained-upgrade deadline carry, exec-fail fd cleanup, and the PID-reuse / retain-timer / reaper-orphan / duplicate-forwarder fixes — are covered by the unit tests above; they don't change the success path the e2e validated. ## Key decisions - **Same-PID `syscall.Exec`, not restart.** A fresh process would be reparented by systemd (`Type=simple`), orphaning the workload; keeping the PID means envd stays the workload's parent and the inherited fds/PTYs stay valid. - **pidfd + pid-specific `wait4`, never `wait4(-1)`.** A blanket reaper would steal children that `os/exec` spawns after the upgrade; targeted reaping keeps both the re-adopted and the new processes correct. - **Trigger at resume, gated + best-effort.** Resume is the natural quiescent window and gives free cohort ramping (the resume-site LD context already carries envd-version/team/template). The whole path is recover-wrapped, version-gated, and bounded, so a failed or slow upgrade degrades to `upgraded=false` and the sandbox resumes on the old envd. *Caveat:* the trigger is currently synchronous on the resume-critical path, so on pathologically slow resumes (large cold-fault working sets) it can add up to its timeout budget to the tail; moving it off the critical path is a candidate follow-up. - **Decide and confirm on the live version, read for free off `/init`.** The orchestrator can't cheaply know the running envd version, but envd can report it — so envd advertises `pkg.Version` via an `X-Envd-Version` header on `/init` (which the resume path already calls) and the orchestrator keys the decision, `from_version`, idempotency, and the success check on it. Re-resumes of an upgraded sandbox become true no-ops and success is ground-truth-accurate, with no new endpoint and no extra round-trip. Version comparison still relies on `packages/envd/pkg/version.go` being bumped per behavioral change (repo rule); the resolver refuses any target that isn't strictly newer. - **Fixed exec target.** envd writes/execs the delivered binary only at a fixed path and refuses any other, so an authenticated-but-malformed `/upgrade` can't turn the same-PID exec into arbitrary code execution. - **`getVersion` injected into the shared resolver** so `packages/shared` gains no dependency on the orchestrator's envd build package. - **Handover blob on tmpfs**, additive-fields-only JSON, so it never enters the rootfs diff and an outgoing/incoming envd built at different versions stay compatible. - **Rollout is off by default** (`envd-upgrade-target=off`, env fallback `off`); dev has no LaunchDarkly so the feature is inert there unless `ENVD_UPGRADE_TARGET` is set. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Nikita Kalyazin <nikita.kalyazin@e2b.dev> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Commit:47a7cc8
Author:Nikita Kalyazin
Committer:GitHub

feat(orch): live-upgrade envd inside a running sandbox at resume (#3333) # feat(orch): live-upgrade envd inside a running sandbox at resume ## Why envd is the in-VM daemon every sandbox depends on, but its version is baked into the snapshot at build time. A long-lived paused sandbox therefore resumes on whatever envd it was built with, potentially many releases behind — there is no way to ship an envd fix or behavioral change to the existing fleet of snapshots short of rebuilding every template. We want to roll a new envd to running sandboxes the same way we roll Firecracker versions: gated by a LaunchDarkly flag, ramped by cohort, and without disturbing the customer's workload. The hard part is doing it without a visible interruption: the swap has to keep arbitrary customer processes — and their stdio/PTYs, timeouts, filesystem watchers, and recently-observed exit codes — alive across the binary change. ## What Two halves: an in-guest handover mechanism in envd, and an orchestrator-side resume-time trigger driven by a flag. They are independent — the envd half is inert until something calls `POST /upgrade`. ### In-guest handover (envd) A new authenticated `POST /upgrade` endpoint performs a **same-PID `syscall.Exec`** into a new envd binary and re-adopts the running workload, so customer processes never notice the daemon changed underneath them. - **Delivery + trigger in one call.** The new binary is streamed in the request body, written inside the guest (default `/usr/bin/envd.next`), then swapped. This uses the token-authenticated `/upgrade` path rather than the unauthenticated `/files` copy, which a live post-`/init` sandbox rejects. - **The swap.** envd freezes the workload cgroups, serializes the process table to a tmpfs handover blob, relocates the carried fds and `syscall.Exec`s under `ForkLock`. On success it never responds — the process image is replaced. The freeze lock is **held across the whole handover** (freeze → serialize → execve, via `FreezeHold`), not just the freeze sweep, so a concurrent `/init` or `/unfreeze` thaw can't slip in and unfreeze the workload mid-serialization; on `execve` it evaporates with the process image, and on any failure it's released before the thaw. The port scanner is left running: holding `ForkLock` across the fd relocation already serializes it against the scanner's fd/socat creation, so there's no CLOEXEC race to quiesce. - **Guarded fd relocation.** The carried fds are `dup3`'d to fixed high numbers (so the fresh runtime's early startup can't have grabbed them) under `ForkLock`, and each target is checked free first — a collision aborts the upgrade (closing what it already relocated) instead of `dup3` silently clobbering a live fd, leaving the old envd running. - **What the handover carries** (tmpfs `/run/e2b/envd-handover.json`, so it never touches the rootfs diff): per-process pid / tag / cgroup / config, the stdio + PTY fds (dup'd with CLOEXEC cleared so they survive execve), the terminal-event **retention cache** (exit codes of processes that ended just before the swap), the filesystem **watchers**, and each process's **remaining timeout**. - **The incoming side** re-adopts each process from its inherited fds, resumes streaming its output, and re-arms its timeout — recording a fresh deadline so a *further* chained upgrade re-carries the remaining timeout. It restores the retention cache (skipping any PID already re-adopted as live, so a reused PID resolves to the live process, not a stale exit) and re-arms the filesystem watchers **before** thawing the workload, so the workload stays frozen until the watches are live and no event is dropped in the gap. Re-adopted processes are reaped via **pidfd + a pid-specific `wait4`** — never `wait4(-1)`, which would steal `os/exec`'s post-upgrade children — and the reaper retains a terminal event **synchronously, before** closing its event channel, so a late reconnect always finds the exit in the cache rather than racing the retain. - **Failure handling — the sandbox is never left worse off.** On an execve failure the outgoing envd closes the fds it relocated (CLOEXEC-cleared — else they'd leak into the still-running old envd and its future children), thaws the workload it froze, and keeps running the old binary. The incoming resume path always thaws via a top-level `defer` (success, error, or panic) and is recover-wrapped, so a malformed blob can't crash envd into a systemd restart that orphans the workload. Degraded at worst, never hung or dead. - **One shared freeze lock.** The freeze/unfreeze sweep and its serializing lock are factored into a single `cgroups.WorkloadFreezer` shared by the HTTP API (`/freeze`, `/unfreeze`, the `/init` deferred thaw) and the upgrade, so the upgrade's freeze can no longer interleave with the resume thaw and strand the workload frozen. envd is bumped to **0.6.11**, the first upgrade-aware version. ### Resume-time trigger + gate (orchestrator + shared) On resume (`Create` with `snapshot=true`, and the checkpoint-resume path), `maybeUpgradeEnvd` decides against the **live running envd version**, delivers, and confirms — best-effort, recover-wrapped, and bounded so it can never disrupt resume. - **Live version, not built-with.** envd advertises its running version via an `X-Envd-Version` response header on `/init`; the orchestrator captures it off the resume-path `/init` it already makes (no extra round-trip) and keys the decision, gate, `from_version`, and success check on it. The template built-with never changes across a live upgrade, so keying on it would re-trigger the handover on every resume. - **Flag-driven target resolution.** `ResolveEnvdUpgrade` (shared, the resume analog of `ResolveFirecrackerVersion`) reads the `envd-upgrade-target` flag and returns the target binary's local path + version, or `""` when the target is not strictly newer than the live version — so an already-upgraded sandbox (live == target) is a true no-op, skipped with no delivery or handover, and a downgrade is refused. Target binaries live on the node-local read-only `/fc-envd` gcsfuse mount. - **Version gate.** The *running* envd must already speak `/upgrade`, so the trigger skips a live version below `MinEnvdVersionForUpgrade` (0.6.11) and counts it (`gated{reason=old_envd}`). - **Delivery + confirmation.** `Sandbox.CallEnvdUpgrade` streams the binary over the authenticated `/upgrade` body to a fixed guest path; envd reading the body then exec'ing without replying (transport drop) is the expected path. After `WaitForEnvd` (which re-reads the version off the new envd's `/init`), success is confirmed by the running version actually equalling the target — a transport quirk can't mislabel the outcome. Only unambiguous never-reached-envd errors (connection refused / dial) are treated as delivery failures. Flags / knobs: | Flag | Values | Default | Meaning | |------|--------|---------|---------| | `envd-upgrade-target` (LD string) | `off` \| `promoted` \| `<git-sha>` | `off` | `off`: no upgrade. `promoted`: track node-local `HOST_ENVD_PATH`, upgrade when strictly newer than the live version. `<sha>`: pin `/fc-envd/envd.<sha>`. | | `ENVD_UPGRADE_TARGET` (env fallback) | same | `off` | Overrides the flag fallback where there is no LD (dev), mirroring `DEFAULT_FIRECRACKER_VERSION`. | | `MinEnvdVersionForUpgrade` (const) | — | `0.6.11` | Below this the running envd lacks `/upgrade`; trigger skips. | ### Rollout metrics envd has no metrics pipeline, so its side reports via Loki summary logs (`handover_resumed{procs,retained}`, `watchers_rearmed{...}`). The orchestrator side adds: - `orchestrator.envd.upgrade.attempts{result,from_version,to_version}` — `result ∈ {success, delivery_failed, not_ready, version_mismatch, panic}`; `from_version` is the **live** running version. `success` is recorded only when the running version actually flips to `to_version` (not inferred from the transport outcome). `success/total` is the rollout success rate; the common per-resume no-op (flag off / already on target) resolves to `""` and is deliberately **not** counted. - `orchestrator.envd.upgrade.duration{result}` — wall-time of delivery + trigger + `WaitForEnvd`, i.e. the overhead added to the resume. - `orchestrator.envd.upgrade.gated{reason=old_envd}` — targeted resumes skipped by the version gate (watch during a ramp). - an `envd.upgraded` bool label on `orchestrator.sandbox.create.duration`, and an `envd-upgrade` Tempo child span. The post-upgrade readiness re-check re-runs `/init` (to re-read the running version) but is guarded to record the per-start resume KPIs — the envd-init duration histogram + call counter and `StartedAt` — **once per start**, so a successful upgrade doesn't double-count the init or push `StartedAt` (and thus `execution_time`) to a later, wrong timestamp. ```promql # rollout success rate sum(rate(orchestrator_envd_upgrade_attempts_total{result="success"}[5m])) / sum(rate(orchestrator_envd_upgrade_attempts_total[5m])) ``` ## Validation - **Unit tests (root-free, in CI):** a `Connect` after a gap-exit returns the retained Start+End by pid/tag (unknown → NotFound); a live PID is never served a stale cached exit and a reused PID keeps its successor; a stale retention timer can't evict a newer entry; a re-adopted process whose `pidfd_open` fails still emits a terminal event (no orphan); a watcher exported then imported keeps its id, delivers post-handover events, and preserves pending events; `ResumeFromHandover` always thaws on a malformed blob and on no blob; `Upgrade` refuses a target other than the fixed path; the resolver is table-driven over off/unset/promoted/sha × newer/same/older/missing (downgrade refused) without a LaunchDarkly client; `/init` advertises `X-Envd-Version`; the delivery-error classifier flags only never-reached-envd; the three metrics have description+unit entries and construct. - **Dev cluster, end-to-end (exact branch binaries; re-validated 2026-07-23 on the final code):** this branch's envd was rebuilt at 0.6.11 (baked into the template) and 0.6.12 (staged as the `promoted` target), and this branch's orchestrator (`0.2.0-c21705aeb`) deployed to the dev client nodes. Validated against a live workload — a stdout-writing + tmpfs-appending background counter, so a broken fd-carry would SIGPIPE-kill it and line growth proves it kept running: - **upgrade fires + workload survives:** flag → target ⇒ resume auto-upgrades 0.6.11→0.6.12; the running envd becomes `/usr/bin/envd.next --resume-handover` and the counter process **keeps the same PID and keeps writing across the swap** (7→20 lines) — exercising stdio fd inheritance and pidfd reaping end to end. The orchestrator confirms by ground truth: `envd_upgrade_attempts_total{result="success",from_version="0.6.11",to_version="0.6.12"} = 1`. - **flag off ⇒ no upgrade:** plain resume, workload preserved on 0.6.11, no `self-upgrade` in the journal. - **gap-exit retention:** a process that exits during the handover gap has its exit code (42) recovered by a late `Connect` via the carried retention cache. The rollout metrics are emitted on the same path (`attempts{result,from,to}`, `duration`, `gated`, `create.duration{envd_upgraded}`), observed on dev Prometheus with a per-upgrade `duration` of ≈ 345–440 ms. - **Note:** the 2026-07-23 re-run above exercised the **final** binaries (all the review-driven changes: live-version decision, version-confirmed success, the shared freeze lock held across the handover, the freeze/serialize/exec path, and the once-per-start resume-KPI guard on the post-upgrade readiness re-check). The remaining review changes that only manifest on failure/edge paths — reaper retaining before closing its channel, watcher re-arm before thaw, chained-upgrade deadline carry, exec-fail fd cleanup, and the PID-reuse / retain-timer / reaper-orphan / duplicate-forwarder fixes — are covered by the unit tests above; they don't change the success path the e2e validated. ## Key decisions - **Same-PID `syscall.Exec`, not restart.** A fresh process would be reparented by systemd (`Type=simple`), orphaning the workload; keeping the PID means envd stays the workload's parent and the inherited fds/PTYs stay valid. - **pidfd + pid-specific `wait4`, never `wait4(-1)`.** A blanket reaper would steal children that `os/exec` spawns after the upgrade; targeted reaping keeps both the re-adopted and the new processes correct. - **Trigger at resume, gated + best-effort.** Resume is the natural quiescent window and gives free cohort ramping (the resume-site LD context already carries envd-version/team/template). The whole path is recover-wrapped, version-gated, and bounded, so a failed or slow upgrade degrades to `upgraded=false` and the sandbox resumes on the old envd. *Caveat:* the trigger is currently synchronous on the resume-critical path, so on pathologically slow resumes (large cold-fault working sets) it can add up to its timeout budget to the tail; moving it off the critical path is a candidate follow-up. - **Decide and confirm on the live version, read for free off `/init`.** The orchestrator can't cheaply know the running envd version, but envd can report it — so envd advertises `pkg.Version` via an `X-Envd-Version` header on `/init` (which the resume path already calls) and the orchestrator keys the decision, `from_version`, idempotency, and the success check on it. Re-resumes of an upgraded sandbox become true no-ops and success is ground-truth-accurate, with no new endpoint and no extra round-trip. Version comparison still relies on `packages/envd/pkg/version.go` being bumped per behavioral change (repo rule); the resolver refuses any target that isn't strictly newer. - **Fixed exec target.** envd writes/execs the delivered binary only at a fixed path and refuses any other, so an authenticated-but-malformed `/upgrade` can't turn the same-PID exec into arbitrary code execution. - **`getVersion` injected into the shared resolver** so `packages/shared` gains no dependency on the orchestrator's envd build package. - **Handover blob on tmpfs**, additive-fields-only JSON, so it never enters the rootfs diff and an outgoing/incoming envd built at different versions stay compatible. - **Rollout is off by default** (`envd-upgrade-target=off`, env fallback `off`); dev has no LaunchDarkly so the feature is inert there unless `ENVD_UPGRADE_TARGET` is set. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Nikita Kalyazin <nikita.kalyazin@e2b.dev> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Commit:e4a21ce
Author:Nikita Kalyazin
Committer:Nikita Kalyazin

feat(envd): in-guest live-upgrade handover mechanism Upgrade the envd binary inside a running sandbox by re-exec'ing into the new binary with the same PID and re-adopting the workload, so arbitrary customer processes and their stdio/PTY survive the swap. - POST /upgrade (authenticated): the new binary is streamed in the body, written, and swapped via same-PID syscall.Exec under ForkLock with the workload frozen (via the shared WorkloadFreezer); on success envd never responds (it has exec'd). - Handover blob (tmpfs) is a protobuf HandoverState (spec/upgrade/handover.proto) with an explicit schema + abort-on-newer reader rule: it carries per-process pid/tag/cgroup/config + stdio/PTY fds (CLOEXEC cleared), the terminal-event retention cache, and filesystem watchers, all as native nested messages. The incoming envd re-adopts each process (reaped via pidfd + pid-specific wait4), re-arms watchers/timeouts, restores the retention cache, and thaws the workload. - The workload stays frozen until the post-upgrade /init restores the access token (WorkloadFreezer.FreezeHold), and the pre-init auth gate keeps control endpoints closed in that window. - Retention cache is keyed by pid but also cleared by tag on Start, so a tagged restart within the TTL can't serve a predecessor's stale exit code. - Failure handling: a failed upgrade never leaves the sandbox worse off — on an execve failure the outgoing envd thaws and keeps running the old binary; the incoming resume path always thaws (success/error/panic) and is recover-wrapped. Bump envd to 0.6.12 (the first version that writes the protobuf handover) and MinEnvdVersionForUpgrade with it. Signed-off-by: Nikita Kalyazin <nikita.kalyazin@e2b.dev>

Commit:7aa45b0
Author:Nikita Kalyazin
Committer:Nikita Kalyazin

feat(envd): carry NFS mounts and port-forwards across the handover Two more pieces of re-creatable state were dropped on a live-upgrade because they live only in envd's heap while the kernel objects survive the execve: - The NFS mount ledger (path -> lifecycle) on the API service. Lost, the new envd's post-upgrade /init saw an empty ledger, decided every volume needed (re)mounting, and force-unmounted + remounted a still-live mount — ESTALE for the workload and a possible failed resume. - The port forwarder's socat set. Lost, the new forwarder rescanned and spawned a duplicate socat per already-forwarded port, while the originals were orphaned and leaked as un-reaped zombies. Carry both in the protobuf HandoverState (new MountEntry and ForwardedPort messages): - API.ExportMounts/ImportMounts round-trips the ledger so /init recognizes a matching-lifecycle mount and leaves it in place. - Forwarder.ExportForwards/ImportForwards carries each socat's pid; the new forwarder re-adopts the live ones (skipping any that didn't survive), seeding the ports map so the next scan doesn't spawn a duplicate, and reaps each re-adopted socat via wait4 (there is no surviving *exec.Cmd Wait goroutine). A mutex now guards the ports map since the upgrade goroutine exports it concurrently with the scan loop. These are returned from ResumeFromHandover (rather than applied via a callback like watchers) because their owners — the API service and the forwarder — are constructed after the resume runs. Tests: proto round-trip of mounts + forwards; the mount ledger skipping a remount for an unchanged lifecycle; and the forwarder re-adopting a live socat, skipping a dead one, and exporting only real socats. Signed-off-by: Nikita Kalyazin <nikita.kalyazin@e2b.dev>

Commit:13ddb3d
Author:gm-e2b
Committer:GitHub

feat(api): add sandbox workload identity permission (#3319) feat(api): add sandbox IAM workload token configuration Add an optional iam.tokens map to sandbox create, keyed by a caller-chosen token name, where each definition carries an exact audience and tokenType. A non-empty, validated map enables workload identity for the sandbox; there is no separate enabled flag. Admission requires a non-empty audience and tokenType "JWT-SVID", preserves the audience exactly as received, and rejects an empty token name or any non-null filePath (file-based delivery is unsupported) with a 400 naming the offending field. The definitions are propagated to the orchestrator in SandboxConfig.iam and persisted in the running-sandbox (Redis) and paused-snapshot (Postgres) state, so they survive pause/resume and orchestrator re-sync. A resume or fork restores the configuration from the snapshot, so a fork inherits the parent sandbox's IAM configuration; the resumed/forked execution still obtains a freshly generated execution ID. The API mints, signs, injects, or delivers no credential. Refs: EN-1860 --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

Commit:e903061
Author:Giulio Micheloni

feat(api): add sandbox workload identity permission Add an optional perms.id_token permission to sandbox create. When enabled, the sandbox exposes a workload identity to the orchestrator that is derived from the existing authenticated team, sandbox, execution and template identifiers plus the current deadline. No credential is minted, signed, injected, or delivered here. File-based delivery is unsupported: a non-null perms.id_token.filepath is rejected at admission (absent or null is accepted). The permission is carried through the orchestrator SandboxConfig (new additive SandboxPermissions message) and its persisted/recovery copies, so it survives pause/resume, fork, and API re-sync. A resumed or forked execution gets a freshly generated execution ID and never restores an old identity subject. Network updates leave the permission untouched. Refs: EN-1860

Commit:f8e5ba4
Author:Matt Brockman

Revert "Fence orchestrator drains by status epoch" This reverts commit 2151595b73efcc787ba42659e978ab440720b7ac.

Commit:2151595
Author:Matt Brockman

Fence orchestrator drains by status epoch Use a dedicated Healthy-to-Draining RPC so rollout callers cannot race a process restart between observation and drain. Edge integration, module publication, and the dependent Belt pin must still land in that order.

Commit:5ae1618
Author:Matt Brockman

Gate orchestrator rollout promotion by status epoch Replacement processes can start non-admitting and are promoted only through an exact identity and status-epoch transition. Legacy status overrides cannot bypass the Standby gate.

Commit:14501a4
Author:Matt Brockman
Committer:Matt Brockman

Fence orchestrator status changes to exact processes Use a new RPC so older servers fail closed instead of ignoring identity fields. Both API and Edge callers must bind mutations to the discovered node and service instance before changing runtime admission.

Commit:65e7f99
Author:Matt Brockman
Committer:Matt Brockman

Expose exact orchestrator drain readiness Bind replacement readiness to a process-authored drain fence and every startup, running, stopping, or failed-cleanup lifecycle. Admission closes before quiescence checks, duplicate starts are reserved before resource allocation, and force shutdown remains bounded.

Commit:646cdab
Author:Babis Chalios

feat(checkpoint): allow fs-only in-place checkpoints in orchestrator Wire a filesystem_only flag through the Checkpoint gRPC handler. Unlike the earlier destroy-and-resume-fresh design, the in-place checkpoint needs no new mode: fs-only checkpoint is just Pause with WithFilesystemSnapshot() + WithMaintainSandbox() together, which the current Pause already handles — it fsfreeze-quiesces the guest, skips the memory snapshot (mem stays NoDiff, so no memfile is produced or uploaded), seals the rootfs in the background, resumes the same VM in place, and thaws it (bestEffortFsthaw is already gated on envd fsfreeze support, so no 404 teardown on old envd). The persisted template is marked filesystem-only and cold-boots (reboots) on launch; the live sandbox keeps its memory and running processes. Only the handler flag mapping changes (was hardcoded false); prefetch is already omitted for in-place checkpoints. No SnapshotMode enum, no build_upload guards. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Babis Chalios <babis.chalios@e2b.dev>

Commit:2aad55c
Author:Matt Brockman
Committer:Matt Brockman

Fence orchestrator status changes to exact processes Use a new RPC so older servers fail closed instead of ignoring identity fields. Both API and Edge callers must bind mutations to the discovered node and service instance before changing runtime admission.

Commit:a9989fc
Author:Matt Brockman
Committer:Matt Brockman

Expose exact orchestrator drain readiness Bind replacement readiness to a process-authored drain fence and every startup, running, stopping, or failed-cleanup lifecycle. Admission closes before quiescence checks, duplicate starts are reserved before resource allocation, and force shutdown remains bounded.

Commit:ac92a85
Author:Rauf Guliyev

feat: add snapshot peer registration and policy fields to proto Adds SnapshotSourcePolicy (Create request) and SnapshotPeerRegistration (Pause response) plus their enums. Pure API-contract addition, no callers yet — orchestrator.pb.go regenerated via protoc, verified byte-identical to a fresh codegen run against this proto.

Commit:5e2664d
Author:Babis Chalios
Committer:Babis Chalios

feat(checkpoint): allow fs-only checkpoints in orchestrator Orchestrator now allows creating filesystem only checkpoints. The orchestrator server now accepts a filesystem_only flag in the checkpoint request to control whether it will include a snapshot of the memory. Checkpoint is a bit different than pause, because we need to handle the existing sandbox. The way we do this now is creating a full snapshot, destroying the previous sandbox and starting a fresh one from the freshly created sandbox. Ideally, we should be reusing the existing sandbox and not starting a new one, but this is a bigger change. For the time being, we don't change the way things are done. We create the full snapshot always. If the checkpoint is filesystem only, we mark the Snapshot object as such and use the memory part only locally (we don't upload it). Signed-off-by: Babis Chalios <babis.chalios@e2b.dev>

Commit:c1406f8
Author:Giulio Micheloni
Committer:Giulio Micheloni

[EN-1038] feat: add LD-gated template disk enforcement Read template disk defaults and maximums from team_limits_v2, and use the new default when registering builds. Propagate the build-start maximum through the internal TemplateCreate RPC for both build endpoints. At build completion, compare the exact final rootfs size with the maximum, emit disk-limit telemetry, and reject oversized builds when build-enforce-max-disk-size is enabled. Keep missing or invalid maximums fail-open and document the updated build flow.

Commit:c90d6b2
Author:Jakub Novak

feat(proxy): support HTTPS sandbox backends Allow sandbox network config to select HTTPS per port. Accept self-signed backend certificates on trusted orchestrator-to-sandbox hop.

Commit:51e7be0
Author:Mish Ushakov

feat: derive filesystem-only snapshot from memory checkpoint Rework the memory:false snapshot flow so the source sandbox keeps its memory: the checkpoint takes a full memory snapshot (build A, the sandbox's own snapshot row) after a guest sync quiesce, derives a filesystem-only sibling build (B) whose rootfs header references A's data without duplicating it, memory-resumes the sandbox from A as a normal checkpoint, and returns B as the snapshot template. Sandboxes created from B cold-boot from the captured filesystem. The sibling upload is header+metadata only and runs after A's upload (its ancestor wait requires A's finalized header). Quiesce uses guest sync rather than fsfreeze: RAM captured with a frozen filesystem would memory-resume frozen and wedge all writes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Commit:4ef8048
Author:Mish Ushakov
Committer:Mish Ushakov

feat(envd): kill tree on request timeout when started with child_processes Address PR review (timeout killed leader only): add a child_processes field to StartRequest so a command started with it also tears down its whole process group when a request timeout fires, keeping the timeout path consistent with an opted-in SendSignal. Non-opted-in commands and PTY processes keep the original leader-only behavior. Regenerated the process protos and added a timeout regression test. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Commit:bc47f29
Author:Mish Ushakov
Committer:GitHub

Merge branch 'main' into mishushakov/envd-kill-process-tree

Commit:9269abd
Author:Mish Ushakov

feat(envd): make process-tree kill opt-in via child_processes flag Add a child_processes field to SendSignalRequest. When set, the signal is delivered to the command's whole process group (leader + children); otherwise only the leader is signaled, preserving the original behavior. Non-PTY commands still always start in their own process group so the opt-in kill can reach the tree, but the timeout-driven kill is left as leader-only. Regenerated the process protos and added a test covering the leader-only default. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Commit:282de08
Author:Mish Ushakov

feat: filesystem-only variant of sandbox snapshot endpoint (memory: false) Add a memory flag to POST /sandboxes/{sandboxID}/snapshots. When false, the snapshot template persists only the rootfs: sandboxes created from it cold-boot, and the source sandbox is rebooted in place (checkpoint cannot memory-resume without a memory snapshot) while keeping its identity and end time. Wired through SandboxCheckpointRequest.filesystem_only; the orchestrator branches to RebootSandbox and skips memory-prefetch work. Also tag linux-only files in cmd/create-build, cmd/resume-build, cmd/inspect-build, and pkg/server/prefetch_harvest.go with //go:build linux so make lint passes on macOS. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Commit:f76b2cb
Author:Jakub Novák
Committer:GitHub

feat: per-team events TTL limit (tier + addons) (#3181) Allow customization for teams to extend ttl for events

Commit:7735bae
Author:Jakub Novák
Committer:GitHub

feat(orchestrator): report hugepage metrics to API (#3182) Expose HugePages_Total/Used/Rsvd and hugepage size from /proc/meminfo via ServiceInfo gRPC and NodeMetrics on the admin /nodes endpoints. Page counts, not bytes, to match kernel accounting.

Commit:4c0f658
Author:Jakub Novak

chore: clean up

Commit:ebfba1c
Author:Jakub Novak

feat: per-team events TTL limit (tier + addons) Add events_ttl_days to tiers with extra_events_ttl_days addon column, summed into the team_limits view. The limit flows through team limits into SandboxConfig (proto field 26) so the orchestrator stamps it on every published sandbox event; the events store TTL was switched to read it in a previous change. Dashboard exposes the limit and uses it for the sandbox record retention-expired check instead of a hardcoded 7 days.

Commit:0cf3173
Author:ValentaTomas

refactor: keep layer-size categories whole - DB carries only logical sizes Move the rootfs mapped/diff sizes out of the gRPC/DB path so each size category lives in one place: logical sizes go via gRPC into env_builds (memfile logical; rootfs logical is already total_disk_size_mb), while mapped and diff sizes for both artifacts are written to the data objects' metadata (see the memfile-layer -metadata PR). Drops the rootfs_*_size_bytes columns and the now-unused Mapping.MappedBytes helper.

Commit:a131641
Author:ValentaTomas

feat: persist synchronously-available layer sizes to env_builds via gRPC Replace the object-metadata approach with returning per-artifact layer sizes over gRPC (pause/checkpoint responses and TemplateBuildMetadata) and persisting them on the env_builds row. Only sizes available without blocking on the async memfile dedup header are included: rootfs mapped/diff and memfile logical (rootfs logical is already stored as total_disk_size_mb). The async memfile mapped/diff sizes are handled separately.

Commit:ef856b2
Author:Tomas Valenta

storage-api: drop stale "block size" from ingest RPC comment No block_size field exists on the proto; align the comment.

Commit:5320d56
Author:Tomas Valenta

orchestrator: index builds into storage-api after headers are ready Best-effort, feature-flagged (storage-api-ingest-build); no-op when unconfigured.

Commit:a7c8320
Author:Weilu Jia
Committer:Weilu Jia

feat(orchestrator): force-stop sandboxes and bound shutdown drain Escalate shutdown from graceful drain to a bounded force stop. Add SHUTDOWN_DRAIN_TIMEOUT to cap the graceful phase; on expiry (or when force-stop is requested) the orchestrator force-closes live sandboxes and cancels in-flight template builds, detaching cleanup from the canceled shutdown context while preserving its deadline. Introduce the drain-gate state both servers shut down through: the sandbox server's start gate and the template server's build gate, with StartDraining replacing the old done channel and the drain paths waiting for admitted in-flight starts to finish before snapshotting the live set. New work is still admitted here; rejecting it is layered in separately. Plumb a force_stop flag from the node status API through gRPC so an operator can drain-and-force a node, make ServiceInfo status transitions atomic so a drain override cannot race back to healthy and drain cannot be reversed, and run drains as managed services with deterministic exit reporting. Cleanup rollbacks now preserve deadlines via RunRollback.

Commit:ecd13b0
Author:Babis Chalios
Committer:GitHub

Filesystem-only snapshots for auto-pause (#3055) Summary Adds a per-sandbox autoPauseMemory create option (default true). When false, a sandbox that auto-pauses on timeout takes a filesystem-only snapshot (no memory) instead of a full memory snapshot, so it cold-boots from disk on resume. Builds on the filesystem-only snapshot machinery from #3027 — this PR just lets the timeout auto-pause path opt into it, where previously only an explicit pause(memory=false) could. How it works The FilesystemOnly flag was already plumbed end-to-end through the pause path (RemoveOpts → pauseSandbox → snapshot → Pause RPC); the only missing piece was a per-sandbox policy for the evictor to act on. 1. Create — autoPauseMemory=false → Sandbox.AutoPauseFilesystemOnly=true (runtime state, persisted to Redis). 2. Evict — when the evictor auto-pauses on timeout, it threads opts.FilesystemOnly = sbx.AutoPauseFilesystemOnly. 3. Persist — the policy is stamped onto the snapshot's PausedSandboxConfig, and... 4. Restore — re-read on resume (covers explicit /resume, /connect, and auto-resume — they share buildResumeSandboxData), so the policy survives a pause/resume cycle. 5. Re-sync — also carried through SandboxConfig.auto_pause_filesystem_only in the orchestrator proto, mirroring auto_pause, so an API restart that rebuilds from the orchestrator's sandbox list doesn't lose it. The orchestrator is unchanged — the existing IsFilesystemOnly() dispatch already routes any resume of an fs-only snapshot to RebootSandbox. Constraints (rejected at create, 400) - autoPauseMemory=false requires autoPause=true (it only governs the timeout auto-pause; otherwise a no-op). - autoPauseMemory=false is incompatible with autoResume — a filesystem-only snapshot can't be auto-resumed by arbitrary traffic (it'd cold-boot, losing processes/connections) and must be resumed explicitly. Design notes - Two distinct fields. AutoPauseFilesystemOnly (the policy for the next auto-pause) is kept separate from FilesystemOnly (the kind of this snapshot), so an explicit pause(memory=true) on such a sandbox produces a memory snapshot yet still restores the fs-only policy on the next cycle. - Resume inherits the policy. Unlike auto_pause (overridable via the resume body, now deprecated), the auto-pause kind is always inherited from the snapshot — changing it means creating a new sandbox. Deliberate, to avoid extending a deprecated path. - Backward-compatible: default true, omitempty on both the JSON config and the proto, so existing sandboxes/snapshots decode to memory auto-pause. Testing - Unit — evictor wiring (kill/pause × snapshot kind); PausedSandboxConfig serialization round-trip + legacy-row default; proto re-sync round-trip via GetSandboxes. - Integration — create → timeout auto-pause → resume cold-boots (rootfs marker survives, fresh boot id, correct default user), and the policy persists across a second pause/resume cycle; both create-time guards return 400. Usage ``` # auto-pause this sandbox as a filesystem-only snapshot on timeout POST /sandboxes { "templateID": "...", "autoPause": true, "autoPauseMemory": false } ``` --------- Signed-off-by: Babis Chalios <babis.chalios@e2b.dev> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

Commit:ce556e8
Author:Weilu Jia
Committer:Weilu Jia

fix(orchestrator): honor drain status overrides Make ServiceInfo status transitions atomic via TransitionStatus so a drain override cannot race a transition back to healthy, reject drain reversal, and guard ForceStop with sync.Once. Plumb a forceStop flag from the API through the node status change so draining can force stop sandboxes and template builds.

Commit:7b45c54
Author:Babis Chalios
Committer:Babis Chalios

feat(api): support filesystem-only snapshots for auto-pause Timeout auto-pause always took a full memory snapshot. Add a per-sandbox autoPauseMemory option (default true) so a timeout auto-pause can instead take a filesystem-only snapshot (no memory; cold-boots on resume), reusing the existing FilesystemOnly pause path. The evictor reads the policy off the sandbox and threads FilesystemOnly into the pause. The policy is persisted on the snapshot (PausedSandboxConfig) and carried through the orchestrator SandboxConfig (auto_pause_filesystem_only) so it survives a pause/resume cycle and an API re-sync from the orchestrator's sandbox list — mirroring how auto_pause itself is handled. Reject autoPauseMemory=false combined with autoResume: a filesystem-only snapshot cannot be auto-resumed by traffic and must be resumed explicitly. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Babis Chalios <babis.chalios@e2b.dev>

Commit:f79be77
Author:Jakub Novák
Committer:GitHub

feat(orchestrator): track and report last status change timestamp (#2980) Add service_status_changed_at to ServiceInfoResponse so the API can see when the orchestrator's status last changed. The timestamp is updated on actual status transitions and initialized to the service startup time. --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

Commit:c09d96a
Author:Babis Chalios
Committer:GitHub

Filesystem-only snapshots: control plane wiring (#3027) ## Summary Adds filesystem-only snapshots: pause can persist only the rootfs (no memory snapshot), and resuming such a snapshot cold-boots (reboots) the guest from disk instead of restoring memory. Useful when in-memory state isn't worth persisting — cheaper, faster pauses. ## What's included ### Resume side - Server.Create dispatches reboot-vs-resume from the snapshot's own metadata.json (IsFilesystemOnly()); a safety gate in reboot.go refuses to cold-boot a memory snapshot. ### Control plane - filesystem_only on the pause gRPC → threaded through the API pause path → persisted in PausedSandboxConfig. - Public trigger: optional memory: bool (default true) on POST /sandboxes/{id}/pause; memory:false requests a filesystem-only snapshot. - Gating (a disk-only resume cold-boots, losing in-memory state, so implicit resumes are refused): - auto-resume → FailedPrecondition (request must resume explicitly) - /connect → 409 - explicit /resume is allowed and performs the reboot. Existing pause callers are unaffected: no body / memory unset = full memory snapshot. ## Tests Five integration tests, one per behavior: - <s>/connect refused (409) on a filesystem-only snapshot</s> - proxy traffic does not auto-resume a filesystem-only snapshot - explicit resume cold-boots: rootfs marker survives, boot_id changes - secure sandbox keeps working envd auth across the reboot (MMDS token path) - build-installed CA is in the live trust store after the reboot (boot-time half of #3025's cert validation) Validation The integration tests above are the validation gates; they require the Firecracker integration runner to actually execute (compile/vet/lint verified locally). They must pass in CI before merge. --------- Signed-off-by: Babis Chalios <babis.chalios@e2b.dev> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: ValentaTomas <49156497+ValentaTomas@users.noreply.github.com>

Commit:79fa9d7
Author:Babis Chalios
Committer:Babis Chalios

feat(orchestrator): add filesystem_only to the pause RPC Adds SandboxPauseRequest.filesystem_only so a caller can request a filesystem-only pause over gRPC. Server.Pause threads it to sandbox.WithFilesystemSnapshot, which (via PR-A1) stamps the snapshot's metadata so resume cold-boots from the rootfs. Default false = full memory snapshot, so existing pause callers are unaffected; Checkpoint always passes false (filesystem-only checkpoint is not supported yet). The public memory:false API surface and DB persistence land next in this PR. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Babis Chalios <babis.chalios@e2b.dev>

Commit:ba8604e
Author:Tomas Valenta
Committer:GitHub

feat(storage): stamp provenance custom metadata on uploaded objects (incl. headers) (#3033) Attach immutable `team_id`, `template_id`, and `origin` (`pause`/`template_build`/`snapshot_template`) as custom object metadata on every uploaded object, including header objects (which previously carried no custom metadata at all). This lets the storage index attribute ownership/usage and classify root kinds from the object it already downloads, without the infra DB. Lineage stays in the header (`base_build_id` + reference edges); mutable, DB-owned state is deliberately excluded. --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

Commit:d5c56aa
Author:Weilu Jia
Committer:Weilu Jia

fix(orchestrator): honor drain status overrides Make ServiceInfo status transitions atomic via TransitionStatus so a drain override cannot race a concurrent transition back to healthy, reject drain reversal (including via an Unhealthy detour), guard ForceStop with sync.Once while preserving drain-to-force-stop escalation, and plumb a forceStop flag from the API through the node status change so draining can force stop sandboxes and template builds.

Commit:9799dd0
Author:Mish Ushakov
Committer:GitHub

feat(envd): allow opting into watching network mounts (#2982) ## Summary - Adds an `allow_network_mounts` field to `WatchDirRequest` and `CreateWatcherRequest` so clients can explicitly opt into watching paths on network filesystem mounts (NFS, CIFS, SMB, FUSE), which are rejected by default since #1810. - The streaming (`WatchDir`) and non-streaming (`CreateWatcher`) handlers skip the network-mount rejection when the flag is set; default behavior is unchanged. - Regenerates the filesystem proto code in envd and shared, and bumps envd to 0.6.4. ## Test Plan - Added `TestCreateWatcherOnNetworkMount` (bindfs FUSE mount, Linux-only) verifying the watcher is rejected without the flag and created with it; existing filesystem watch tests pass. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

Commit:a0660de
Author:Jakub Novak
Committer:Jakub Novak

feat(orchestrator): track and report last status change timestamp Add service_status_changed_at to ServiceInfoResponse so the API can see when the orchestrator's status last changed. The timestamp is updated on actual status transitions and initialized to the service startup time.

Commit:1fc3820
Author:Petr Vaněk
Committer:GitHub

feat(api): SOCKS5 egress proxy on sandbox network config (BYOP) (#2642) Sandboxes have two related egress problems we cannot solve from our side: operators want a predictable egress identity (their public IP, not ours), and they want sandboxes to reach hosts that are reachable from their own proxy server — corporate networks, VPN endpoints, internal LAN. This PR adds Bring-Your-Own-Proxy: an optional `egressProxy: {address, username, password}` on the sandbox network config tunnels all sandbox TCP egress through a user-supplied SOCKS5 server, with our allow/deny rules still applied before the tunnel. Scope here is the API + orchestrator plumbing: OpenAPI, DB types, proto, the API create/get handlers (password omitted from GET), the `BYOPProxyEnabledFlag` LD gate, and a validator that rejects a SOCKS5 endpoint pointing at our own infra at create time and re-checks on every dial (DNS-rebind guard). For BYOP-enabled sandboxes only, the sandbox-netns kernel firewall is narrowed on demand so TCP destined for the user's internal networks reaches the userspace egress proxy instead of being dropped — without this the user's LAN destinations would never make it past the kernel. Non-BYOP sandboxes keep the existing hard kernel drop. --------- Co-authored-by: Jakub Novák <jakub@e2b.dev>