Proto commits in e2b-dev/infra

These commits are when the Protocol Buffers files have changed: (only the last 100 relevant commits are shown)

2026-08-21

feat(envd): freeze the customer's cgroups before a pause, not only our own GitOrigin-RevId: e117443ac949de0810e8eaa9bec9f5d0126d1fa4

2026-08-20

feat: filesystem-only resume of memory-inclusive snapshots GitOrigin-RevId: 60617431dda40837e8530508f2ed29ac077c2319

2026-08-14

refactor(api+orchestrator): narrow the running sandbox list GitOrigin-RevId: 24f431d2a2555138e913dbc722f6e7c593e5561e

2026-08-07

refactor: remove unused ListCachedBuilds RPC GitOrigin-RevId: 2db50dfba5aeca0d06409f3e4f04045692b6911f

2026-08-02

refactor(api): give orphan reconciliation its own narrow sandbox type GetOrphanCandidates returned a sandbox.Sandbox with 8 of its ~25 fields populated, which invites callers to read a field that is always zero. Introduce sandboxtypes.NodeSandbox, holding exactly what reconciliation and the kill path use, and thread it through Storage.Reconcile, Store.Reconcile and killSandboxOnNode. Sandbox.ToNodeSandbox converts at the regular-removal call sites, so both paths share one kill signature. Rename the Callbacks field to KillOrphanSandbox to match what it does and what it now receives. Also restore proto and doc comments that were dropped from the previous commit, and record why vcpu/ram_mb are on the wire: killSandboxOnNode feeds them to Node.OptimisticRemove.

refactor(api): give orphan reconciliation its own narrow sandbox type GetOrphanCandidates returned a sandbox.Sandbox with 8 of its ~25 fields populated, which invites callers to read a field that is always zero. Introduce sandboxtypes.NodeSandbox, holding exactly what reconciliation and the kill path use, and thread it through Storage.Reconcile, Store.Reconcile and killSandboxOnNode. Sandbox.ToNodeSandbox converts at the regular-removal call sites, so both paths share one kill signature. Rename the Callbacks field to KillOrphanSandbox to match what it does and what it now receives. Also restore proto and doc comments that were dropped from the previous commit, and record why vcpu/ram_mb are on the wire: killSandboxOnNode feeds them to Node.OptimisticRemove.

refactor(api): narrow the orchestrator sandbox list to orphan detection Redis became the source of truth for sandbox state, and Store.Reconcile now only compares the node's running sandboxes against the store and kills the ones the store does not know about. It never writes anything back. The API was still rebuilding a full Sandbox from the deprecated SandboxConfig in every RunningSandbox -- network config, volume mounts, auto-resume, IAM tokens, alias, disk size, kernel/firecracker/envd versions, auto-pause flags -- and then discarding all of it. Add the fields that decision actually needs directly to RunningSandbox (sandbox_id, team_id, execution_id, vcpu, ram_mb) and mark config deprecated. The orchestrator still sends config so API instances that predate these fields keep working during a rollout; the API reads the scalar fields and falls back to config, and no longer errors when config is absent. That unblocks dropping config and Sandbox.APIStoredConfig in a follow-up, once every API instance is on this build. Rename GetSandboxes to GetOrphanCandidates: the sandboxes it returns are deliberately partial and must not be used as a general lookup. An unparseable team ID now skips that entry instead of failing the call. Such a sandbox has no store key, so it can be neither confirmed nor killed either way, and aborting here failed the whole node sync and eventually marked the node unhealthy. Drop TestGetSandboxes_RestoresAutoPauseFilesystemOnly and TestGetSandboxes_RestoresIamTokens. Both asserted that config survives an API re-sync, which stopped being true when Reconcile became orphan-only.

refactor(api): narrow the orchestrator sandbox list to orphan detection Redis became the source of truth for sandbox state, and Store.Reconcile now only compares the node's running sandboxes against the store and kills the ones the store does not know about. It never writes anything back. The API was still rebuilding a full Sandbox from the deprecated SandboxConfig in every RunningSandbox -- network config, volume mounts, auto-resume, IAM tokens, alias, disk size, kernel/firecracker/envd versions, auto-pause flags -- and then discarding all of it. Add the fields that decision actually needs directly to RunningSandbox (sandbox_id, team_id, execution_id, vcpu, ram_mb) and mark config deprecated. The orchestrator still sends config so API instances that predate these fields keep working during a rollout; the API reads the scalar fields and falls back to config, and no longer errors when config is absent. That unblocks dropping config and Sandbox.APIStoredConfig in a follow-up, once every API instance is on this build. Rename GetSandboxes to GetOrphanCandidates: the sandboxes it returns are deliberately partial and must not be used as a general lookup. An unparseable team ID now skips that entry instead of failing the call. Such a sandbox has no store key, so it can be neither confirmed nor killed either way, and aborting here failed the whole node sync and eventually marked the node unhealthy. Drop TestGetSandboxes_RestoresAutoPauseFilesystemOnly and TestGetSandboxes_RestoresIamTokens. Both asserted that config survives an API re-sync, which stopped being true when Reconcile became orphan-only.

2026-07-31

feat(checkpoint): allow fs-only in-place checkpoints in orchestrator Wire a filesystem_only flag through the Checkpoint gRPC handler. Unlike the earlier destroy-and-resume-fresh design, the in-place checkpoint needs no new mode: fs-only checkpoint is just Pause with WithFilesystemSnapshot() + WithMaintainSandbox() together, which the current Pause already handles — it fsfreeze-quiesces the guest, skips the memory snapshot (mem stays NoDiff, so no memfile is produced or uploaded), seals the rootfs in the background, resumes the same VM in place, and thaws it (bestEffortFsthaw is already gated on envd fsfreeze support, so no 404 teardown on old envd). The persisted template is marked filesystem-only and cold-boots (reboots) on launch; the live sandbox keeps its memory and running processes. Only the handler flag mapping changes (was hardcoded false); prefetch is already omitted for in-place checkpoints. No SnapshotMode enum, no build_upload guards. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Babis Chalios <babis.chalios@e2b.dev>

2026-07-30

pass placeholder for byoc clusters (#3433)

pass placeholder for byoc clusters (#3433)

only return the volume type when creating

2026-07-29

create the volume first, which gives us (potentially) the volume type

2026-07-28

feat(orch): live-upgrade envd inside a running sandbox at resume (#3333) # feat(orch): live-upgrade envd inside a running sandbox at resume ## Why envd is the in-VM daemon every sandbox depends on, but its version is baked into the snapshot at build time. A long-lived paused sandbox therefore resumes on whatever envd it was built with, potentially many releases behind — there is no way to ship an envd fix or behavioral change to the existing fleet of snapshots short of rebuilding every template. We want to roll a new envd to running sandboxes the same way we roll Firecracker versions: gated by a LaunchDarkly flag, ramped by cohort, and without disturbing the customer's workload. The hard part is doing it without a visible interruption: the swap has to keep arbitrary customer processes — and their stdio/PTYs, timeouts, filesystem watchers, and recently-observed exit codes — alive across the binary change. ## What Two halves: an in-guest handover mechanism in envd, and an orchestrator-side resume-time trigger driven by a flag. They are independent — the envd half is inert until something calls `POST /upgrade`. ### In-guest handover (envd) A new authenticated `POST /upgrade` endpoint performs a **same-PID `syscall.Exec`** into a new envd binary and re-adopts the running workload, so customer processes never notice the daemon changed underneath them. - **Delivery + trigger in one call.** The new binary is streamed in the request body, written inside the guest (default `/usr/bin/envd.next`), then swapped. This uses the token-authenticated `/upgrade` path rather than the unauthenticated `/files` copy, which a live post-`/init` sandbox rejects. - **The swap.** envd freezes the workload cgroups, serializes the process table to a tmpfs handover blob, relocates the carried fds and `syscall.Exec`s under `ForkLock`. On success it never responds — the process image is replaced. The freeze lock is **held across the whole handover** (freeze → serialize → execve, via `FreezeHold`), not just the freeze sweep, so a concurrent `/init` or `/unfreeze` thaw can't slip in and unfreeze the workload mid-serialization; on `execve` it evaporates with the process image, and on any failure it's released before the thaw. The port scanner is left running: holding `ForkLock` across the fd relocation already serializes it against the scanner's fd/socat creation, so there's no CLOEXEC race to quiesce. - **Guarded fd relocation.** The carried fds are `dup3`'d to fixed high numbers (so the fresh runtime's early startup can't have grabbed them) under `ForkLock`, and each target is checked free first — a collision aborts the upgrade (closing what it already relocated) instead of `dup3` silently clobbering a live fd, leaving the old envd running. - **What the handover carries** (tmpfs `/run/e2b/envd-handover.json`, so it never touches the rootfs diff): per-process pid / tag / cgroup / config, the stdio + PTY fds (dup'd with CLOEXEC cleared so they survive execve), the terminal-event **retention cache** (exit codes of processes that ended just before the swap), the filesystem **watchers**, and each process's **remaining timeout**. - **The incoming side** re-adopts each process from its inherited fds, resumes streaming its output, and re-arms its timeout — recording a fresh deadline so a *further* chained upgrade re-carries the remaining timeout. It restores the retention cache (skipping any PID already re-adopted as live, so a reused PID resolves to the live process, not a stale exit) and re-arms the filesystem watchers **before** thawing the workload, so the workload stays frozen until the watches are live and no event is dropped in the gap. Re-adopted processes are reaped via **pidfd + a pid-specific `wait4`** — never `wait4(-1)`, which would steal `os/exec`'s post-upgrade children — and the reaper retains a terminal event **synchronously, before** closing its event channel, so a late reconnect always finds the exit in the cache rather than racing the retain. - **Failure handling — the sandbox is never left worse off.** On an execve failure the outgoing envd closes the fds it relocated (CLOEXEC-cleared — else they'd leak into the still-running old envd and its future children), thaws the workload it froze, and keeps running the old binary. The incoming resume path always thaws via a top-level `defer` (success, error, or panic) and is recover-wrapped, so a malformed blob can't crash envd into a systemd restart that orphans the workload. Degraded at worst, never hung or dead. - **One shared freeze lock.** The freeze/unfreeze sweep and its serializing lock are factored into a single `cgroups.WorkloadFreezer` shared by the HTTP API (`/freeze`, `/unfreeze`, the `/init` deferred thaw) and the upgrade, so the upgrade's freeze can no longer interleave with the resume thaw and strand the workload frozen. envd is bumped to **0.6.11**, the first upgrade-aware version. ### Resume-time trigger + gate (orchestrator + shared) On resume (`Create` with `snapshot=true`, and the checkpoint-resume path), `maybeUpgradeEnvd` decides against the **live running envd version**, delivers, and confirms — best-effort, recover-wrapped, and bounded so it can never disrupt resume. - **Live version, not built-with.** envd advertises its running version via an `X-Envd-Version` response header on `/init`; the orchestrator captures it off the resume-path `/init` it already makes (no extra round-trip) and keys the decision, gate, `from_version`, and success check on it. The template built-with never changes across a live upgrade, so keying on it would re-trigger the handover on every resume. - **Flag-driven target resolution.** `ResolveEnvdUpgrade` (shared, the resume analog of `ResolveFirecrackerVersion`) reads the `envd-upgrade-target` flag and returns the target binary's local path + version, or `""` when the target is not strictly newer than the live version — so an already-upgraded sandbox (live == target) is a true no-op, skipped with no delivery or handover, and a downgrade is refused. Target binaries live on the node-local read-only `/fc-envd` gcsfuse mount. - **Version gate.** The *running* envd must already speak `/upgrade`, so the trigger skips a live version below `MinEnvdVersionForUpgrade` (0.6.11) and counts it (`gated{reason=old_envd}`). - **Delivery + confirmation.** `Sandbox.CallEnvdUpgrade` streams the binary over the authenticated `/upgrade` body to a fixed guest path; envd reading the body then exec'ing without replying (transport drop) is the expected path. After `WaitForEnvd` (which re-reads the version off the new envd's `/init`), success is confirmed by the running version actually equalling the target — a transport quirk can't mislabel the outcome. Only unambiguous never-reached-envd errors (connection refused / dial) are treated as delivery failures. Flags / knobs: | Flag | Values | Default | Meaning | |------|--------|---------|---------| | `envd-upgrade-target` (LD string) | `off` \| `promoted` \| `<git-sha>` | `off` | `off`: no upgrade. `promoted`: track node-local `HOST_ENVD_PATH`, upgrade when strictly newer than the live version. `<sha>`: pin `/fc-envd/envd.<sha>`. | | `ENVD_UPGRADE_TARGET` (env fallback) | same | `off` | Overrides the flag fallback where there is no LD (dev), mirroring `DEFAULT_FIRECRACKER_VERSION`. | | `MinEnvdVersionForUpgrade` (const) | — | `0.6.11` | Below this the running envd lacks `/upgrade`; trigger skips. | ### Rollout metrics envd has no metrics pipeline, so its side reports via Loki summary logs (`handover_resumed{procs,retained}`, `watchers_rearmed{...}`). The orchestrator side adds: - `orchestrator.envd.upgrade.attempts{result,from_version,to_version}` — `result ∈ {success, delivery_failed, not_ready, version_mismatch, panic}`; `from_version` is the **live** running version. `success` is recorded only when the running version actually flips to `to_version` (not inferred from the transport outcome). `success/total` is the rollout success rate; the common per-resume no-op (flag off / already on target) resolves to `""` and is deliberately **not** counted. - `orchestrator.envd.upgrade.duration{result}` — wall-time of delivery + trigger + `WaitForEnvd`, i.e. the overhead added to the resume. - `orchestrator.envd.upgrade.gated{reason=old_envd}` — targeted resumes skipped by the version gate (watch during a ramp). - an `envd.upgraded` bool label on `orchestrator.sandbox.create.duration`, and an `envd-upgrade` Tempo child span. The post-upgrade readiness re-check re-runs `/init` (to re-read the running version) but is guarded to record the per-start resume KPIs — the envd-init duration histogram + call counter and `StartedAt` — **once per start**, so a successful upgrade doesn't double-count the init or push `StartedAt` (and thus `execution_time`) to a later, wrong timestamp. ```promql # rollout success rate sum(rate(orchestrator_envd_upgrade_attempts_total{result="success"}[5m])) / sum(rate(orchestrator_envd_upgrade_attempts_total[5m])) ``` ## Validation - **Unit tests (root-free, in CI):** a `Connect` after a gap-exit returns the retained Start+End by pid/tag (unknown → NotFound); a live PID is never served a stale cached exit and a reused PID keeps its successor; a stale retention timer can't evict a newer entry; a re-adopted process whose `pidfd_open` fails still emits a terminal event (no orphan); a watcher exported then imported keeps its id, delivers post-handover events, and preserves pending events; `ResumeFromHandover` always thaws on a malformed blob and on no blob; `Upgrade` refuses a target other than the fixed path; the resolver is table-driven over off/unset/promoted/sha × newer/same/older/missing (downgrade refused) without a LaunchDarkly client; `/init` advertises `X-Envd-Version`; the delivery-error classifier flags only never-reached-envd; the three metrics have description+unit entries and construct. - **Dev cluster, end-to-end (exact branch binaries; re-validated 2026-07-23 on the final code):** this branch's envd was rebuilt at 0.6.11 (baked into the template) and 0.6.12 (staged as the `promoted` target), and this branch's orchestrator (`0.2.0-c21705aeb`) deployed to the dev client nodes. Validated against a live workload — a stdout-writing + tmpfs-appending background counter, so a broken fd-carry would SIGPIPE-kill it and line growth proves it kept running: - **upgrade fires + workload survives:** flag → target ⇒ resume auto-upgrades 0.6.11→0.6.12; the running envd becomes `/usr/bin/envd.next --resume-handover` and the counter process **keeps the same PID and keeps writing across the swap** (7→20 lines) — exercising stdio fd inheritance and pidfd reaping end to end. The orchestrator confirms by ground truth: `envd_upgrade_attempts_total{result="success",from_version="0.6.11",to_version="0.6.12"} = 1`. - **flag off ⇒ no upgrade:** plain resume, workload preserved on 0.6.11, no `self-upgrade` in the journal. - **gap-exit retention:** a process that exits during the handover gap has its exit code (42) recovered by a late `Connect` via the carried retention cache. The rollout metrics are emitted on the same path (`attempts{result,from,to}`, `duration`, `gated`, `create.duration{envd_upgraded}`), observed on dev Prometheus with a per-upgrade `duration` of ≈ 345–440 ms. - **Note:** the 2026-07-23 re-run above exercised the **final** binaries (all the review-driven changes: live-version decision, version-confirmed success, the shared freeze lock held across the handover, the freeze/serialize/exec path, and the once-per-start resume-KPI guard on the post-upgrade readiness re-check). The remaining review changes that only manifest on failure/edge paths — reaper retaining before closing its channel, watcher re-arm before thaw, chained-upgrade deadline carry, exec-fail fd cleanup, and the PID-reuse / retain-timer / reaper-orphan / duplicate-forwarder fixes — are covered by the unit tests above; they don't change the success path the e2e validated. ## Key decisions - **Same-PID `syscall.Exec`, not restart.** A fresh process would be reparented by systemd (`Type=simple`), orphaning the workload; keeping the PID means envd stays the workload's parent and the inherited fds/PTYs stay valid. - **pidfd + pid-specific `wait4`, never `wait4(-1)`.** A blanket reaper would steal children that `os/exec` spawns after the upgrade; targeted reaping keeps both the re-adopted and the new processes correct. - **Trigger at resume, gated + best-effort.** Resume is the natural quiescent window and gives free cohort ramping (the resume-site LD context already carries envd-version/team/template). The whole path is recover-wrapped, version-gated, and bounded, so a failed or slow upgrade degrades to `upgraded=false` and the sandbox resumes on the old envd. *Caveat:* the trigger is currently synchronous on the resume-critical path, so on pathologically slow resumes (large cold-fault working sets) it can add up to its timeout budget to the tail; moving it off the critical path is a candidate follow-up. - **Decide and confirm on the live version, read for free off `/init`.** The orchestrator can't cheaply know the running envd version, but envd can report it — so envd advertises `pkg.Version` via an `X-Envd-Version` header on `/init` (which the resume path already calls) and the orchestrator keys the decision, `from_version`, idempotency, and the success check on it. Re-resumes of an upgraded sandbox become true no-ops and success is ground-truth-accurate, with no new endpoint and no extra round-trip. Version comparison still relies on `packages/envd/pkg/version.go` being bumped per behavioral change (repo rule); the resolver refuses any target that isn't strictly newer. - **Fixed exec target.** envd writes/execs the delivered binary only at a fixed path and refuses any other, so an authenticated-but-malformed `/upgrade` can't turn the same-PID exec into arbitrary code execution. - **`getVersion` injected into the shared resolver** so `packages/shared` gains no dependency on the orchestrator's envd build package. - **Handover blob on tmpfs**, additive-fields-only JSON, so it never enters the rootfs diff and an outgoing/incoming envd built at different versions stay compatible. - **Rollout is off by default** (`envd-upgrade-target=off`, env fallback `off`); dev has no LaunchDarkly so the feature is inert there unless `ENVD_UPGRADE_TARGET` is set. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Nikita Kalyazin <nikita.kalyazin@e2b.dev> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

feat(orch): live-upgrade envd inside a running sandbox at resume (#3333) # feat(orch): live-upgrade envd inside a running sandbox at resume ## Why envd is the in-VM daemon every sandbox depends on, but its version is baked into the snapshot at build time. A long-lived paused sandbox therefore resumes on whatever envd it was built with, potentially many releases behind — there is no way to ship an envd fix or behavioral change to the existing fleet of snapshots short of rebuilding every template. We want to roll a new envd to running sandboxes the same way we roll Firecracker versions: gated by a LaunchDarkly flag, ramped by cohort, and without disturbing the customer's workload. The hard part is doing it without a visible interruption: the swap has to keep arbitrary customer processes — and their stdio/PTYs, timeouts, filesystem watchers, and recently-observed exit codes — alive across the binary change. ## What Two halves: an in-guest handover mechanism in envd, and an orchestrator-side resume-time trigger driven by a flag. They are independent — the envd half is inert until something calls `POST /upgrade`. ### In-guest handover (envd) A new authenticated `POST /upgrade` endpoint performs a **same-PID `syscall.Exec`** into a new envd binary and re-adopts the running workload, so customer processes never notice the daemon changed underneath them. - **Delivery + trigger in one call.** The new binary is streamed in the request body, written inside the guest (default `/usr/bin/envd.next`), then swapped. This uses the token-authenticated `/upgrade` path rather than the unauthenticated `/files` copy, which a live post-`/init` sandbox rejects. - **The swap.** envd freezes the workload cgroups, serializes the process table to a tmpfs handover blob, relocates the carried fds and `syscall.Exec`s under `ForkLock`. On success it never responds — the process image is replaced. The freeze lock is **held across the whole handover** (freeze → serialize → execve, via `FreezeHold`), not just the freeze sweep, so a concurrent `/init` or `/unfreeze` thaw can't slip in and unfreeze the workload mid-serialization; on `execve` it evaporates with the process image, and on any failure it's released before the thaw. The port scanner is left running: holding `ForkLock` across the fd relocation already serializes it against the scanner's fd/socat creation, so there's no CLOEXEC race to quiesce. - **Guarded fd relocation.** The carried fds are `dup3`'d to fixed high numbers (so the fresh runtime's early startup can't have grabbed them) under `ForkLock`, and each target is checked free first — a collision aborts the upgrade (closing what it already relocated) instead of `dup3` silently clobbering a live fd, leaving the old envd running. - **What the handover carries** (tmpfs `/run/e2b/envd-handover.json`, so it never touches the rootfs diff): per-process pid / tag / cgroup / config, the stdio + PTY fds (dup'd with CLOEXEC cleared so they survive execve), the terminal-event **retention cache** (exit codes of processes that ended just before the swap), the filesystem **watchers**, and each process's **remaining timeout**. - **The incoming side** re-adopts each process from its inherited fds, resumes streaming its output, and re-arms its timeout — recording a fresh deadline so a *further* chained upgrade re-carries the remaining timeout. It restores the retention cache (skipping any PID already re-adopted as live, so a reused PID resolves to the live process, not a stale exit) and re-arms the filesystem watchers **before** thawing the workload, so the workload stays frozen until the watches are live and no event is dropped in the gap. Re-adopted processes are reaped via **pidfd + a pid-specific `wait4`** — never `wait4(-1)`, which would steal `os/exec`'s post-upgrade children — and the reaper retains a terminal event **synchronously, before** closing its event channel, so a late reconnect always finds the exit in the cache rather than racing the retain. - **Failure handling — the sandbox is never left worse off.** On an execve failure the outgoing envd closes the fds it relocated (CLOEXEC-cleared — else they'd leak into the still-running old envd and its future children), thaws the workload it froze, and keeps running the old binary. The incoming resume path always thaws via a top-level `defer` (success, error, or panic) and is recover-wrapped, so a malformed blob can't crash envd into a systemd restart that orphans the workload. Degraded at worst, never hung or dead. - **One shared freeze lock.** The freeze/unfreeze sweep and its serializing lock are factored into a single `cgroups.WorkloadFreezer` shared by the HTTP API (`/freeze`, `/unfreeze`, the `/init` deferred thaw) and the upgrade, so the upgrade's freeze can no longer interleave with the resume thaw and strand the workload frozen. envd is bumped to **0.6.11**, the first upgrade-aware version. ### Resume-time trigger + gate (orchestrator + shared) On resume (`Create` with `snapshot=true`, and the checkpoint-resume path), `maybeUpgradeEnvd` decides against the **live running envd version**, delivers, and confirms — best-effort, recover-wrapped, and bounded so it can never disrupt resume. - **Live version, not built-with.** envd advertises its running version via an `X-Envd-Version` response header on `/init`; the orchestrator captures it off the resume-path `/init` it already makes (no extra round-trip) and keys the decision, gate, `from_version`, and success check on it. The template built-with never changes across a live upgrade, so keying on it would re-trigger the handover on every resume. - **Flag-driven target resolution.** `ResolveEnvdUpgrade` (shared, the resume analog of `ResolveFirecrackerVersion`) reads the `envd-upgrade-target` flag and returns the target binary's local path + version, or `""` when the target is not strictly newer than the live version — so an already-upgraded sandbox (live == target) is a true no-op, skipped with no delivery or handover, and a downgrade is refused. Target binaries live on the node-local read-only `/fc-envd` gcsfuse mount. - **Version gate.** The *running* envd must already speak `/upgrade`, so the trigger skips a live version below `MinEnvdVersionForUpgrade` (0.6.11) and counts it (`gated{reason=old_envd}`). - **Delivery + confirmation.** `Sandbox.CallEnvdUpgrade` streams the binary over the authenticated `/upgrade` body to a fixed guest path; envd reading the body then exec'ing without replying (transport drop) is the expected path. After `WaitForEnvd` (which re-reads the version off the new envd's `/init`), success is confirmed by the running version actually equalling the target — a transport quirk can't mislabel the outcome. Only unambiguous never-reached-envd errors (connection refused / dial) are treated as delivery failures. Flags / knobs: | Flag | Values | Default | Meaning | |------|--------|---------|---------| | `envd-upgrade-target` (LD string) | `off` \| `promoted` \| `<git-sha>` | `off` | `off`: no upgrade. `promoted`: track node-local `HOST_ENVD_PATH`, upgrade when strictly newer than the live version. `<sha>`: pin `/fc-envd/envd.<sha>`. | | `ENVD_UPGRADE_TARGET` (env fallback) | same | `off` | Overrides the flag fallback where there is no LD (dev), mirroring `DEFAULT_FIRECRACKER_VERSION`. | | `MinEnvdVersionForUpgrade` (const) | — | `0.6.11` | Below this the running envd lacks `/upgrade`; trigger skips. | ### Rollout metrics envd has no metrics pipeline, so its side reports via Loki summary logs (`handover_resumed{procs,retained}`, `watchers_rearmed{...}`). The orchestrator side adds: - `orchestrator.envd.upgrade.attempts{result,from_version,to_version}` — `result ∈ {success, delivery_failed, not_ready, version_mismatch, panic}`; `from_version` is the **live** running version. `success` is recorded only when the running version actually flips to `to_version` (not inferred from the transport outcome). `success/total` is the rollout success rate; the common per-resume no-op (flag off / already on target) resolves to `""` and is deliberately **not** counted. - `orchestrator.envd.upgrade.duration{result}` — wall-time of delivery + trigger + `WaitForEnvd`, i.e. the overhead added to the resume. - `orchestrator.envd.upgrade.gated{reason=old_envd}` — targeted resumes skipped by the version gate (watch during a ramp). - an `envd.upgraded` bool label on `orchestrator.sandbox.create.duration`, and an `envd-upgrade` Tempo child span. The post-upgrade readiness re-check re-runs `/init` (to re-read the running version) but is guarded to record the per-start resume KPIs — the envd-init duration histogram + call counter and `StartedAt` — **once per start**, so a successful upgrade doesn't double-count the init or push `StartedAt` (and thus `execution_time`) to a later, wrong timestamp. ```promql # rollout success rate sum(rate(orchestrator_envd_upgrade_attempts_total{result="success"}[5m])) / sum(rate(orchestrator_envd_upgrade_attempts_total[5m])) ``` ## Validation - **Unit tests (root-free, in CI):** a `Connect` after a gap-exit returns the retained Start+End by pid/tag (unknown → NotFound); a live PID is never served a stale cached exit and a reused PID keeps its successor; a stale retention timer can't evict a newer entry; a re-adopted process whose `pidfd_open` fails still emits a terminal event (no orphan); a watcher exported then imported keeps its id, delivers post-handover events, and preserves pending events; `ResumeFromHandover` always thaws on a malformed blob and on no blob; `Upgrade` refuses a target other than the fixed path; the resolver is table-driven over off/unset/promoted/sha × newer/same/older/missing (downgrade refused) without a LaunchDarkly client; `/init` advertises `X-Envd-Version`; the delivery-error classifier flags only never-reached-envd; the three metrics have description+unit entries and construct. - **Dev cluster, end-to-end (exact branch binaries; re-validated 2026-07-23 on the final code):** this branch's envd was rebuilt at 0.6.11 (baked into the template) and 0.6.12 (staged as the `promoted` target), and this branch's orchestrator (`0.2.0-c21705aeb`) deployed to the dev client nodes. Validated against a live workload — a stdout-writing + tmpfs-appending background counter, so a broken fd-carry would SIGPIPE-kill it and line growth proves it kept running: - **upgrade fires + workload survives:** flag → target ⇒ resume auto-upgrades 0.6.11→0.6.12; the running envd becomes `/usr/bin/envd.next --resume-handover` and the counter process **keeps the same PID and keeps writing across the swap** (7→20 lines) — exercising stdio fd inheritance and pidfd reaping end to end. The orchestrator confirms by ground truth: `envd_upgrade_attempts_total{result="success",from_version="0.6.11",to_version="0.6.12"} = 1`. - **flag off ⇒ no upgrade:** plain resume, workload preserved on 0.6.11, no `self-upgrade` in the journal. - **gap-exit retention:** a process that exits during the handover gap has its exit code (42) recovered by a late `Connect` via the carried retention cache. The rollout metrics are emitted on the same path (`attempts{result,from,to}`, `duration`, `gated`, `create.duration{envd_upgraded}`), observed on dev Prometheus with a per-upgrade `duration` of ≈ 345–440 ms. - **Note:** the 2026-07-23 re-run above exercised the **final** binaries (all the review-driven changes: live-version decision, version-confirmed success, the shared freeze lock held across the handover, the freeze/serialize/exec path, and the once-per-start resume-KPI guard on the post-upgrade readiness re-check). The remaining review changes that only manifest on failure/edge paths — reaper retaining before closing its channel, watcher re-arm before thaw, chained-upgrade deadline carry, exec-fail fd cleanup, and the PID-reuse / retain-timer / reaper-orphan / duplicate-forwarder fixes — are covered by the unit tests above; they don't change the success path the e2e validated. ## Key decisions - **Same-PID `syscall.Exec`, not restart.** A fresh process would be reparented by systemd (`Type=simple`), orphaning the workload; keeping the PID means envd stays the workload's parent and the inherited fds/PTYs stay valid. - **pidfd + pid-specific `wait4`, never `wait4(-1)`.** A blanket reaper would steal children that `os/exec` spawns after the upgrade; targeted reaping keeps both the re-adopted and the new processes correct. - **Trigger at resume, gated + best-effort.** Resume is the natural quiescent window and gives free cohort ramping (the resume-site LD context already carries envd-version/team/template). The whole path is recover-wrapped, version-gated, and bounded, so a failed or slow upgrade degrades to `upgraded=false` and the sandbox resumes on the old envd. *Caveat:* the trigger is currently synchronous on the resume-critical path, so on pathologically slow resumes (large cold-fault working sets) it can add up to its timeout budget to the tail; moving it off the critical path is a candidate follow-up. - **Decide and confirm on the live version, read for free off `/init`.** The orchestrator can't cheaply know the running envd version, but envd can report it — so envd advertises `pkg.Version` via an `X-Envd-Version` header on `/init` (which the resume path already calls) and the orchestrator keys the decision, `from_version`, idempotency, and the success check on it. Re-resumes of an upgraded sandbox become true no-ops and success is ground-truth-accurate, with no new endpoint and no extra round-trip. Version comparison still relies on `packages/envd/pkg/version.go` being bumped per behavioral change (repo rule); the resolver refuses any target that isn't strictly newer. - **Fixed exec target.** envd writes/execs the delivered binary only at a fixed path and refuses any other, so an authenticated-but-malformed `/upgrade` can't turn the same-PID exec into arbitrary code execution. - **`getVersion` injected into the shared resolver** so `packages/shared` gains no dependency on the orchestrator's envd build package. - **Handover blob on tmpfs**, additive-fields-only JSON, so it never enters the rootfs diff and an outgoing/incoming envd built at different versions stay compatible. - **Rollout is off by default** (`envd-upgrade-target=off`, env fallback `off`); dev has no LaunchDarkly so the feature is inert there unless `ENVD_UPGRADE_TARGET` is set. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Nikita Kalyazin <nikita.kalyazin@e2b.dev> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

2026-07-27

feat(envd): in-guest live-upgrade handover mechanism Upgrade the envd binary inside a running sandbox by re-exec'ing into the new binary with the same PID and re-adopting the workload, so arbitrary customer processes and their stdio/PTY survive the swap. - POST /upgrade (authenticated): the new binary is streamed in the body, written, and swapped via same-PID syscall.Exec under ForkLock with the workload frozen (via the shared WorkloadFreezer); on success envd never responds (it has exec'd). - Handover blob (tmpfs) is a protobuf HandoverState (spec/upgrade/handover.proto) with an explicit schema + abort-on-newer reader rule: it carries per-process pid/tag/cgroup/config + stdio/PTY fds (CLOEXEC cleared), the terminal-event retention cache, and filesystem watchers, all as native nested messages. The incoming envd re-adopts each process (reaped via pidfd + pid-specific wait4), re-arms watchers/timeouts, restores the retention cache, and thaws the workload. - The workload stays frozen until the post-upgrade /init restores the access token (WorkloadFreezer.FreezeHold), and the pre-init auth gate keeps control endpoints closed in that window. - Retention cache is keyed by pid but also cleared by tag on Start, so a tagged restart within the TTL can't serve a predecessor's stale exit code. - Failure handling: a failed upgrade never leaves the sandbox worse off — on an execve failure the outgoing envd thaws and keeps running the old binary; the incoming resume path always thaws (success/error/panic) and is recover-wrapped. Bump envd to 0.6.12 (the first version that writes the protobuf handover) and MinEnvdVersionForUpgrade with it. Signed-off-by: Nikita Kalyazin <nikita.kalyazin@e2b.dev>

feat(envd): carry NFS mounts and port-forwards across the handover Two more pieces of re-creatable state were dropped on a live-upgrade because they live only in envd's heap while the kernel objects survive the execve: - The NFS mount ledger (path -> lifecycle) on the API service. Lost, the new envd's post-upgrade /init saw an empty ledger, decided every volume needed (re)mounting, and force-unmounted + remounted a still-live mount — ESTALE for the workload and a possible failed resume. - The port forwarder's socat set. Lost, the new forwarder rescanned and spawned a duplicate socat per already-forwarded port, while the originals were orphaned and leaked as un-reaped zombies. Carry both in the protobuf HandoverState (new MountEntry and ForwardedPort messages): - API.ExportMounts/ImportMounts round-trips the ledger so /init recognizes a matching-lifecycle mount and leaves it in place. - Forwarder.ExportForwards/ImportForwards carries each socat's pid; the new forwarder re-adopts the live ones (skipping any that didn't survive), seeding the ports map so the next scan doesn't spawn a duplicate, and reaps each re-adopted socat via wait4 (there is no surviving *exec.Cmd Wait goroutine). A mutex now guards the ports map since the upgrade goroutine exports it concurrently with the scan loop. These are returned from ResumeFromHandover (rather than applied via a callback like watchers) because their owners — the API service and the forwarder — are constructed after the resume runs. Tests: proto round-trip of mounts + forwards; the mount ledger skipping a remount for an unchanged lifecycle; and the forwarder re-adopting a live socat, skipping a dead one, and exporting only real socats. Signed-off-by: Nikita Kalyazin <nikita.kalyazin@e2b.dev>

2026-07-21

feat(api): add sandbox workload identity permission (#3319) feat(api): add sandbox IAM workload token configuration Add an optional iam.tokens map to sandbox create, keyed by a caller-chosen token name, where each definition carries an exact audience and tokenType. A non-empty, validated map enables workload identity for the sandbox; there is no separate enabled flag. Admission requires a non-empty audience and tokenType "JWT-SVID", preserves the audience exactly as received, and rejects an empty token name or any non-null filePath (file-based delivery is unsupported) with a 400 naming the offending field. The definitions are propagated to the orchestrator in SandboxConfig.iam and persisted in the running-sandbox (Redis) and paused-snapshot (Postgres) state, so they survive pause/resume and orchestrator re-sync. A resume or fork restores the configuration from the snapshot, so a fork inherits the parent sandbox's IAM configuration; the resumed/forked execution still obtains a freshly generated execution ID. The API mints, signs, injects, or delivers no credential. Refs: EN-1860 --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

feat(api): add sandbox workload identity permission Add an optional perms.id_token permission to sandbox create. When enabled, the sandbox exposes a workload identity to the orchestrator that is derived from the existing authenticated team, sandbox, execution and template identifiers plus the current deadline. No credential is minted, signed, injected, or delivered here. File-based delivery is unsupported: a non-null perms.id_token.filepath is rejected at admission (absent or null is accepted). The permission is carried through the orchestrator SandboxConfig (new additive SandboxPermissions message) and its persisted/recovery copies, so it survives pause/resume, fork, and API re-sync. A resumed or forked execution gets a freshly generated execution ID and never restores an old identity subject. Network updates leave the permission untouched. Refs: EN-1860

2026-07-20

Revert "Fence orchestrator drains by status epoch" This reverts commit 2151595b73efcc787ba42659e978ab440720b7ac.

Fence orchestrator drains by status epoch Use a dedicated Healthy-to-Draining RPC so rollout callers cannot race a process restart between observation and drain. Edge integration, module publication, and the dependent Belt pin must still land in that order.

Gate orchestrator rollout promotion by status epoch Replacement processes can start non-admitting and are promoted only through an exact identity and status-epoch transition. Legacy status overrides cannot bypass the Standby gate.

2026-07-19

Fence orchestrator status changes to exact processes Use a new RPC so older servers fail closed instead of ignoring identity fields. Both API and Edge callers must bind mutations to the discovered node and service instance before changing runtime admission.

Expose exact orchestrator drain readiness Bind replacement readiness to a process-authored drain fence and every startup, running, stopping, or failed-cleanup lifecycle. Admission closes before quiescence checks, duplicate starts are reserved before resource allocation, and force shutdown remains bounded.

feat(checkpoint): allow fs-only in-place checkpoints in orchestrator Wire a filesystem_only flag through the Checkpoint gRPC handler. Unlike the earlier destroy-and-resume-fresh design, the in-place checkpoint needs no new mode: fs-only checkpoint is just Pause with WithFilesystemSnapshot() + WithMaintainSandbox() together, which the current Pause already handles — it fsfreeze-quiesces the guest, skips the memory snapshot (mem stays NoDiff, so no memfile is produced or uploaded), seals the rootfs in the background, resumes the same VM in place, and thaws it (bestEffortFsthaw is already gated on envd fsfreeze support, so no 404 teardown on old envd). The persisted template is marked filesystem-only and cold-boots (reboots) on launch; the live sandbox keeps its memory and running processes. Only the handler flag mapping changes (was hardcoded false); prefetch is already omitted for in-place checkpoints. No SnapshotMode enum, no build_upload guards. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Babis Chalios <babis.chalios@e2b.dev>

Fence orchestrator status changes to exact processes Use a new RPC so older servers fail closed instead of ignoring identity fields. Both API and Edge callers must bind mutations to the discovered node and service instance before changing runtime admission.

Expose exact orchestrator drain readiness Bind replacement readiness to a process-authored drain fence and every startup, running, stopping, or failed-cleanup lifecycle. Admission closes before quiescence checks, duplicate starts are reserved before resource allocation, and force shutdown remains bounded.

2026-07-17

feat: add snapshot peer registration and policy fields to proto Adds SnapshotSourcePolicy (Create request) and SnapshotPeerRegistration (Pause response) plus their enums. Pure API-contract addition, no callers yet — orchestrator.pb.go regenerated via protoc, verified byte-identical to a fresh codegen run against this proto.

2026-07-15

feat(checkpoint): allow fs-only checkpoints in orchestrator Orchestrator now allows creating filesystem only checkpoints. The orchestrator server now accepts a filesystem_only flag in the checkpoint request to control whether it will include a snapshot of the memory. Checkpoint is a bit different than pause, because we need to handle the existing sandbox. The way we do this now is creating a full snapshot, destroying the previous sandbox and starting a fresh one from the freshly created sandbox. Ideally, we should be reusing the existing sandbox and not starting a new one, but this is a bigger change. For the time being, we don't change the way things are done. We create the full snapshot always. If the checkpoint is filesystem only, we mark the Snapshot object as such and use the memory part only locally (we don't upload it). Signed-off-by: Babis Chalios <babis.chalios@e2b.dev>

2026-07-14

[EN-1038] feat: add LD-gated template disk enforcement Read template disk defaults and maximums from team_limits_v2, and use the new default when registering builds. Propagate the build-start maximum through the internal TemplateCreate RPC for both build endpoints. At build completion, compare the exact final rootfs size with the maximum, emit disk-limit telemetry, and reject oversized builds when build-enforce-max-disk-size is enabled. Keep missing or invalid maximums fail-open and document the updated build flow.

2026-07-13

feat(proxy): support HTTPS sandbox backends Allow sandbox network config to select HTTPS per port. Accept self-signed backend certificates on trusted orchestrator-to-sandbox hop.

2026-07-03

feat: derive filesystem-only snapshot from memory checkpoint Rework the memory:false snapshot flow so the source sandbox keeps its memory: the checkpoint takes a full memory snapshot (build A, the sandbox's own snapshot row) after a guest sync quiesce, derives a filesystem-only sibling build (B) whose rootfs header references A's data without duplicating it, memory-resumes the sandbox from A as a normal checkpoint, and returns B as the snapshot template. Sandboxes created from B cold-boot from the captured filesystem. The sibling upload is header+metadata only and runs after A's upload (its ancestor wait requires A's finalized header). Quiesce uses guest sync rather than fsfreeze: RAM captured with a frozen filesystem would memory-resume frozen and wedge all writes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

feat(envd): kill tree on request timeout when started with child_processes Address PR review (timeout killed leader only): add a child_processes field to StartRequest so a command started with it also tears down its whole process group when a request timeout fires, keeping the timeout path consistent with an opted-in SendSignal. Non-opted-in commands and PTY processes keep the original leader-only behavior. Regenerated the process protos and added a timeout regression test. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Merge branch 'main' into mishushakov/envd-kill-process-tree

feat(envd): make process-tree kill opt-in via child_processes flag Add a child_processes field to SendSignalRequest. When set, the signal is delivered to the command's whole process group (leader + children); otherwise only the leader is signaled, preserving the original behavior. Non-PTY commands still always start in their own process group so the opt-in kill can reach the tree, but the timeout-driven kill is left as leader-only. Regenerated the process protos and added a test covering the leader-only default. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

feat: filesystem-only variant of sandbox snapshot endpoint (memory: false) Add a memory flag to POST /sandboxes/{sandboxID}/snapshots. When false, the snapshot template persists only the rootfs: sandboxes created from it cold-boot, and the source sandbox is rebooted in place (checkpoint cannot memory-resume without a memory snapshot) while keeping its identity and end time. Wired through SandboxCheckpointRequest.filesystem_only; the orchestrator branches to RebootSandbox and skips memory-prefetch work. Also tag linux-only files in cmd/create-build, cmd/resume-build, cmd/inspect-build, and pkg/server/prefetch_harvest.go with //go:build linux so make lint passes on macOS. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

feat: per-team events TTL limit (tier + addons) (#3181) Allow customization for teams to extend ttl for events

2026-07-02

feat(orchestrator): report hugepage metrics to API (#3182) Expose HugePages_Total/Used/Rsvd and hugepage size from /proc/meminfo via ServiceInfo gRPC and NodeMetrics on the admin /nodes endpoints. Page counts, not bytes, to match kernel accounting.

chore: clean up

feat: per-team events TTL limit (tier + addons) Add events_ttl_days to tiers with extra_events_ttl_days addon column, summed into the team_limits view. The limit flows through team limits into SandboxConfig (proto field 26) so the orchestrator stamps it on every published sandbox event; the events store TTL was switched to read it in a previous change. Dashboard exposes the limit and uses it for the sandbox record retention-expired check instead of a hardcoded 7 days.

2026-06-28

refactor: keep layer-size categories whole - DB carries only logical sizes Move the rootfs mapped/diff sizes out of the gRPC/DB path so each size category lives in one place: logical sizes go via gRPC into env_builds (memfile logical; rootfs logical is already total_disk_size_mb), while mapped and diff sizes for both artifacts are written to the data objects' metadata (see the memfile-layer -metadata PR). Drops the rootfs_*_size_bytes columns and the now-unused Mapping.MappedBytes helper.

feat: persist synchronously-available layer sizes to env_builds via gRPC Replace the object-metadata approach with returning per-artifact layer sizes over gRPC (pause/checkpoint responses and TemplateBuildMetadata) and persisting them on the env_builds row. Only sizes available without blocking on the async memfile dedup header are included: rootfs mapped/diff and memfile logical (rootfs logical is already stored as total_disk_size_mb). The async memfile mapped/diff sizes are handled separately.

2026-06-25

storage-api: drop stale "block size" from ingest RPC comment No block_size field exists on the proto; align the comment.

orchestrator: index builds into storage-api after headers are ready Best-effort, feature-flagged (storage-api-ingest-build); no-op when unconfigured.

feat(orchestrator): force-stop sandboxes and bound shutdown drain Escalate shutdown from graceful drain to a bounded force stop. Add SHUTDOWN_DRAIN_TIMEOUT to cap the graceful phase; on expiry (or when force-stop is requested) the orchestrator force-closes live sandboxes and cancels in-flight template builds, detaching cleanup from the canceled shutdown context while preserving its deadline. Introduce the drain-gate state both servers shut down through: the sandbox server's start gate and the template server's build gate, with StartDraining replacing the old done channel and the drain paths waiting for admitted in-flight starts to finish before snapshotting the live set. New work is still admitted here; rejecting it is layered in separately. Plumb a force_stop flag from the node status API through gRPC so an operator can drain-and-force a node, make ServiceInfo status transitions atomic so a drain override cannot race back to healthy and drain cannot be reversed, and run drains as managed services with deterministic exit reporting. Cleanup rollbacks now preserve deadlines via RunRollback.

2026-06-23

Filesystem-only snapshots for auto-pause (#3055) Summary Adds a per-sandbox autoPauseMemory create option (default true). When false, a sandbox that auto-pauses on timeout takes a filesystem-only snapshot (no memory) instead of a full memory snapshot, so it cold-boots from disk on resume. Builds on the filesystem-only snapshot machinery from #3027 — this PR just lets the timeout auto-pause path opt into it, where previously only an explicit pause(memory=false) could. How it works The FilesystemOnly flag was already plumbed end-to-end through the pause path (RemoveOpts → pauseSandbox → snapshot → Pause RPC); the only missing piece was a per-sandbox policy for the evictor to act on. 1. Create — autoPauseMemory=false → Sandbox.AutoPauseFilesystemOnly=true (runtime state, persisted to Redis). 2. Evict — when the evictor auto-pauses on timeout, it threads opts.FilesystemOnly = sbx.AutoPauseFilesystemOnly. 3. Persist — the policy is stamped onto the snapshot's PausedSandboxConfig, and... 4. Restore — re-read on resume (covers explicit /resume, /connect, and auto-resume — they share buildResumeSandboxData), so the policy survives a pause/resume cycle. 5. Re-sync — also carried through SandboxConfig.auto_pause_filesystem_only in the orchestrator proto, mirroring auto_pause, so an API restart that rebuilds from the orchestrator's sandbox list doesn't lose it. The orchestrator is unchanged — the existing IsFilesystemOnly() dispatch already routes any resume of an fs-only snapshot to RebootSandbox. Constraints (rejected at create, 400) - autoPauseMemory=false requires autoPause=true (it only governs the timeout auto-pause; otherwise a no-op). - autoPauseMemory=false is incompatible with autoResume — a filesystem-only snapshot can't be auto-resumed by arbitrary traffic (it'd cold-boot, losing processes/connections) and must be resumed explicitly. Design notes - Two distinct fields. AutoPauseFilesystemOnly (the policy for the next auto-pause) is kept separate from FilesystemOnly (the kind of this snapshot), so an explicit pause(memory=true) on such a sandbox produces a memory snapshot yet still restores the fs-only policy on the next cycle. - Resume inherits the policy. Unlike auto_pause (overridable via the resume body, now deprecated), the auto-pause kind is always inherited from the snapshot — changing it means creating a new sandbox. Deliberate, to avoid extending a deprecated path. - Backward-compatible: default true, omitempty on both the JSON config and the proto, so existing sandboxes/snapshots decode to memory auto-pause. Testing - Unit — evictor wiring (kill/pause × snapshot kind); PausedSandboxConfig serialization round-trip + legacy-row default; proto re-sync round-trip via GetSandboxes. - Integration — create → timeout auto-pause → resume cold-boots (rootfs marker survives, fresh boot id, correct default user), and the policy persists across a second pause/resume cycle; both create-time guards return 400. Usage ``` # auto-pause this sandbox as a filesystem-only snapshot on timeout POST /sandboxes { "templateID": "...", "autoPause": true, "autoPauseMemory": false } ``` --------- Signed-off-by: Babis Chalios <babis.chalios@e2b.dev> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

2026-06-22

fix(orchestrator): honor drain status overrides Make ServiceInfo status transitions atomic via TransitionStatus so a drain override cannot race a transition back to healthy, reject drain reversal, and guard ForceStop with sync.Once. Plumb a forceStop flag from the API through the node status change so draining can force stop sandboxes and template builds.

feat(api): support filesystem-only snapshots for auto-pause Timeout auto-pause always took a full memory snapshot. Add a per-sandbox autoPauseMemory option (default true) so a timeout auto-pause can instead take a filesystem-only snapshot (no memory; cold-boots on resume), reusing the existing FilesystemOnly pause path. The evictor reads the policy off the sandbox and threads FilesystemOnly into the pause. The policy is persisted on the snapshot (PausedSandboxConfig) and carried through the orchestrator SandboxConfig (auto_pause_filesystem_only) so it survives a pause/resume cycle and an API re-sync from the orchestrator's sandbox list — mirroring how auto_pause itself is handled. Reject autoPauseMemory=false combined with autoResume: a filesystem-only snapshot cannot be auto-resumed by traffic and must be resumed explicitly. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Babis Chalios <babis.chalios@e2b.dev>

feat(orchestrator): track and report last status change timestamp (#2980) Add service_status_changed_at to ServiceInfoResponse so the API can see when the orchestrator's status last changed. The timestamp is updated on actual status transitions and initialized to the service startup time. --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

2026-06-20

Filesystem-only snapshots: control plane wiring (#3027) ## Summary Adds filesystem-only snapshots: pause can persist only the rootfs (no memory snapshot), and resuming such a snapshot cold-boots (reboots) the guest from disk instead of restoring memory. Useful when in-memory state isn't worth persisting — cheaper, faster pauses. ## What's included ### Resume side - Server.Create dispatches reboot-vs-resume from the snapshot's own metadata.json (IsFilesystemOnly()); a safety gate in reboot.go refuses to cold-boot a memory snapshot. ### Control plane - filesystem_only on the pause gRPC → threaded through the API pause path → persisted in PausedSandboxConfig. - Public trigger: optional memory: bool (default true) on POST /sandboxes/{id}/pause; memory:false requests a filesystem-only snapshot. - Gating (a disk-only resume cold-boots, losing in-memory state, so implicit resumes are refused): - auto-resume → FailedPrecondition (request must resume explicitly) - /connect → 409 - explicit /resume is allowed and performs the reboot. Existing pause callers are unaffected: no body / memory unset = full memory snapshot. ## Tests Five integration tests, one per behavior: - <s>/connect refused (409) on a filesystem-only snapshot</s> - proxy traffic does not auto-resume a filesystem-only snapshot - explicit resume cold-boots: rootfs marker survives, boot_id changes - secure sandbox keeps working envd auth across the reboot (MMDS token path) - build-installed CA is in the live trust store after the reboot (boot-time half of #3025's cert validation) Validation The integration tests above are the validation gates; they require the Firecracker integration runner to actually execute (compile/vet/lint verified locally). They must pass in CI before merge. --------- Signed-off-by: Babis Chalios <babis.chalios@e2b.dev> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: ValentaTomas <49156497+ValentaTomas@users.noreply.github.com>

2026-06-19

feat(orchestrator): add filesystem_only to the pause RPC Adds SandboxPauseRequest.filesystem_only so a caller can request a filesystem-only pause over gRPC. Server.Pause threads it to sandbox.WithFilesystemSnapshot, which (via PR-A1) stamps the snapshot's metadata so resume cold-boots from the rootfs. Default false = full memory snapshot, so existing pause callers are unaffected; Checkpoint always passes false (filesystem-only checkpoint is not supported yet). The public memory:false API surface and DB persistence land next in this PR. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Babis Chalios <babis.chalios@e2b.dev>

2026-06-18

feat(storage): stamp provenance custom metadata on uploaded objects (incl. headers) (#3033) Attach immutable `team_id`, `template_id`, and `origin` (`pause`/`template_build`/`snapshot_template`) as custom object metadata on every uploaded object, including header objects (which previously carried no custom metadata at all). This lets the storage index attribute ownership/usage and classify root kinds from the object it already downloads, without the infra DB. Lineage stays in the header (`base_build_id` + reference edges); mutable, DB-owned state is deliberately excluded. --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

2026-06-12

fix(orchestrator): honor drain status overrides Make ServiceInfo status transitions atomic via TransitionStatus so a drain override cannot race a concurrent transition back to healthy, reject drain reversal (including via an Unhealthy detour), guard ForceStop with sync.Once while preserving drain-to-force-stop escalation, and plumb a forceStop flag from the API through the node status change so draining can force stop sandboxes and template builds.

feat(envd): allow opting into watching network mounts (#2982) ## Summary - Adds an `allow_network_mounts` field to `WatchDirRequest` and `CreateWatcherRequest` so clients can explicitly opt into watching paths on network filesystem mounts (NFS, CIFS, SMB, FUSE), which are rejected by default since #1810. - The streaming (`WatchDir`) and non-streaming (`CreateWatcher`) handlers skip the network-mount rejection when the flag is set; default behavior is unchanged. - Regenerates the filesystem proto code in envd and shared, and bumps envd to 0.6.4. ## Test Plan - Added `TestCreateWatcherOnNetworkMount` (bindfs FUSE mount, Linux-only) verifying the watcher is rejected without the flag and created with it; existing filesystem watch tests pass. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

feat(orchestrator): track and report last status change timestamp Add service_status_changed_at to ServiceInfoResponse so the API can see when the orchestrator's status last changed. The timestamp is updated on actual status transitions and initialized to the service startup time.

feat(api): SOCKS5 egress proxy on sandbox network config (BYOP) (#2642) Sandboxes have two related egress problems we cannot solve from our side: operators want a predictable egress identity (their public IP, not ours), and they want sandboxes to reach hosts that are reachable from their own proxy server — corporate networks, VPN endpoints, internal LAN. This PR adds Bring-Your-Own-Proxy: an optional `egressProxy: {address, username, password}` on the sandbox network config tunnels all sandbox TCP egress through a user-supplied SOCKS5 server, with our allow/deny rules still applied before the tunnel. Scope here is the API + orchestrator plumbing: OpenAPI, DB types, proto, the API create/get handlers (password omitted from GET), the `BYOPProxyEnabledFlag` LD gate, and a validator that rejects a SOCKS5 endpoint pointing at our own infra at create time and re-checks on every dial (DNS-rebind guard). For BYOP-enabled sandboxes only, the sandbox-netns kernel firewall is narrowed on demand so TCP destined for the user's internal networks reaches the userspace egress proxy instead of being dropped — without this the user's LAN destinations would never make it past the kernel. Non-BYOP sandboxes keep the existing hard kernel drop. --------- Co-authored-by: Jakub Novák <jakub@e2b.dev>

2026-06-10

feat(envd): add optional EntryInfo to watch FilesystemEvent (#2930) Adds an `include_entryinfo` flag to `WatchDirRequest` and `CreateWatcherRequest`; when set, each `FilesystemEvent` carries the affected entry's `EntryInfo` (new `optional EntryInfo entry` field). The entry is populated best-effort and left `nil` for remove/rename-away events where the path no longer exists, so the watch stream/watcher is never torn down by a stat of a gone file. Both the streaming `WatchDir` and the polling `CreateWatcher`/`GetWatcherEvents` paths are supported, with regenerated proto code in `packages/envd` and `packages/shared`. Bumps the envd version to `0.6.2` and adds `watch_test.go` covering entry-present, entry-absent, and removed-entry cases. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

feat(envd): support user-defined file metadata via xattrs (#2732) ## Summary - Adds optional user-defined file metadata to `POST /files`. Any request header of the form `X-Metadata-<key>: <value>` is persisted as an extended attribute (xattr) on the uploaded file. The `X-Metadata-` prefix is stripped and the remaining header name is lowercased to form the key. Multiple files in one multipart upload receive the same metadata. - Surfaces that metadata on every `EntryInfo`: the HTTP upload response, and the filesystem gRPC service (`Stat`, `ListDir`, `Move`, `MakeDir`, etc.) via a new `map<string, string> metadata` proto field (field 11). The wiring is centralized in `filesystem.GetEntryInfo`, so all callers pick it up. - xattrs are stored under the `user.e2b.` namespace — `user.` is required by the Linux VFS for unprivileged xattrs, and `e2b.` namespaces our keys so they don't collide with other tooling writing to `user.*`. Foreign `user.*` xattrs are filtered out and not surfaced. - Replace-on-upload semantics: each upload rewrites the file's full metadata set. Keys present from a prior upload but absent from the new request are removed, and an upload with no `X-Metadata-*` headers clears all existing metadata (`O_TRUNC` preserves xattrs, so we always rewrite). - Validation: keys and values must be printable US-ASCII (`0x20`–`0x7E`), else HTTP 400. Keys are capped at 246 bytes (255-byte VFS xattr-name limit minus the `user.e2b.` prefix); values at 1024 bytes. - Error handling on write: xattr-unsupported filesystems (e.g. `/proc`, `/sys`) are best-effort — the body is persisted and we log a warning rather than fail; `ENOSPC`/`EDQUOT` map to HTTP 507. The response `EntryInfo` reads xattrs back from disk, so it never falsely claims metadata was persisted. ## Implementation - xattr read/write/validate helpers live in a single cross-platform `packages/shared/pkg/filesystem/xattr.go`, backed by the [`github.com/pkg/xattr`](https://github.com/pkg/xattr) library rather than hand-rolled syscalls. The library handles the size-probe/retry and name parsing that previously lived in our own `xattr_linux.go`. - `ReadMetadata`/`WriteMetadata` carry the e2b-specific policy on top of the library: `user.e2b.` namespacing, full-set replace semantics, and `ValidateMetadata`. `IsXattrUnsupported` is exported so `upload.go` can treat xattr-less filesystems as best-effort without duplicating the errno check. - No platform shims needed — `pkg/xattr` is cross-platform, so the metadata tests run on Linux and macOS alike (only `extractStatTimes` keeps a `linux`/`darwin` split, for `Stat_t` field differences). - Bumps envd version to `0.5.28` and the OpenAPI spec to `0.1.3`. ## Test plan - [x] `make lint` clean for envd and shared - [x] `go test ./...` passes for `packages/envd` and `packages/shared/pkg/filesystem` (metadata tests now run on macOS too) - [ ] Upload a file with `-H 'X-Metadata-author: mish' -H 'X-Metadata-purpose: upload'` and confirm `Stat` returns the same map - [ ] Confirm xattrs land under `user.e2b.*` via `getfattr -d <path>` inside a sandbox - [ ] Re-upload the same path with different headers and confirm prior keys are cleared 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> Co-authored-by: Jakub Novak <jakub@e2b.dev>

2026-06-09

refactor: split sandbox logs LogQL query filter into its own PR Move the /v2/sandboxes/{sandboxID}/logs `query` field and its supporting provider/resources/edge changes out to a separate PR (#2963). This branch now contains only the envd command-output persistence: stdout/stderr is written to the log pipeline (stamped with pid, capped per command). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Merge remote-tracking branch 'origin/main' into mishushakov/command-logs-method # Conflicts: # packages/api/internal/api/api.gen.go

2026-06-06

refactor(api): identify command logs by pid + time window instead of cid Drop the envd-assigned cid (and its proto changes); commands are already identified by the pid returned in StartEvent. Output lines are now stamped with pid, and the retrieval endpoint becomes GET /v2/sandboxes/{sandboxID}/commands/{pid}/logs with start/end query params. The time window disambiguates a reused pid: within [start, end] a pid maps to a single command execution. Filter stays scoped to event_type=process_output. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

fix(orch): split scheduling base build id per artifact (#2920) Memfile and rootfs can have different base builds, so the single `base_build_id` in `SchedulingMetadata` was lossy. Replace it with `memfile_base_build_id` and `rootfs_base_build_id`, each pinned into its own artifact build list. Made with [Cursor](https://cursor.com) Co-authored-by: Cursor <cursoragent@cursor.com>

2026-06-05

feat(envd): persist command output and add per-command log retrieval Commands started via envd now get a unique cid that is returned in the StartEvent and stamped on every stdout/stderr log line, so their output is persisted through the existing Loki pipeline and capped per command. Adds GET /v2/sandboxes/{sandboxID}/commands/{cid}/logs to retrieve a single command's output, with the cid filter threaded through the local Loki query and the remote edge contract. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

2026-06-01

Expose internal scheduling metadata (#2873) Expose compact scheduling metadata from orchestrator and template-manager without DB or public API changes. --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

2026-05-28

feat(observability): add kill_reason to sandbox.lifecycle.killed (#2833) Sandbox terminations were unattributed at the event/metric layer, leaving only `"SandboxError: 2: [unknown] terminated"` as a signal. Adds a `KillReason` on `RemoveOpts` and an optional `kill_reason` field on SandboxDeleteRequest; Server.Delete writes it to `event_data.kill_reason` (consumed by the existing webhook and ClickHouse delivery, no schema change) and increments `orchestrator.sandbox.killed` labeled by reason. Reasons: `request`, `timeout`, `admin`, `orphaned`, `base_template_missing`. Kills that reach Server.Delete without a specific reason fall through as `unknown`. --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

2026-05-21

fix(orchestrator): preserve checkpoint state

2026-05-12

Merge origin/main into feat/traffic-keepalive

2026-05-11

fix(orch): P2P reader race on peer→storage transition When a peer signaled UseStorage to indicate its upload finished, the current reader saw `uploaded=true` and routed to base storage, but concurrent sibling reads on the same File could fall through with a stale FrameTable — V4 chunk offsets shift between the pre-finalize in-memory header and the post-finalize GCS header, so reads decoded the wrong bytes. The peer now ships the final V4 header inline on its UseStorage response (new `header_bytes` field on `PeerAvailability`). The bytes are the same `[]byte` produced once by `header.StoreHeader` during upload — captured on `*Upload` and forwarded into the server's `uploadedBuilds` cache, no second serialization, no failure path. Client side, the resolver's per-buildID state holds the parsed header. `build.File.installPendingHeader` polls the state at the top of every Slice/ReadAt iteration and CAS-installs the new header atomically. Concurrent readers either install the same value or observe it already installed — no coordination primitive needed, no GCS poll. Also: mid-stream peer abort (NotAvailable from cache eviction) now surfaces as `storage.ErrPeerAborted` so the chunker reopens via base on a single retry, instead of silently truncating the stream.

2026-05-07

fix(peerclient): resolve base path per-call from live FrameTable (#2585) After a peer→storage transition, peerSeekable kept opening base against the original (uncompressed) path captured at construction. With the recent compression work, post-transition reads now target a compressed object, so the cached base resolved to a non-existent path. P2P routing produced the stale binding (it captured a path while the build was still uncompressed); it also contains the fix. peerSeekable now holds (buildID, basic name, base provider, objType) and composes the actual storage path at base-open time using the CompressionType from the live FrameTable. The base seekable is reopened on ct change. No changes outside peerclient.

Merge remote-tracking branch 'origin/main' into feat/traffic-keepalive

2026-05-05

Persist keepalive policy in orchestrator config

feature: Compression for memfile/rootfs assets (#2034) Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Jakub Dobry <jakub.dobry8@gmail.com>

Merge branch 'main' into feat/traffic-keepalive # Conflicts: # packages/api/internal/api/api.gen.go # packages/client-proxy/internal/proxy/proxy.go

Api for setting up domain transform (#2515) Support for setting injection headers. Sandbox creation and network update are supported. Configuration is stored in the database to support the sandbox resume flow. --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

2026-04-24

Merge remote-tracking branch 'origin/main' into feat/traffic-keepalive # Conflicts: # packages/api/internal/api/api.gen.go

2026-04-23

keepalive only via redis

fix(orch): resolve firecracker version via feature flag in template builder (#2484) * fix(orch): resolve firecracker version via feature flag in template build path The template build path took the API-supplied firecracker version verbatim, while the sandbox create path (pkg/server/sandboxes.go:161) already routed it through featureflags.ResolveFirecrackerVersion. Orchestrators that lag behind the API's preferred FC binary would fail builds because the binary did not exist on disk. Resolve the version inside Builder.Build right after the LD context is set up, mirroring the sandbox path. Also leaves TODOs at the two API call sites to stop sending the version entirely in a follow-up. * feat(template-manager): return kernel and firecracker versions in build metadata Kernel and firecracker versions in TemplateConfig are deprecated on the wire: the template-manager now selects its own versions (falling back to DefaultKernelVersion and the BuildFirecrackerVersion feature flag when the API leaves them empty, then running them through ResolveFirecrackerVersion) and reports the versions it actually used back via TemplateBuildMetadata. The API still seeds kernel_version and firecracker_version on the env_builds row at registration time for backwards compatibility with consumers that read the record before the build completes. Once FinishTemplateBuild runs, it overwrites the row with whatever the template-manager reported. A COALESCE(NULLIF(..., ''), column) fallback in the SQL keeps the seeded values intact when talking to an older template-manager that does not populate the new fields. * chore: clean up * fix(api): format Deprecated notices in dedicated paragraphs (gocritic) * fix(orch): recompute HugePages in Builder.Build from the resolved FC version The API still computes HugePages from its own BuildFirecrackerVersion evaluation and sends it to the template-manager over gRPC. The API and the orchestrator evaluate the feature flag under different LaunchDarkly contexts (API: team/user/tier/cluster; orchestrator: template/team), and the resolutions happen at different times, so the two sides can legitimately disagree on the firecracker version. HasHugePages() flips at v1.7, and a cross-1.7 divergence configures the VM's memfile page size (2 MiB vs 4 KiB) for a firecracker binary the orchestrator is not actually going to launch — corrupting the memory file. Move the firecracker version parsing helper to packages/shared/pkg/fcversion so both services share a single implementation, and have Builder.Build override cfg.HugePages right after it has resolved the firecracker version it will actually use. The API keeps computing HugePages for backwards compatibility with older orchestrators; new orchestrators treat the wire value as a hint and always recompute locally. * chore: auto-commit generated changes * refactor(orch): resolve kernel/FC version and HugePages in the gRPC handler Builder.Build was unconditionally rewriting cfg.KernelVersion, cfg.FirecrackerVersion and cfg.HugePages on every call. That was fine for the gRPC path where those fields come from a deprecated proto message and are effectively unset, but it silently clobbered in-process callers: packages/orchestrator/cmd/create-build/main.go passes explicit -kernel and -firecracker CLI flags, and packages/orchestrator/cmd/smoketest iterates FirecrackerVersionMap and hands each version to Builder.Build directly. Move the resolution to the gRPC handler in packages/orchestrator/pkg/template/server/create_template.go: - ServerStore now holds the featureflags client. - TemplateCreate sets up the LaunchDarkly template+team context, reads the BuildFirecrackerVersion flag, derives HugePages from the resolved firecracker semver (fcversion.New + HasHugePages), and passes all three into the TemplateConfig it hands to Builder.Build. - Builder.Build itself no longer mutates these fields; it trusts what the caller passed. The LD context it adds remains (for any downstream flag reads from the builder) and is override-safe when the handler already added the same keys. Also mark TemplateConfig.hugePages as [deprecated = true] on the wire, mirroring the existing kernel/firecracker deprecations. HugePages is derived from the firecracker version and the orchestrator now computes it locally, so senders should leave it at the zero value. * chore: clean up * chore: clean up * chore: clean up * chore: add reference to linear task --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

2026-04-21

PR feedback: switch to PATCH and carry live metadata on RunningSandbox - PUT → PATCH /sandboxes/{id}/metadata with merge semantics (k=v upserts, k=null or "" removes, absent keys untouched). Addresses ValentaTomas's concern that full-replace would force clients to round-trip existing tags to avoid wiping them. - Keep APIStoredConfig write-once (it was already deprecated and never mutated elsewhere). Add a metadata field to RunningSandbox so List ships the live sbx.GetAPIMetadata() instead of the create-time snapshot; the API-side nodemanager now reads from there. Checkpoint overrides the resumed sandbox's metadata with the live value after ResumeSandbox seeds from the stale proto. Fixes the claude[bot]/codex P1 about PATCH values reverting on resume and being invisible to List reconciles.

feat(api): add PUT /sandboxes/{id}/metadata Replace metadata on a running sandbox. Full-replace semantics — keys absent from the body are removed. `null` and `{}` both clear the map. Paused sandboxes and merge semantics deferred. Wire-up: gRPC SandboxUpdateRequest gains an optional SandboxMetadataUpdate wrapper so proto3 can distinguish "don't touch" from "clear" (bare map<string,string> cannot).

Refresh sandboxes from proxy traffic

Add traffic keepalive lifecycle config

2026-04-10

more progress!

2026-04-02

feat: persist volume path to database and pass through gRPC Adds volume_path persistence throughout the stack so the orchestrator no longer needs to recompute filesystem paths for volumes. Changes: - DB: add nullable volume_path column to volumes table - Proto: add optional volume_path to VolumeInfo, SandboxVolumeMount, and CreateVolumeResponse - Orchestrator: CreateVolume returns the computed path; getVolumeRootPath and getFilesystemAndPath use persisted path when available, falling back to BuildVolumePath for existing volumes without a path - NFS handler: same fallback pattern for chroot resolution - API: store orchestrator-returned path in DB after creation, include in JWT (volpath claim), pass through to sandbox mounts and resume - Types: add VolumePath to SandboxVolumeMountConfig for snapshot persistence Fully backwards-compatible: existing volumes without volume_path continue to work via the legacy path computation.

2026-03-25

Merge remote-tracking branch 'e2b/main' into lev-ingress-control

2026-03-24

Autoresume timeout persisted on resume (#1969) * autoresume: persist and clamp timeout on resume to min 5 minutes

2026-03-22

Merge branch 'main' of github.com:e2b-dev/infra into lev-ingress-control

2026-03-20

Simplify delete file|dir to just delete path (#2181)

Merge branch 'main' of github.com:e2b-dev/infra into lev-ingress-control

Unify egress/ingress proto format to repeated string CIDRs Delete IngressRule proto message; ingress now uses repeated string allowed_cidrs/denied_cidrs in CIDR[:port[-port]] format, matching egress. Merge parseEgressRules and parseIngressRules into a single ParseValidRules in sandbox-network. Strings pass through from API to orchestrator without intermediate conversion. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

2026-03-19

use chroot/umount/pivotroot instead of jailing (#2091) Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Claude <noreply@anthropic.com>

Merge branch 'main' of github.com:e2b-dev/infra into lev-ingress-control

feat(orch): add ScallignDown status (#2146) * feat(orch): add ScallignDown status * chore: add description * chore: auto-commit generated changes * chore: fix TOCTOU race * chore: regenerate spec * chore: rename scalling down to standy --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

2026-03-18

feat: add egress/ingress Off switches for O(1) traffic blocking Add `off` bool to egress/ingress proto configs, OpenAPI spec, and DB types. When set, all traffic is denied immediately without iterating rules. Replaces the legacy `allow_internet_access=false` pattern of injecting 0.0.0.0/0 into denied list. Off is mutually exclusive with allow/deny rules (validated at API layer). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

2026-03-17

feat: dynamic egress network rule updates for running sandboxes (#2074)

refactor: revert egress to allow-deny API, add structured ingress proto Egress: revert gRPC to allowed_cidrs/denied_cidrs/allowed_domains (no port support). Firewall reverts to IP-set-only nftables rules. TCP proxy reverts to no-port egress matching. Ingress: add IngressRule proto {cidr, port_low, port_high}. API does all parsing/validation and sends pre-validated structs to orchestrator. Orchestrator trusts data, no re-parsing. API now rejects bare IPs — inputs must be CIDRs or domains (egress) and CIDRs only (ingress). Remove egress port integration tests. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

2026-03-15

Merge branch 'lev-egress-port-ranges' into lev-ingress-control

feat(orch): unify proto fields, add port-specific egress rules, integration tests Rename proto fields (allowed_cidrs→allowed, denied_cidrs→denied, drop allowed_domains) and update all consumers. Refactor isEgressAllowed into matchRule with domain/CIDR+port matching. Fix domain entry leak into nftables path. Add comprehensive integration tests for port-specific egress rules covering TCP, UDP, port ranges, CIDR:port, domain:port, and allow/deny precedence. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

2026-03-12

Merge remote-tracking branch 'e2b/lev-allow-deny-dynamic' into lev-ingress-control

Merge branch 'main' of github.com:e2b-dev/infra into lev-allow-deny-dynamic

Implement depth parameter (#2102) Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>