Proto commits in unkeyed/unkey

These commits are when the Protocol Buffers files have changed: (only the last 100 relevant commits are shown)

2026-07-08

feat(api): create deployment endpoint (#6617) * feat(api): create deployment endpoint * fix: generation * feat: add handler and tests * refactor: move audit logging to rpc and clean up callers * refactor: change ui permissions and remove old endpoints * fix: redeploy bug and add missing fork on redeploy * chore: tidy * refactor: prevent leaking internals * chore: refine comment * revert: restore old endpoint and mark them as deprecated instead * revert: rest of the removed

feat(billing): Deploy spend-cap detection and budget alerts (ENG-2904) Set Resend Idempotency-Key on budget alert sends so a retried Restate step does not double-email after a transient failure that still reached Resend.

feat(billing): Deploy spend-cap detection and budget alerts (ENG-2904)

feat(ctrl): rewire Deploy cancel through ctrl-api (ENG-2922)

feat(ctrl): Deploy spend-cap enforcement suspend/resume (ENG-2923) Suspend/resume via teardown, wake gate, dashboard meter parity, and deploy_spend_check integration test (requires DeployTeardownService).

feat(ctrl): Deploy teardown service (ENG-2922)

refactor(ctrl)!: per-workspace push invocations for deploy billing Resolve close.go rebase conflict with eng-2865: keep fanout push/finalize flow and eng-2865 defer-on-close-failure behavior (no customer-scoped fallback).

feat(ctrl): month-end invoice closing webhook for Deploy billing Stripe invoice.created webhook on ctrl-api plus backup cron to push final period usage and finalize renewal drafts. Bounded retries with Kill on exhaustion, per-workspace defer on close failures, skip when subscription id is missing. Midnight-UTC anchor at Deploy checkout; no migration needed.

2026-07-07

feat(api): create deployment endpoint (#6617) * feat(api): create deployment endpoint * fix: generation * feat: add handler and tests * refactor: move audit logging to rpc and clean up callers * refactor: change ui permissions and remove old endpoints * fix: redeploy bug and add missing fork on redeploy * chore: tidy * refactor: prevent leaking internals * chore: refine comment * revert: restore old endpoint and mark them as deprecated instead * revert: rest of the removed

2026-07-06

feat(api): create deployment endpoint (#6617) * feat(api): create deployment endpoint * fix: generation * feat: add handler and tests * refactor: move audit logging to rpc and clean up callers * refactor: change ui permissions and remove old endpoints * fix: redeploy bug and add missing fork on redeploy * chore: tidy * refactor: prevent leaking internals * chore: refine comment * revert: restore old endpoint and mark them as deprecated instead * revert: rest of the removed

2026-07-05

Make slack dedupe and installed workspace wide so each project can have a channel. Make it easier to configure.

2026-07-04

fix: slack approved

feat(ctrl): Deploy spend-cap enforcement suspend/resume (ENG-2923) Suspend/resume via teardown, wake gate, dashboard meter parity, and deploy_spend_check integration test (requires DeployTeardownService).

feat(ctrl): rewire Deploy cancel through ctrl-api (ENG-2922)

init: make slack notifications great

2026-07-03

feat(ctrl): Deploy teardown service (ENG-2922) Adds the DeployTeardownService virtual object: it snapshots a workspace's running deployments, clears apps.current_deployment_id, and schedules archive or standby with drain polling. SUSPEND records the running set so a later resume can restore it. Also removes svc/ctrl/integration/sync_test.go, which stopped compiling after the harness changes and is not re-added here.

feat(billing): Deploy spend-cap detection and budget alerts (ENG-2904)

feat(api): create deployment endpoint (#6617) * feat(api): create deployment endpoint * fix: generation * feat: add handler and tests * refactor: move audit logging to rpc and clean up callers * refactor: change ui permissions and remove old endpoints * fix: redeploy bug and add missing fork on redeploy * chore: tidy * refactor: prevent leaking internals * chore: refine comment * revert: restore old endpoint and mark them as deprecated instead * revert: rest of the removed

feat(ctrl): Deploy teardown service (ENG-2922) Adds the DeployTeardownService virtual object: it snapshots a workspace's running deployments, clears apps.current_deployment_id, and schedules archive or standby with drain polling. SUSPEND records the running set so a later resume can restore it. Also removes svc/ctrl/integration/sync_test.go, which stopped compiling after the harness changes and is not re-added here.

feat(ctrl): Deploy spend-cap enforcement suspend/resume (ENG-2923) Suspend/resume via teardown, wake gate, dashboard meter parity, and deploy_spend_check integration test (requires DeployTeardownService).

feat(ctrl): rewire Deploy cancel through ctrl-api (ENG-2922)

refactor(ctrl)!: per-workspace push invocations for deploy billing Not behavior-preserving. The hourly push and month-end close now fan out one Restate invocation per workspace instead of pushing inline, so each push retries and fails in isolation: one workspace's broken Stripe state no longer stalls the others or the tick. The close awaits its pushes and, when any fail, leaves those workspaces' invoices open and returns a TerminalError so the fresh-key backup cron genuinely re-pushes them rather than finalizing an under-billed total. The close also gates finalization on two durable sleeps: a 15-minute ingestion buffer after the period ends before the final ClickHouse read (late rows would otherwise be permanently unbilled once the invoice finalizes), and a 20-minute wait between the final meter push and finalization so Stripe has aggregated the events into the draft's metered lines. DEPLOY NOTE: this changes the journaled step sequence of RunDeployBillingPush and RunDeployBillingClose. Drain in-flight deploy-billing invocations before deploying, or old journals may fail replay.

feat(ctrl): month-end invoice closing webhook for Deploy billing Closes the last correctness gap in the Deploy money path: without this, the billed month-end total was whatever the last hourly push happened to deliver, losing up to an hour of usage at every period roll. New pkg/webhook: a small receiver framework for inbound provider webhooks (method/body policing, signature verification via pluggable verifiers, event-type routing, middleware for integration-specific concerns, and transport metrics so webhook integrations never fly blind): - unkey_webhook_events_total{provider,event,outcome} with outcomes handled / ignored / unhandled / error / verification_failed / bad_request, and unkey_webhook_handler_duration_seconds. Defined in the package itself, next to the code that emits them. - Handlers return nil (200), ErrIgnore (200, counted separately), or an error (500, provider retries) — retry semantics are explicit. - Verifiers live under pkg/webhook/verifiers/ (stripe today; the GitHub webhook can migrate later). Flow: - ctrl-api gains POST /webhooks/stripe (new StripeConfig). On invoice.created for a Deploy workspace renewal invoice it claims the draft (auto_advance off) and dispatches RunDeployBillingClose for the closed period via Restate ingress; the per-customer invoice storm at the roll dedupes onto one durable invocation through the idempotency key. Everything else is ignored and keeps Stripe's own finalization. - RunDeployBillingClose (new CronService handler, keyed by the CLOSED "YYYY-MM" period) pushes every billable workspace's final full-period usage, stamped one second before period end so the "last"-formula meters bill the complete month, then finalizes Deploy workspaces' draft renewal invoices (new internal/invoicecloser), fanned out in bounded batches with one journaled step per workspace. Refuses to close a period that has not ended; losing a finalize race counts as done. - A 01:05 UTC backup cron invokes the same handler with the same idempotency key: if the webhook ran it attaches to that invocation, otherwise drafts are still drafts (auto_advance was disabled) and the sweep closes them. Total ctrl outage degrades to Stripe's one-hour auto-finalization. Prerequisite correctness fix: subscription billing anchors now pin midnight UTC (hour/minute/second 0), not just day_of_month 1. Without it periods are not calendar months and the renewal invoice's meter window would swallow the next month's early month-to-date events, billing the new month's tiny running total instead of the closed month's final one. The hourly push and the close share one pipeline (pushUsage) with an explicit window end and meter-event timestamp; billingperiod gains End(). ENG-2865

Merge remote-tracking branch 'origin/main' into worktree-delayed-soft-deletes # Conflicts: # dev/k8s/charts/restate-cronjobs/values.yaml # docs/engineering/docs.json # gen/proto/ctrl/v1/project.pb.go # gen/proto/hydra/v1/app.pb.go # gen/proto/hydra/v1/cron.pb.go # gen/proto/hydra/v1/cron_restate.pb.go # gen/proto/hydra/v1/environment.pb.go # gen/proto/hydra/v1/project.pb.go # pkg/db/BUILD.bazel # pkg/db/app_find_with_settings.sql_generated.go # pkg/db/app_list_by_project.sql_generated.go # pkg/db/github_repo_connection_list_deploy_context.sql_generated.go # pkg/db/project_find_by_workspace_slug.sql_generated.go # pkg/db/querier_generated.go # pkg/db/queries/app_list_by_project.sql # pkg/db/queries/project_find_by_workspace_slug.sql # svc/ctrl/api/run.go # svc/ctrl/proto/hydra/v1/app.proto # svc/ctrl/proto/hydra/v1/cron.proto # svc/ctrl/proto/hydra/v1/environment.proto # svc/ctrl/proto/hydra/v1/project.proto # svc/ctrl/services/app/delete_app.go # svc/ctrl/services/project/BUILD.bazel # svc/ctrl/services/project/delete_project.go # svc/ctrl/services/project/service.go # svc/ctrl/worker/app/delete_permanently_handler.go # svc/ctrl/worker/cron/BUILD.bazel # svc/ctrl/worker/cron/cron.go # svc/ctrl/worker/environment/delete_permanently_handler.go # svc/ctrl/worker/project/delete_handler.go # svc/ctrl/worker/run.go # web/apps/dashboard/components/navigation/sidebar-v2/sidebar-body.tsx # web/apps/dashboard/gen/proto/ctrl/v1/project_pb.ts # web/apps/dashboard/lib/flags/index.ts # web/apps/dashboard/lib/flags/resolve.ts # web/apps/dashboard/lib/navigation/leaves.ts # web/apps/dashboard/lib/trpc/routers/index.ts

refactor: move audit logging to rpc and clean up callers

feat(ctrl): Deploy spend-cap enforcement suspend/resume (ENG-2923) Suspend/resume via teardown, wake gate, dashboard meter parity, and deploy_spend_check integration test (requires DeployTeardownService).

feat(ctrl): rewire Deploy cancel through ctrl-api (ENG-2922)

feat(ctrl): Deploy teardown service (ENG-2922) Adds the DeployTeardownService virtual object: it snapshots a workspace's running deployments, clears apps.current_deployment_id, and schedules archive or standby with drain polling. SUSPEND records the running set so a later resume can restore it. Also removes svc/ctrl/integration/sync_test.go, which stopped compiling after the harness changes and is not re-added here.

feat(billing): Deploy spend-cap detection and budget alerts (ENG-2904) deploy-spend-check cron, workos client, deployspendcheck package, orchestrator integration test, and deployCredits tests. Drop deploy_spend_check integration test from this PR; it needs DeployTeardownService (ENG-2922+).

refactor(ctrl)!: per-workspace push invocations for deploy billing Not behavior-preserving. The hourly push and month-end close now fan out one Restate invocation per workspace instead of pushing inline, so each push retries and fails in isolation: one workspace's broken Stripe state no longer stalls the others or the tick. The close awaits its pushes and, when any fail, leaves those workspaces' invoices open and returns a TerminalError so the fresh-key backup cron genuinely re-pushes them rather than finalizing an under-billed total. The close also gates finalization on two durable sleeps: a 15-minute ingestion buffer after the period ends before the final ClickHouse read (late rows would otherwise be permanently unbilled once the invoice finalizes), and a 20-minute wait between the final meter push and finalization so Stripe has aggregated the events into the draft's metered lines. DEPLOY NOTE: this changes the journaled step sequence of RunDeployBillingPush and RunDeployBillingClose. Drain in-flight deploy-billing invocations before deploying, or old journals may fail replay.

feat(ctrl): month-end invoice closing webhook for Deploy billing Closes the last correctness gap in the Deploy money path: without this, the billed month-end total was whatever the last hourly push happened to deliver, losing up to an hour of usage at every period roll. New pkg/webhook: a small receiver framework for inbound provider webhooks (method/body policing, signature verification via pluggable verifiers, event-type routing, middleware for integration-specific concerns, and transport metrics so webhook integrations never fly blind): - unkey_webhook_events_total{provider,event,outcome} with outcomes handled / ignored / unhandled / error / verification_failed / bad_request, and unkey_webhook_handler_duration_seconds. Defined in the package itself, next to the code that emits them. - Handlers return nil (200), ErrIgnore (200, counted separately), or an error (500, provider retries) — retry semantics are explicit. - Verifiers live under pkg/webhook/verifiers/ (stripe today; the GitHub webhook can migrate later). Flow: - ctrl-api gains POST /webhooks/stripe (new StripeConfig). On invoice.created for a Deploy workspace renewal invoice it claims the draft (auto_advance off) and dispatches RunDeployBillingClose for the closed period via Restate ingress; the per-customer invoice storm at the roll dedupes onto one durable invocation through the idempotency key. Everything else is ignored and keeps Stripe's own finalization. - RunDeployBillingClose (new CronService handler, keyed by the CLOSED "YYYY-MM" period) pushes every billable workspace's final full-period usage, stamped one second before period end so the "last"-formula meters bill the complete month, then finalizes Deploy workspaces' draft renewal invoices (new internal/invoicecloser), fanned out in bounded batches with one journaled step per workspace. Refuses to close a period that has not ended; losing a finalize race counts as done. - A 01:05 UTC backup cron invokes the same handler with the same idempotency key: if the webhook ran it attaches to that invocation, otherwise drafts are still drafts (auto_advance was disabled) and the sweep closes them. Total ctrl outage degrades to Stripe's one-hour auto-finalization. Prerequisite correctness fix: subscription billing anchors now pin midnight UTC (hour/minute/second 0), not just day_of_month 1. Without it periods are not calendar months and the renewal invoice's meter window would swallow the next month's early month-to-date events, billing the new month's tiny running total instead of the closed month's final one. The hourly push and the close share one pipeline (pushUsage) with an explicit window end and meter-event timestamp; billingperiod gains End(). ENG-2865

2026-07-02

feat(ctrl): Deploy spend-cap enforcement suspend/resume (ENG-2923)

feat(ctrl): rewire Deploy cancel through ctrl-api (ENG-2922)

feat(ctrl): Deploy teardown service (ENG-2922) Adds the DeployTeardownService virtual object: it snapshots a workspace's running deployments, clears apps.current_deployment_id, and schedules archive or standby with drain polling. SUSPEND records the running set so a later resume can restore it. Also removes svc/ctrl/integration/sync_test.go, which stopped compiling after the harness changes and is not re-added here.

feat(billing): Deploy spend-cap detection and budget alerts (ENG-2904) deploy-spend-check cron prices each budgeted workspace's month-to-date Deploy usage locally from ClickHouse, computes net-of-credit overage against the configured budget, and emails the workspace at 50/75/100%. Enforcement (ENG-2923 SUSPEND) is gated on the ENG-2922 teardown primitive and follows. Plan: todo/eng-2904-2923-spend-cap-enforcement.md

refactor(ctrl)!: per-workspace push invocations for deploy billing Not behavior-preserving. The hourly push and month-end close now fan out one Restate invocation per workspace instead of pushing inline, so each push retries and fails in isolation: one workspace's broken Stripe state no longer stalls the others or the tick. The close awaits its pushes and, when any fail, leaves those workspaces' invoices open and returns a TerminalError so the fresh-key backup cron genuinely re-pushes them rather than finalizing an under-billed total. The close also gates finalization on two durable sleeps: a 15-minute ingestion buffer after the period ends before the final ClickHouse read (late rows would otherwise be permanently unbilled once the invoice finalizes), and a 20-minute wait between the final meter push and finalization so Stripe has aggregated the events into the draft's metered lines. DEPLOY NOTE: this changes the journaled step sequence of RunDeployBillingPush and RunDeployBillingClose. Drain in-flight deploy-billing invocations before deploying, or old journals may fail replay.

feat: allow ratelimit configuration for keys (#6562) * feat: allow ratelimit configuration for keys * implement union with some refining * remove required

feat(ctrl): month-end invoice closing webhook for Deploy billing Closes the last correctness gap in the Deploy money path: without this, the billed month-end total was whatever the last hourly push happened to deliver, losing up to an hour of usage at every period roll. New pkg/webhook: a small receiver framework for inbound provider webhooks (method/body policing, signature verification via pluggable verifiers, event-type routing, middleware for integration-specific concerns, and transport metrics so webhook integrations never fly blind): - unkey_webhook_events_total{provider,event,outcome} with outcomes handled / ignored / unhandled / error / verification_failed / bad_request, and unkey_webhook_handler_duration_seconds. Defined in the package itself, next to the code that emits them. - Handlers return nil (200), ErrIgnore (200, counted separately), or an error (500, provider retries) — retry semantics are explicit. - Verifiers live under pkg/webhook/verifiers/ (stripe today; the GitHub webhook can migrate later). Flow: - ctrl-api gains POST /webhooks/stripe (new StripeConfig). On invoice.created for a Deploy workspace renewal invoice it claims the draft (auto_advance off) and dispatches RunDeployBillingClose for the closed period via Restate ingress; the per-customer invoice storm at the roll dedupes onto one durable invocation through the idempotency key. Everything else is ignored and keeps Stripe's own finalization. - RunDeployBillingClose (new CronService handler, keyed by the CLOSED "YYYY-MM" period) pushes every billable workspace's final full-period usage, stamped one second before period end so the "last"-formula meters bill the complete month, then finalizes Deploy workspaces' draft renewal invoices (new internal/invoicecloser), fanned out in bounded batches with one journaled step per workspace. Refuses to close a period that has not ended; losing a finalize race counts as done. - A 01:05 UTC backup cron invokes the same handler with the same idempotency key: if the webhook ran it attaches to that invocation, otherwise drafts are still drafts (auto_advance was disabled) and the sweep closes them. Total ctrl outage degrades to Stripe's one-hour auto-finalization. Prerequisite correctness fix: subscription billing anchors now pin midnight UTC (hour/minute/second 0), not just day_of_month 1. Without it periods are not calendar months and the renewal invoice's meter window would swallow the next month's early month-to-date events, billing the new month's tiny running total instead of the closed month's final one. The hourly push and the close share one pipeline (pushUsage) with an explicit window end and meter-event timestamp; billingperiod gains End(). ENG-2865

feat(ctrl): Deploy spend-cap enforcement suspend/resume (ENG-2923)

feat(ctrl): rewire Deploy cancel through ctrl-api (ENG-2922)

feat(billing): Deploy spend-cap detection and budget alerts (ENG-2904) deploy-spend-check cron prices each budgeted workspace's month-to-date Deploy usage locally from ClickHouse, computes net-of-credit overage against the configured budget, and emails the workspace at 50/75/100%. Enforcement (ENG-2923 SUSPEND) is gated on the ENG-2922 teardown primitive and follows. Plan: todo/eng-2904-2923-spend-cap-enforcement.md

feat(ctrl): Deploy teardown service (ENG-2922) Adds the DeployTeardownService virtual object: it snapshots a workspace's running deployments, clears apps.current_deployment_id, and schedules archive or standby with drain polling. SUSPEND records the running set so a later resume can restore it. Also removes svc/ctrl/integration/sync_test.go, which stopped compiling after the harness changes and is not re-added here.

refactor(ctrl)!: per-workspace push invocations for deploy billing Not behavior-preserving. The hourly push and month-end close now fan out one Restate invocation per workspace instead of pushing inline, so each push retries and fails in isolation: one workspace's broken Stripe state no longer stalls the others or the tick. The close awaits its pushes and, when any fail, leaves those workspaces' invoices open and returns a TerminalError so the fresh-key backup cron genuinely re-pushes them rather than finalizing an under-billed total. The close also gates finalization on two durable sleeps: a 15-minute ingestion buffer after the period ends before the final ClickHouse read (late rows would otherwise be permanently unbilled once the invoice finalizes), and a 20-minute wait between the final meter push and finalization so Stripe has aggregated the events into the draft's metered lines. DEPLOY NOTE: this changes the journaled step sequence of RunDeployBillingPush and RunDeployBillingClose. Drain in-flight deploy-billing invocations before deploying, or old journals may fail replay.

feat(ctrl): month-end invoice closing webhook for Deploy billing Closes the last correctness gap in the Deploy money path: without this, the billed month-end total was whatever the last hourly push happened to deliver, losing up to an hour of usage at every period roll. New pkg/webhook: a small receiver framework for inbound provider webhooks (method/body policing, signature verification via pluggable verifiers, event-type routing, middleware for integration-specific concerns, and transport metrics so webhook integrations never fly blind): - unkey_webhook_events_total{provider,event,outcome} with outcomes handled / ignored / unhandled / error / verification_failed / bad_request, and unkey_webhook_handler_duration_seconds. Defined in the package itself, next to the code that emits them. - Handlers return nil (200), ErrIgnore (200, counted separately), or an error (500, provider retries) — retry semantics are explicit. - Verifiers live under pkg/webhook/verifiers/ (stripe today; the GitHub webhook can migrate later). Flow: - ctrl-api gains POST /webhooks/stripe (new StripeConfig). On invoice.created for a Deploy workspace renewal invoice it claims the draft (auto_advance off) and dispatches RunDeployBillingClose for the closed period via Restate ingress; the per-customer invoice storm at the roll dedupes onto one durable invocation through the idempotency key. Everything else is ignored and keeps Stripe's own finalization. - RunDeployBillingClose (new CronService handler, keyed by the CLOSED "YYYY-MM" period) pushes every billable workspace's final full-period usage, stamped one second before period end so the "last"-formula meters bill the complete month, then finalizes Deploy workspaces' draft renewal invoices (new internal/invoicecloser), fanned out in bounded batches with one journaled step per workspace. Refuses to close a period that has not ended; losing a finalize race counts as done. - A 01:05 UTC backup cron invokes the same handler with the same idempotency key: if the webhook ran it attaches to that invocation, otherwise drafts are still drafts (auto_advance was disabled) and the sweep closes them. Total ctrl outage degrades to Stripe's one-hour auto-finalization. Prerequisite correctness fix: subscription billing anchors now pin midnight UTC (hour/minute/second 0), not just day_of_month 1. Without it periods are not calendar months and the renewal invoice's meter window would swallow the next month's early month-to-date events, billing the new month's tiny running total instead of the closed month's final one. The hourly push and the close share one pipeline (pushUsage) with an explicit window end and meter-event timestamp; billingperiod gains End(). ENG-2865

2026-07-01

feat(api): add app delete endpoint (#6502) * feat(api): add app delete endpoint * fix: accept both slug and id * fix: correlation id and retryable auditlogs * fix: permissions and tests * chore: tidy

Merge branch 'main' into perkinsjr/eng-2979-allow-configuring-sentinel-policy-rate

2026-06-30

feat(ctrl): Deploy spend-cap enforcement suspend/resume (ENG-2923)

feat(ctrl): rewire Deploy cancel through ctrl-api (ENG-2922)

feat(ctrl): Deploy teardown service (ENG-2922) Adds the DeployTeardownService virtual object: it snapshots a workspace's running deployments, clears apps.current_deployment_id, and schedules archive or standby with drain polling. SUSPEND records the running set so a later resume can restore it. Also removes svc/ctrl/integration/sync_test.go, which stopped compiling after the harness changes and is not re-added here.

feat(billing): Deploy spend-cap detection and budget alerts (ENG-2904) deploy-spend-check cron prices each budgeted workspace's month-to-date Deploy usage locally from ClickHouse, computes net-of-credit overage against the configured budget, and emails the workspace at 50/75/100%. Enforcement (ENG-2923 SUSPEND) is gated on the ENG-2922 teardown primitive and follows. Plan: todo/eng-2904-2923-spend-cap-enforcement.md

refactor(ctrl)!: per-workspace push invocations for deploy billing Not behavior-preserving. The hourly push and month-end close now fan out one Restate invocation per workspace instead of pushing inline, so each push retries and fails in isolation: one workspace's broken Stripe state no longer stalls the others or the tick. The close awaits its pushes and, when any fail, leaves those workspaces' invoices open and returns a TerminalError so the fresh-key backup cron genuinely re-pushes them rather than finalizing an under-billed total.

feat(ctrl): month-end invoice closing webhook for Deploy billing Closes the last correctness gap in the Deploy money path: without this, the billed month-end total was whatever the last hourly push happened to deliver, losing up to an hour of usage at every period roll. New pkg/webhook: a small receiver framework for inbound provider webhooks (method/body policing, signature verification via pluggable verifiers, event-type routing, middleware for integration-specific concerns, and transport metrics so webhook integrations never fly blind): - unkey_webhook_events_total{provider,event,outcome} with outcomes handled / ignored / unhandled / error / verification_failed / bad_request, and unkey_webhook_handler_duration_seconds. Defined in the package itself, next to the code that emits them. - Handlers return nil (200), ErrIgnore (200, counted separately), or an error (500, provider retries) — retry semantics are explicit. - Verifiers live under pkg/webhook/verifiers/ (stripe today; the GitHub webhook can migrate later). Flow: - ctrl-api gains POST /webhooks/stripe (new StripeConfig). On invoice.created for a Deploy workspace renewal invoice it claims the draft (auto_advance off) and dispatches RunDeployBillingClose for the closed period via Restate ingress; the per-customer invoice storm at the roll dedupes onto one durable invocation through the idempotency key. Everything else is ignored and keeps Stripe's own finalization. - RunDeployBillingClose (new CronService handler, keyed by the CLOSED "YYYY-MM" period) pushes every billable workspace's final full-period usage, stamped one second before period end so the "last"-formula meters bill the complete month, then finalizes Deploy workspaces' draft renewal invoices (new internal/invoicecloser), fanned out in bounded batches with one journaled step per workspace. Refuses to close a period that has not ended; losing a finalize race counts as done. - A 01:05 UTC backup cron invokes the same handler with the same idempotency key: if the webhook ran it attaches to that invocation, otherwise drafts are still drafts (auto_advance was disabled) and the sweep closes them. Total ctrl outage degrades to Stripe's one-hour auto-finalization. Prerequisite correctness fix: subscription billing anchors now pin midnight UTC (hour/minute/second 0), not just day_of_month 1. Without it periods are not calendar months and the renewal invoice's meter window would swallow the next month's early month-to-date events, billing the new month's tiny running total instead of the closed month's final one. The hourly push and the close share one pipeline (pushUsage) with an explicit window end and meter-event timestamp; billingperiod gains End(). ENG-2865

fix: correlation id and retryable auditlogs

feat(api): add app delete endpoint

feat: Allow custom build commands for railpack (#6583) * Railpack config for builds and installs * fix: remove install * Fix: Remove the install error

fix: remove install

fix: correlation id and retryable auditlogs

feat(api): add app delete endpoint

2026-06-29

Railpack config for builds and installs

feat(ctrl): Deploy spend-cap enforcement suspend/resume (ENG-2923)

feat(ctrl): Deploy teardown service (ENG-2922) Adds the DeployTeardownService virtual object: it snapshots a workspace's running deployments, clears apps.current_deployment_id, and schedules archive or standby with drain polling. SUSPEND records the running set so a later resume can restore it. Also removes svc/ctrl/integration/sync_test.go, which stopped compiling after the harness changes and is not re-added here.

feat(ctrl): rewire Deploy cancel through ctrl-api (ENG-2922)

feat(billing): Deploy spend-cap detection and budget alerts (ENG-2904) deploy-spend-check cron prices each budgeted workspace's month-to-date Deploy usage locally from ClickHouse, computes net-of-credit overage against the configured budget, and emails the workspace at 50/75/100%. Enforcement (ENG-2923 SUSPEND) is gated on the ENG-2922 teardown primitive and follows. Plan: todo/eng-2904-2923-spend-cap-enforcement.md

refactor(ctrl)!: per-workspace push invocations for deploy billing Not behavior-preserving. The hourly push and month-end close now fan out one Restate invocation per workspace instead of pushing inline, so each push retries and fails in isolation: one workspace's broken Stripe state no longer stalls the others or the tick. The close awaits its pushes and, when any fail, leaves those workspaces' invoices open and returns a TerminalError so the fresh-key backup cron genuinely re-pushes them rather than finalizing an under-billed total.

feat(ctrl): month-end invoice closing webhook for Deploy billing Closes the last correctness gap in the Deploy money path: without this, the billed month-end total was whatever the last hourly push happened to deliver, losing up to an hour of usage at every period roll. New pkg/webhook: a small receiver framework for inbound provider webhooks (method/body policing, signature verification via pluggable verifiers, event-type routing, middleware for integration-specific concerns, and transport metrics so webhook integrations never fly blind): - unkey_webhook_events_total{provider,event,outcome} with outcomes handled / ignored / unhandled / error / verification_failed / bad_request, and unkey_webhook_handler_duration_seconds. Defined in the package itself, next to the code that emits them. - Handlers return nil (200), ErrIgnore (200, counted separately), or an error (500, provider retries) — retry semantics are explicit. - Verifiers live under pkg/webhook/verifiers/ (stripe today; the GitHub webhook can migrate later). Flow: - ctrl-api gains POST /webhooks/stripe (new StripeConfig). On invoice.created for a Deploy workspace renewal invoice it claims the draft (auto_advance off) and dispatches RunDeployBillingClose for the closed period via Restate ingress; the per-customer invoice storm at the roll dedupes onto one durable invocation through the idempotency key. Everything else is ignored and keeps Stripe's own finalization. - RunDeployBillingClose (new CronService handler, keyed by the CLOSED "YYYY-MM" period) pushes every billable workspace's final full-period usage, stamped one second before period end so the "last"-formula meters bill the complete month, then finalizes Deploy workspaces' draft renewal invoices (new internal/invoicecloser), fanned out in bounded batches with one journaled step per workspace. Refuses to close a period that has not ended; losing a finalize race counts as done. - A 01:05 UTC backup cron invokes the same handler with the same idempotency key: if the webhook ran it attaches to that invocation, otherwise drafts are still drafts (auto_advance was disabled) and the sweep closes them. Total ctrl outage degrades to Stripe's one-hour auto-finalization. Prerequisite correctness fix: subscription billing anchors now pin midnight UTC (hour/minute/second 0), not just day_of_month 1. Without it periods are not calendar months and the renewal invoice's meter window would swallow the next month's early month-to-date events, billing the new month's tiny running total instead of the closed month's final one. The hourly push and the close share one pipeline (pushUsage) with an explicit window end and meter-event timestamp; billingperiod gains End(). ENG-2865

fix: correlation id and retryable auditlogs

feat(api): add app delete endpoint

2026-06-26

feat(ctrl): Deploy spend-cap enforcement suspend/resume (ENG-2923)

feat(ctrl): rewire Deploy cancel through ctrl-api (ENG-2922)

feat(ctrl,frontline): Deploy teardown service + offline 503 (ENG-2922)

feat(billing): Deploy spend-cap detection and budget alerts (ENG-2904) deploy-spend-check cron prices each budgeted workspace's month-to-date Deploy usage locally from ClickHouse, computes net-of-credit overage against the configured budget, and emails the workspace at 50/75/100%. Enforcement (ENG-2923 SUSPEND) is gated on the ENG-2922 teardown primitive and follows. Plan: todo/eng-2904-2923-spend-cap-enforcement.md

refactor(ctrl): per-workspace push invocations for deploy billing Replace the journaled-batch push (one big invocation, per-workspace failures recorded as a bool) with a fan-out to DeployBillingPushService: the orchestrator resolves billable workspaces and dispatches one push invocation per workspace, keyed by workspace id. Each push retries and fails in isolation as its own visible invocation and returns a real error. Hourly orchestrator fires-and-forgets; the month-end close awaits the pushes so the closing invoice sees the final total before finalizing.

feat(ctrl): month-end invoice closing webhook for Deploy billing Closes the last correctness gap in the Deploy money path: without this, the billed month-end total was whatever the last hourly push happened to deliver, losing up to an hour of usage at every period roll. New pkg/webhook: a small receiver framework for inbound provider webhooks (method/body policing, signature verification via pluggable verifiers, event-type routing, middleware for integration-specific concerns, and transport metrics so webhook integrations never fly blind): - unkey_webhook_events_total{provider,event,outcome} with outcomes handled / ignored / unhandled / error / verification_failed / bad_request, and unkey_webhook_handler_duration_seconds. Defined in the package itself, next to the code that emits them. - Handlers return nil (200), ErrIgnore (200, counted separately), or an error (500, provider retries) — retry semantics are explicit. - Verifiers live under pkg/webhook/verifiers/ (stripe today; the GitHub webhook can migrate later). Flow: - ctrl-api gains POST /webhooks/stripe (new StripeConfig). On invoice.created for a Deploy workspace renewal invoice it claims the draft (auto_advance off) and dispatches RunDeployBillingClose for the closed period via Restate ingress; the per-customer invoice storm at the roll dedupes onto one durable invocation through the idempotency key. Everything else is ignored and keeps Stripe's own finalization. - RunDeployBillingClose (new CronService handler, keyed by the CLOSED "YYYY-MM" period) pushes every billable workspace's final full-period usage, stamped one second before period end so the "last"-formula meters bill the complete month, then finalizes Deploy workspaces' draft renewal invoices (new internal/invoicecloser), fanned out in bounded batches with one journaled step per workspace. Refuses to close a period that has not ended; losing a finalize race counts as done. - A 01:05 UTC backup cron invokes the same handler with the same idempotency key: if the webhook ran it attaches to that invocation, otherwise drafts are still drafts (auto_advance was disabled) and the sweep closes them. Total ctrl outage degrades to Stripe's one-hour auto-finalization. Prerequisite correctness fix: subscription billing anchors now pin midnight UTC (hour/minute/second 0), not just day_of_month 1. Without it periods are not calendar months and the renewal invoice's meter window would swallow the next month's early month-to-date events, billing the new month's tiny running total instead of the closed month's final one. The hourly push and the close share one pipeline (pushUsage) with an explicit window end and meter-event timestamp; billingperiod gains End(). ENG-2865

2026-06-25

feat: allow ratelimit configuration for keys

fix: correlation id and retryable auditlogs

feat(api): add app delete endpoint

feat(api): add app create endpoint (#6489) * feat(dashboard): add app create * refactor: revert rbac coz ui doesn't know how to handle it * refactor: move audit to ctrl api * fix: use actor * refactor: use projectId instead of projectSlug * fix: permission explanation we are still using the old convention * refactor: use project slug instead of projectId * fix: create app accept both project id and slug * fix: adjust query name * fix: openapi schema * chore: tidy permission ui * fix: regex and permission

2026-06-24

feat(ctrl,frontline): Deploy teardown service + offline 503 (ENG-2922)

feat(billing): Deploy spend-cap detection and budget alerts (ENG-2904) deploy-spend-check cron prices each budgeted workspace's month-to-date Deploy usage locally from ClickHouse, computes net-of-credit overage against the configured budget, and emails the workspace at 50/75/100%. Enforcement (ENG-2923 SUSPEND) is gated on the ENG-2922 teardown primitive and follows. Plan: todo/eng-2904-2923-spend-cap-enforcement.md

feat(ctrl): Deploy spend-cap enforcement suspend/resume (ENG-2923)

feat(ctrl): rewire Deploy cancel through ctrl-api (ENG-2922)

refactor(ctrl): per-workspace push invocations for deploy billing Replace the journaled-batch push (one big invocation, per-workspace failures recorded as a bool) with a fan-out to DeployBillingPushService: the orchestrator resolves billable workspaces and dispatches one push invocation per workspace, keyed by workspace id. Each push retries and fails in isolation as its own visible invocation and returns a real error. Hourly orchestrator fires-and-forgets; the month-end close awaits the pushes so the closing invoice sees the final total before finalizing.

feat(ctrl): month-end invoice closing webhook for Deploy billing Closes the last correctness gap in the Deploy money path: without this, the billed month-end total was whatever the last hourly push happened to deliver, losing up to an hour of usage at every period roll. New pkg/webhook: a small receiver framework for inbound provider webhooks (method/body policing, signature verification via pluggable verifiers, event-type routing, middleware for integration-specific concerns, and transport metrics so webhook integrations never fly blind): - unkey_webhook_events_total{provider,event,outcome} with outcomes handled / ignored / unhandled / error / verification_failed / bad_request, and unkey_webhook_handler_duration_seconds. Defined in the package itself, next to the code that emits them. - Handlers return nil (200), ErrIgnore (200, counted separately), or an error (500, provider retries) — retry semantics are explicit. - Verifiers live under pkg/webhook/verifiers/ (stripe today; the GitHub webhook can migrate later). Flow: - ctrl-api gains POST /webhooks/stripe (new StripeConfig). On invoice.created for a Deploy workspace renewal invoice it claims the draft (auto_advance off) and dispatches RunDeployBillingClose for the closed period via Restate ingress; the per-customer invoice storm at the roll dedupes onto one durable invocation through the idempotency key. Everything else is ignored and keeps Stripe's own finalization. - RunDeployBillingClose (new CronService handler, keyed by the CLOSED "YYYY-MM" period) pushes every billable workspace's final full-period usage, stamped one second before period end so the "last"-formula meters bill the complete month, then finalizes Deploy workspaces' draft renewal invoices (new internal/invoicecloser), fanned out in bounded batches with one journaled step per workspace. Refuses to close a period that has not ended; losing a finalize race counts as done. - A 01:05 UTC backup cron invokes the same handler with the same idempotency key: if the webhook ran it attaches to that invocation, otherwise drafts are still drafts (auto_advance was disabled) and the sweep closes them. Total ctrl outage degrades to Stripe's one-hour auto-finalization. Prerequisite correctness fix: subscription billing anchors now pin midnight UTC (hour/minute/second 0), not just day_of_month 1. Without it periods are not calendar months and the renewal invoice's meter window would swallow the next month's early month-to-date events, billing the new month's tiny running total instead of the closed month's final one. The hourly push and the close share one pipeline (pushUsage) with an explicit window end and meter-event timestamp; billingperiod gains End(). ENG-2865

refactor: move audit to ctrl api

fix: correlation id and retryable auditlogs

feat(api): add app delete endpoint

refactor: move audit to ctrl api

feat: stop/start and autoscale previews to 0 (#6488) * feat: stop/start and autoscale previews to 0 * clean up * fix: claude hallucinations * fix: address oz comments * merge

2026-06-23

fix: Clickhouse user lock down

2026-06-22

feat(billing): Deploy spend-cap detection and budget alerts (ENG-2904) deploy-spend-check cron prices each budgeted workspace's month-to-date Deploy usage locally from ClickHouse, computes net-of-credit overage against the configured budget, and emails the workspace at 50/75/100%. Enforcement (ENG-2923 SUSPEND) is gated on the ENG-2922 teardown primitive and follows. Plan: todo/eng-2904-2923-spend-cap-enforcement.md

refactor(ctrl): per-workspace push invocations for deploy billing Replace the journaled-batch push (one big invocation, per-workspace failures recorded as a bool) with a fan-out to DeployBillingPushService: the orchestrator resolves billable workspaces and dispatches one push invocation per workspace, keyed by workspace id. Each push retries and fails in isolation as its own visible invocation and returns a real error. Hourly orchestrator fires-and-forgets; the month-end close awaits the pushes so the closing invoice sees the final total before finalizing.

feat(ctrl): month-end invoice closing webhook for Deploy billing Closes the last correctness gap in the Deploy money path: without this, the billed month-end total was whatever the last hourly push happened to deliver, losing up to an hour of usage at every period roll. New pkg/webhook: a small receiver framework for inbound provider webhooks (method/body policing, signature verification via pluggable verifiers, event-type routing, middleware for integration-specific concerns, and transport metrics so webhook integrations never fly blind): - unkey_webhook_events_total{provider,event,outcome} with outcomes handled / ignored / unhandled / error / verification_failed / bad_request, and unkey_webhook_handler_duration_seconds. Defined in the package itself, next to the code that emits them. - Handlers return nil (200), ErrIgnore (200, counted separately), or an error (500, provider retries) — retry semantics are explicit. - Verifiers live under pkg/webhook/verifiers/ (stripe today; the GitHub webhook can migrate later). Flow: - ctrl-api gains POST /webhooks/stripe (new StripeConfig). On invoice.created for a Deploy workspace renewal invoice it claims the draft (auto_advance off) and dispatches RunDeployBillingClose for the closed period via Restate ingress; the per-customer invoice storm at the roll dedupes onto one durable invocation through the idempotency key. Everything else is ignored and keeps Stripe's own finalization. - RunDeployBillingClose (new CronService handler, keyed by the CLOSED "YYYY-MM" period) pushes every billable workspace's final full-period usage, stamped one second before period end so the "last"-formula meters bill the complete month, then finalizes Deploy workspaces' draft renewal invoices (new internal/invoicecloser), fanned out in bounded batches with one journaled step per workspace. Refuses to close a period that has not ended; losing a finalize race counts as done. - A 01:05 UTC backup cron invokes the same handler with the same idempotency key: if the webhook ran it attaches to that invocation, otherwise drafts are still drafts (auto_advance was disabled) and the sweep closes them. Total ctrl outage degrades to Stripe's one-hour auto-finalization. Prerequisite correctness fix: subscription billing anchors now pin midnight UTC (hour/minute/second 0), not just day_of_month 1. Without it periods are not calendar months and the renewal invoice's meter window would swallow the next month's early month-to-date events, billing the new month's tiny running total instead of the closed month's final one. The hourly push and the close share one pipeline (pushUsage) with an explicit window end and meter-event timestamp; billingperiod gains End(). ENG-2865

feat(api): add app delete endpoint

refactor: move audit to ctrl api

2026-06-19

feat(api): add project delete (#6476) * feat(api): add project delete * fix: regex * refactor: move audit to ctrl api * refactor: actor usage * refactor: use id instead of slug

feat(api): add app delete endpoint

refactor: move audit to ctrl api

refactor: move audit to ctrl api

feat(api): add project creation endpoint (#6472) * feat(api): add project creation endpoint * chore: rename query query name was off. it felt like we are querying projects by workspaceSlug and workspaceID * refactor: add projectID to response * refactor: stricter slug * fix: nits * chore: address comments * fix: pullfrog comment * chore: tidy * refactor: allow deleteProtection flag * refactor: call ctrl instead for billing integrations * feat: allow audit in ctrl * chore: fmt * refactor: allow actor util to accept session

feat(billing): Deploy spend-cap detection and budget alerts (ENG-2904) deploy-spend-check cron prices each budgeted workspace's month-to-date Deploy usage locally from ClickHouse, computes net-of-credit overage against the configured budget, and emails the workspace at 50/75/100%. Enforcement (ENG-2923 SUSPEND) is gated on the ENG-2922 teardown primitive and follows. Plan: todo/eng-2904-2923-spend-cap-enforcement.md