These commits are when the Protocol Buffers files have changed: (only the last 100 relevant commits are shown)
| Commit: | 8781f31 | |
|---|---|---|
| Author: | mrkaye97 | |
feat: wire up proto
| Commit: | b286891 | |
|---|---|---|
| Author: | mrkaye97 | |
fix: simplify enums
| Commit: | 8838b55 | |
|---|---|---|
| Author: | mrkaye97 | |
fix: use enum for reasons
| Commit: | 8756316 | |
|---|---|---|
| Author: | mrkaye97 | |
Merge branch 'main' into mk/send-cancellation-reason
| Commit: | 75a8c49 | |
|---|---|---|
| Author: | Julius Park | |
| Committer: | GitHub | |
feat(durable-streams): add optional cursor argument to `reset` (#5335) * initial commit * generate
| Commit: | 7bae7e8 | |
|---|---|---|
| Author: | matt | |
| Committer: | GitHub | |
fix(engine): durable invocation hangs until eviction on stream disconnect (#5307) * fix: bugfix + go test * test: regression test * fix: greptile comment * fix: attempt 2, featuring a proto change * chore: versions, changelogs * fix: greptile comments * chore: gen * chore: gen more
| Commit: | aef20b0 | |
|---|---|---|
| Author: | Julius Park | |
| Committer: | GitHub | |
fix: allow batching messages on publish for durable streams (#5308) * initial commit * forgot files * generate * some more fixes
| Commit: | 2af17f6 | |
|---|---|---|
| Author: | mrkaye97 | |
Merge branch 'main' into mk/fix-durable-reconnect-hang
| Commit: | cb415f9 | |
|---|---|---|
| Author: | mrkaye97 | |
fix: attempt 2, featuring a proto change
| Commit: | 0aadfd1 | |
|---|---|---|
| Author: | abelanger5 | |
| Committer: | GitHub | |
feat: add durable streams to the serverless operator (#5293)
| Commit: | 3ea515e | |
|---|---|---|
| Author: | Alexander Belanger | |
Merge branch 'belanger/ts-core-rest' into belanger/serverless-ts-core
| Commit: | fde8e39 | |
|---|---|---|
| Author: | Alexander Belanger | |
Merge remote-tracking branch 'origin/main' into belanger/serverless-stream-subscribe
| Commit: | 69eacd8 | |
|---|---|---|
| Author: | Alexander Belanger | |
Relay durable stream subscriptions over the serverless socket A serverless invocation socket may now open /v1.V1Streams/Subscribe beside the three run streams: the request is a SubscribeStreamRequest and the frames back carry StreamMessage, both as protojson. A topic names no run, so it is admitted by the tenant-wide filter; the scope note next to allowRun says why nothing cross-tenant is reachable. The engine refusing an unentitled tenant closes the stream with code 7 like every other refusal. The proto comment lists the four procedures; the Go and TypeScript serverless bindings are regenerated, with streams.proto added to the TypeScript set so an endpoint has the message types. The docs gain the WS_MAX_STREAMS row. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK
| Commit: | 79f2e20 | |
|---|---|---|
| Author: | Julius Park | |
| Committer: | GitHub | |
feat(engine): rate limit pause (#5209) * initial commit * sdk changes * some refactors etc * more refactors * get rid of isPaused flag * more refactors * tighten things up * lint stuff * fix refill bug, add godoc * refactor tests * update changelogs to be more detailed * don't use join * advisory lock * batch query * improve performance of the other query as well * fix refill issue * fix migration * add e2e test * local env is super slow, testing in CI * fix test * lint * rename migration * clean up rate limits in test * only trust cached pauses, not cached non-pauses * fix merge
| Commit: | 58cb47d | |
|---|---|---|
| Author: | Julius Park | |
merge
| Commit: | 3d22fcd | |
|---|---|---|
| Author: | abelanger5 | |
| Committer: | GitHub | |
feat: serverless operator (#4912) * feat(engine): gRPC operator service for out-of-process operators Adds v1.OperatorService so an operator can run outside the engine and talk to it over gRPC with a tenant-scoped token: - Register (unary) upserts the operator by (tenant, name, kind = GRPC) and creates the worker for the connection, or resumes a previous worker of the same operator. Registration carries no workflows and no actions. - Listen (bidi) activates the worker for the lifetime of the stream: the first message is start{worker_id}, then heartbeats and action deltas flow in and AssignedAction messages flow out straight from the dispatcher fan-out. Deltas are applied incrementally (at most 1000 ids per message) and the scheduler is notified at most once per second while they arrive. - SendStepActionEvent and DurableTask delegate to the dispatcher; the durable stream is wrapped so the first register_worker message is checked for worker ownership before the dispatcher sees it. - Every RPC after Register, including Listen, carries hatchet-operator-id metadata validated against the token's tenant through a cached lookup. Repository: CreateWorkerOpts.OperatorId flows into the existing CreateWorker query and skips WORKER / WORKER_SLOT metering, since operator workers are excluded from those limit counts. AddWorkerActions / RemoveWorkerActions apply deltas in one transaction under the worker row lock and maintain the action hash as the XOR of per-action sha256 digests, so equal sets hash equal in any order and a delta costs O(chunk). The proto keeps importing the package-less dispatcher.proto for the shared action types (no SDK regeneration). SERVER_GRPC_OPERATORS_ENABLED (default false) gates service registration. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * feat(client): gRPC operator client with streamed action deltas Go client for v1.OperatorService in pkg/client, on the existing reconnecting stream machinery: Connect calls Register and opens Listen with the start message; heartbeats and action deltas go through retrySend so a dead stream reconnects and resumes the worker. AddActions / RemoveActions never block: a flusher coalesces pending ops (an add followed by a remove of an id that was never sent cancels out), chunks them to 1000 ids and sends every 250ms or as soon as a chunk is full. Flush waits for the queue to drain. The session keeps the desired action set and replays it as chunked deltas when a reconnect's Register did not resume the previous worker. PutWorkflow is a thin helper over the admin service that returns the derived action ids for the caller to AddActions. Harness-based e2e suite covering echo, durable memo, deactivation on close, bulk deltas with hash consistency, resume without replay, non-resume replay, and the operator id and worker ownership checks. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * feat(engine): serverless operator schema and repository Adds the v1_serverless_endpoint, v1_serverless_tenant, v1_serverless_process and v1_serverless_lease tables with a SERVERLESS operator kind, the ServerlessRepository (also constructible from a standalone pgxpool like the OLAP repository), data-only encryption constructors, and a shared pgx pool builder for out-of-process binaries. Leases are per (tenant, shard) unit and defer owner liveness to a per process heartbeat row, so holding a unit costs no writes; claims use FOR UPDATE SKIP LOCKED over the unowned and dead-process sets only. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * feat(api): serverless endpoint management API Tenant-scoped REST endpoints to create, list, get, update and delete serverless endpoints, plus per-tenant serverless settings (shard count). Signing secrets are encrypted at rest and never returned; each endpoint receives a unique namespace at creation. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * feat(serverless): operator core, gRPC link and binary Adds pkg/serverlessoperator: (tenant, shard) leasing with a per-process heartbeat row, per-unit registrations against a Link interface, a namespace-prefixed routing cache per tenant, signed healthcheck polling that registers workflows with the engine, signed POST delivery with status mapping, and the hatchet-serverless-operator binary. The gRPC link maps tenants to tokens through a TenantTokenExchange (YAML-backed LocalExchange in v1). Durable delivery and the in-engine link follow. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * feat(serverless): durable task delivery over websockets A durable invocation is one request: the operator dials trigger_url with a signed websocket upgrade, sends the action as the first frame, relays DurableTaskRequest and DurableTaskResponse frames between the endpoint and the engine, and reads the endpoint's final done frame as the terminal outcome. The gRPC link backs the per-invocation channel with the client's DurableTaskListener. The websocket dial goes through the safeclient policy dialer so SSRF hardening applies to upgrades. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * feat(engine): in-engine serverless operator Adds the engine link for the serverless operator core: registrations are operator-backed dispatcher sessions, workflows are put through the admin service, events go straight to the dispatcher, and durable channels use the in-process durable task registration. Enabled with SERVER_SERVERLESS_OPERATOR_ENABLED (default false) and configured with SERVER_SERVERLESS_OPERATOR_* settings; the core runs with the dispatcher id as its process id and stops before the dispatcher drains. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * fix(serverless): fair share counts dead-process units, engine channel orders entries after acks Three defects found by the end-to-end suite: - A survivor already at its fair share never claimed a crashed process's units because they were counted neither as unowned nor as live weight. CountUnowned now counts units held by dead processes too. - A zero durable slot count fell through as "no capacity" instead of the default, so registrations advertised no durable slots. - The in-process durable channel forwarded entry_completed responses in arrival order; on a replay the engine emits a satisfied entry before the endpoint has sent the wait that names it. Entries are now held until the ack carrying their ref has been returned, matching the gRPC listener. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * test(serverless): end-to-end suite and Cloudflare Workers example Harness-based e2e scenarios run against both links (in-engine and gRPC): echo routing by namespace, same-named workflows in one tenant, durable memo with eviction and replay, durable crash retry, healthcheck-driven workflow registration, endpoint deletion, lease failover after a crash, shedding on scale-out, and graceful shutdown. Adds a reference Cloudflare Worker implementing the endpoint contract for the manual smoke test. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * refactor(serverless): adapt to streamed action deltas and session fencing The gRPC operator branch replaced full action set updates with streamed add/remove deltas, dropped PutWorkflow and UpdateWorker from OperatorService, and fenced listener sessions with a session id. The serverless link seam follows: - link.Registration gains AddActions, RemoveActions and Flush; PutWorkflow only puts the workflow and returns its derived action ids. OpenOpts no longer carries workflows: the pollers put them as they learn them and the engine keeps them, so a fresh registration opens with the tenant's action union alone. - grpclink connects without actions, streams the initial union and flushes it before returning, and delegates deltas to the session, which coalesces and chunks them and replays the desired set when a reconnect does not resume the worker. - enginelink creates the worker through Workers().CreateNewWorker with the operator id, activates it under a fresh listener session id that Close deactivates against, applies deltas with AddWorkerActions and RemoveWorkerActions in chunks of 1000, and notifies the scheduler once per delta that changed something. - The core keeps the set each registration last advertised and pushes only the difference on every sync; a failed push leaves the advertised set unchanged so the next sync retries the same delta. The endpoint poller decides the registered_actions write against what it knows the row carries, so a delta the engine refused does not leave the write skipped once the engine accepts it. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * refactor(serverless): drop per-endpoint slot columns Per-endpoint concurrency is going to be governed by namespace-scoped tokens with their own rate limits, so the endpoint rows no longer carry slots and durable_slots and the operator no longer holds a semaphore per endpoint around delivery. Worker slot config stays a process-level setting on the registration. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * feat(serverless): define the endpoint wire contract in protobuf api-contracts/v1/serverless.proto is the single source for every message the serverless operator exchanges with an endpoint: the healthcheck request and response, the non-durable trigger request and its error override, and the durable websocket frame (a oneof of first, request, response, error and done). Messages travel as protojson, so the Go operator and the TypeScript package read and write the same encoding from the same definitions. The Go bindings are generated into internal/services/shared/proto/v1 with the same dual proto path as operator.proto, since both reference .AssignedAction from the package-less dispatcher.proto. The TypeScript bindings live in sdks/typescript-serverless, the future @hatchet-dev/serverless package, which for now holds only the generation script (ts-proto pinned to the SDK's version and options) and its output under src/generated/proto. The SDK's own protoc script skips serverless.proto the way it skips operator.proto. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * feat(serverless): switch the operator to the generated contract messages The operator encodes and decodes every endpoint message with protojson through the generated v1.Serverless* types; pkg/serverlessoperator/contract keeps the header names, the upgrade signature payload, wire constants and the protojson helpers, and pkg/serverlessoperator/durable keeps the relay and its close codes. On the wire this means lowerCamelCase field names (endpointId, invocationCount, inlineWaitBudgetMs, sdkVersion), the first durable frame wrapped as {"first": ...}, and the done frame's output carried as a JSON string so any JSON value reaches the engine byte for byte. A done frame whose output is not JSON is a protocol violation and closes the socket with 4005. The Cloudflare Workers example mirrors the new shapes by hand, as it does for the rest of the contract until the package replaces it. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * ci(serverless): regenerate the TypeScript contract bindings in the generate job generate-all runs generate-serverless-proto, which installs the package's pinned toolchain and regenerates sdks/typescript-serverless/src/generated/proto, so the generate job's diff check fails when the committed bindings are stale. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * fix(dispatcher): keep operator stream sends serialised and drains unblockable The shutdown drain ranges the session map and did a blocking send on each stream session's fin channel. A Listen handler that exited concurrently released its session and stopped selecting on fin, so a pointer the drain had already captured blocked it forever. Operator stream sessions now carry a done channel that Release closes, and the drain selects on fin or done through requestFin, so a released session is skipped. Sends on one stream are serialised by the per-worker send lock, but the lock was released by the caller's deferred Release as soon as its context ended, while the SendMsg goroutine was still blocked by flow control. The next caller then started a second SendMsg on the same stream, which gRPC forbids. The lock is now released by the send goroutine once SendMsg exits; a caller whose context ends returns without the lock, and the next caller fails fast with errFlowControlActive after the lock timeout instead of overlapping. AddOperatorStreamSession returns an OperatorStreamSession handle with Fin, Send and Release. Send shares the send serialisation with the action fan-out so an operator service can write its own protocol messages on the stream, and the optional wrap function converts each assigned action into the stream's server message type before it is encoded. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * fix(grpcoperator): acknowledge action deltas and retain them until committed The Listen protocol had no delta acknowledgement. A delta the stream had accepted but the engine had not committed (stream ended before the apply, database error, rejected chunk) was dropped by the client as soon as the send returned, and a resumed worker never learned about it: the client only replayed its desired set for a fresh worker, and repeating the same AddActions was a no-op against the desired set. Listen now returns a stream of OperatorListenResponse, a oneof of the dispatcher's AssignedAction and an OperatorActionsAck. Each delta carries a positive, strictly increasing sequence; the engine applies deltas in stream order and acks a sequence once the change is committed, and an ack for N confirms every lower sequence. A delta that cannot be applied, or whose ack cannot be written, ends the stream instead of leaving it unconfirmed. The dispatcher session wraps actions into the envelope and writes acks through the same per-stream send serialisation as the fan-out. The client keeps every sent chunk until its ack arrives and replays the unacked chunks in order on every reconnect. When the reconnect did not resume the worker it replaces the queue with a snapshot of the desired set, taken under the same lock AddActions and RemoveActions use, so a removal that races the replay is queued relative to the snapshot instead of being coalesced away with an add that was never sent. Replay runs as the reconnecting stream's replay callback, under sendMu before the stream is published, so no chunk is sent in between. A chunk that waits longer than the ack timeout hangs the stream up so the reconnect replays it. Flush now waits for the acks, so a nil Flush means the engine committed the set. Because Flush is called before Actions by the serverless link, the session runs its receive and heartbeat loops from connect until Close; the receive loop demultiplexes acks and actions, buffering actions in an inbox that the consumer Actions attaches drains. Starting the loops and attaching the consumer add to the WaitGroup under the mutex Close takes, which removes the Actions/Close race on the WaitGroup; Actions after Close is refused. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * fix(repository): hash worker action sets canonically and check the tenant on links The worker action hash was the XOR of per-action digests seeded from whatever the row held. A worker created with an initial set stored the sha256 of the ordered list while a worker built by deltas stored the XOR fold, so equal sets hashed differently across the two paths, and the XOR fold is linear, so a chosen set of ids could be solved to collide with any target hash. GetWorkerActionsByWorkerActionHash shares one worker's action list with every worker of the same hash, so both properties matter. hashActions is now the canonical digest: sha256 over the lower-cased, deduplicated ids sorted by byte order, each followed by ";". The delta transactions recompute the same digest in SQL from the linked rows (ComputeWorkerActionHash, sha256 over string_agg ordered with the C collation) while holding the worker's row lock, so the stored hash is always the digest of the links the transaction leaves behind. UpdateWorker, which only adds links, recomputes it the same way. At 100,000 linked actions one add chunk of 1,000 costs about 85 ms, a no-op re-add about 12 ms and a remove about 65 ms on the review database. Deltas no longer upsert the shared Action rows: existing rows are read, the missing ones are inserted in sorted order with ON CONFLICT DO NOTHING, and the resulting row ids are linked in sorted order. Concurrent replays of the same set by different workers therefore take no locks on existing Action rows, write no new row versions, and cannot deadlock. The row lock, the link and the unlink statements all carry the tenant: LockWorkerActionHash finds no row for a worker of another tenant, which is reported as an error wrapping pgx.ErrNoRows before anything is mutated, and the link and unlink statements join Worker and Action on the tenant so a cross-tenant pair affects zero rows. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * fix(grpcoperator): refuse a second register_worker on a DurableTask stream The durable stream wrapper checked worker ownership on the first register_worker only and passed every later message through, so a second register_worker naming another operator's worker in the same tenant reached the dispatcher, which rebinds the named worker's durable dispatcher without an operator check. The SDK's durable listener registers exactly once per stream and opens a new stream to register again, so a second register_worker is a protocol error: the wrapper now refuses it with InvalidArgument, whichever worker it names, and the stream stays bound to the worker it was authorized for. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * fix(grpcoperator): cap Listen streams and action links per operator Public gRPC admission had no cumulative limit: an authenticated tenant token could open any number of Listen streams for one operator, each a live worker exempt from the worker and slot metering, and stream deltas that grow the operator's action links without bound. The service now enforces two per-operator caps, refused with ResourceExhausted. WithMaxListenStreamsPerOperator bounds the Listen streams one operator holds open on this replica (default 100); a stream over the cap is refused before its worker is activated and the slot is returned when the stream ends. WithMaxActionsPerOperator bounds the action links held across all workers of the operator (default 1,000,000). The budget is read once per stream with CountOperatorWorkerActions and then tracked from the deltas the stream applies; the repository enforces it on the links a delta would newly create, inside the delta's transaction, so a delta over the budget links nothing and a delta that repeats actions the worker already holds (a replay after a resumed reconnect) never counts. Zero disables either cap. The caps are options on the service. Wiring them to server configuration (runtime.grpcOperatorMaxListenStreamsPerOperator and runtime.grpcOperatorMaxActionsPerOperator with env bindings, next to grpcOperatorsEnabled) is a change to pkg/config/server and cmd/hatchet-engine outside this branch's scope. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * fix(serverless): resolve durable relay outcomes through a phase table The relay recorded eviction acks and engine errors as independent flags that only the closed-without-done path consulted. A request timeout, missed pongs, backpressure, a link failure or a protocol violation after an acknowledged eviction turned into a retryable FAILED event for an invocation the engine had already evicted, and a done frame with output after an engine error revived a permanently failed invocation (correctness F10). A done {"status": "evicted"} sent without any eviction ack was accepted as an eviction, so the terminal event was suppressed for a task the engine still considered running (correctness F11). The relay now tracks a phase (running, eviction acked, engine errored) that moves forward once, on the pump goroutine, before the frame announcing it is queued, and every way the relay can end is an exit value. finish resolves (phase, exit) through one table in outcome.go: - after an eviction ack, every exit is the evicted outcome (no terminal event) except an explicit done with an error, which is reported as sent; - after an engine error, every exit is failed non-retryable with the engine's message; - a server_evict and an engine cancel win in every phase; - done evicted while running is a protocol violation: close 4005 and failed retryable, so the task gets a terminal state. Close codes stay the exit's own. The engine-link failure detail moves to the operator log; the task error and the close reason say "engine link failed". TestResolveTable covers every (phase, exit) pair and the two reproductions from the audit are regression tests with a goleak check on the relay lifecycle. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * fix(serverless): bound the durable upgrade response headers gorilla parses the upgrade response with http.ReadResponse on the raw connection, outside any net/http header budget, and the relay's frame limit only applies once the websocket exists. An endpoint could therefore make the operator allocate an arbitrary amount of memory per handshake (security F01). The dialer now installs a headerLimitConn under both NetDialContext (ws, development only) and NetDialTLSContext (wss). For wss the TLS session is negotiated here with the URL's host name, so the wrapper sits on the decrypted stream and budgets exactly the bytes the handshake parser sees, up to and including the blank line that ends the response head; frames after that are not counted. Crossing the budget fails the dial with ErrUpgradeHeadersTooLarge, which the relay reports as a non-retryable failure naming the limit. The limit is Params.MaxUpgradeHeaderBytes, 64 KiB by default, exposed as Config.WSMaxUpgradeHeaderBytes, SERVERLESS_OPERATOR_WS_MAX_UPGRADE_HEADER_BYTES on the binary and SERVER_SERVERLESS_OPERATOR_WS_MAX_UPGRADE_HEADER_BYTES on the engine. The regression test answers the upgrade with a 16 MiB header over plain and TLS sockets. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * fix(serverless): budget the durable relay's queued bytes The send queue held up to 256 encoded frames with no bound on their size, so a slow endpoint with large engine responses could pin hundreds of megabytes per relay (performance F11). The relay now charges every frame against Params.MaxQueuedBytes before queueing it and releases the charge once the writer has taken it off the queue; crossing the budget ends the relay with the existing CloseBackpressure code, the frame count limit stays as it was. The budget is 16 MiB by default and configurable as Config.WSMaxQueuedBytes, SERVERLESS_OPERATOR_WS_MAX_QUEUED_BYTES on the binary and SERVER_SERVERLESS_OPERATOR_WS_MAX_QUEUED_BYTES on the engine. The regression test parks the writer and queues eight 16 KiB frames against a 64 KiB budget. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * fix(safeclient): bound idle connections, stop at the response cap, redact transport errors Three findings share this package. Idle pool (correctness F19, performance F7): both transports were zero literals, so MaxIdleConns and IdleConnTimeout were unbounded and one operator process talking to an ever-growing set of origins kept a socket per origin until the far side closed it. The transport now takes Config.MaxIdleConns (256), MaxIdleConnsPerHost (4) and IdleConnTimeout (90s), and Sender.CloseIdleConnections releases the pool; the binary binds SERVERLESS_OPERATOR_HTTP_MAX_IDLE_CONNS, _PER_HOST and _IDLE_CONN_TIMEOUT and closes the pool on exit. The engine's server config carries the same three settings. safeurl clones the transport it is given, so the Sender keeps the clone for CloseIdleConnections. Response cap (correctness F20, security F11): an oversize response was drained to EOF before ErrResponseTooLarge was returned, so a streaming endpoint held the delivery slot until the caller's deadline. Deliver now reads one byte past the cap and returns; the deferred body close drops the connection instead of draining it. An 8-byte cap consumes at most 9 bytes and a stalled 1025-byte stream against a 1024-byte cap returns immediately. Redaction (security F12): dial, DNS and read failures were returned with the resolved address, port, path and socket error, and those strings end up in task errors and endpoint status. Transport failures are now returned as *EndpointError, which names the configured host and the stage (resolve, connect, tls, read, request) and, for certificate verification failures, the x509 message the tenant needs; the full error goes to the operator log. errors.Is and errors.As still reach the cause. A blocked resolution no longer names the address it resolved to. The durable dialer applies PublicError to its own TLS handshake errors. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * fix(serverless): confine nested durable references to the endpoint namespace The relay stamped the task id and invocation on every nested request but forwarded the workflow names in trigger_runs and the user event keys in wait_for conditions as the endpoint sent them, so a durable task could trigger any workflow and wait for any event of the tenant (security F13). stamp now ends with confine, in the shared relay so both links get it: trigger option names and user event keys are prefixed with the endpoint's namespace through contract.ApplyNamespace, the same rule the operator applies to what it registers (a name that already carries the prefix is left alone). Engine-generated identifiers are untouched: log entry refs, readable data keys and or-group ids label the task's own log, sleep conditions name no resource, event scopes are matched within the namespaced key, and memo keys are private to the task. The relay comment documents this as the confinement boundary. contract.NamespaceSeparator, NamespacePrefix and ApplyNamespace carry the rule; routing.go's prefixName implements the same one and should call ApplyNamespace once that package is touched. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * fix(serverless): make signed-request freshness part of the endpoint contract The reference Worker verified the HMAC on POST bodies but never looked at the timestamp the signature covers, accepted upgrade timestamps from the future, kept no nonce set, and ran whatever task the first frame named regardless of the task the upgrade was signed for (security F07). The contract now states the freshness rule in contract/http.go: the timestamp inside every signed POST body and the upgrade's X-Hatchet-Timestamp must lie within RequestMaxAge (5 minutes) of the endpoint's clock in either direction; UpgradeMaxAge is the same window. Endpoints consume the upgrade nonce after the signature verified and check the first frame's task id and invocation against the verified headers. The Worker example implements it: verifySignedBody (HMAC, then the body's timestamp, then the endpoint id when HATCHET_ENDPOINT_ID is set) replaces the bare HMAC check in both POST handlers; verifyUpgradeSignature rejects future timestamps and consumes the nonce from a bounded, expiring NonceSet (4096 entries per isolate, documented as needing a Durable Object or KV across isolates); DurableClient.assertMatches closes the socket with 1008 when the first frame does not name the verified task and invocation. The README documents the checks and the idempotency key for side effects. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * fix(serverless): decide lease liveness in the claim statement and release units with process rows The claim statement now decides which units are claimable in its own snapshot: units with no owner, plus units whose owner's heartbeat row has expired, and it requires the claimer's own row to be live. A dead-process list read earlier is no longer a claim input, so an owner that heartbeats between another process's look and its claim keeps its units (correctness F04), and an expired or swept process cannot claim until it heartbeats. Deleting a process row, by the periodic sweep or a graceful shutdown, releases whatever the row still owns in the same statement, and the migration releases units whose owner row is already gone, so a swept owner never strands units (correctness F03). Count and claim share one liveness rule. The claim no longer sorts the candidate population: unowned units are walked in key order from a random start key with a wraparound, through a covering partial index, and units of expired processes are walked per dead process through the owner index (performance F2). On the perf session's fixtures a claim of 64 from one million unowned units went from 551 ms with a 47 MB external sort to 2 ms, and a claim among 100k dead-owned units from 115 ms to 2 ms. The per-tick claim budget is the process's share of the claimable units (claimable divided by live processes) bounded by MaxClaimPerTick, taken in statements of at most ClaimBatch units, instead of a fixed 128; the claimable count is windowed at the fleet's budget for one tick. The leaser retries its initial heartbeat with backoff instead of exiting (groundwork for correctness F16), and a heartbeat that lands more than the TTL after the previous one triggers an immediate reconcile so lost units are noticed without waiting for the next tick. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * perf(serverless): reference-counted action union, versioned refresh and one status write per poll The tenant action union is now a reference count per action (how many enabled endpoints advertise it) with a revision and a bounded delta log. An action enters the union on the 0 to 1 transition and leaves it on 1 to 0, so a healthcheck change costs that endpoint's actions and every batch of rows (a load, a refresh, a page of a gained unit) publishes at most one revision; the sorted form is built once per revision on demand. Registrations advertise a revision and catch up from the delta log, falling back to a full diff only when the log no longer reaches back, so an unchanged union costs a sync a revision comparison (performance F1, F5). Loading 3,000 endpoints with 10 actions each went from 19.5 s and 18.3 GB to under the 200 ms and 50 MB budget the regression test holds it to; 100 unchanged syncs of a 100k-action union went from 860 MB of allocation to none. The routing refresh is keyed on the row version, the later of updated_at and status_changed_at, with a (version, id) keyset and a matching expression index instead of a one-second overlap window, so an unchanged tenant reads nothing (performance F4: 609 ms and a 88 MB sort for a 100k-row tenant to 0.06 ms) and status transitions written by another owner surface within a refresh. Status writes return the database's status_changed_at and the cache keeps a local write only while it is newer than the row's, so a takeover sees the previous owner's status and recovers the endpoint (correctness F18). A poll now computes one outcome after both the healthcheck and the catalog application and writes the status once when it differs from the row; an unchanged catalog the engine rejected is retried with backoff instead of re-put on every poll, so a persistently rejected endpoint is one write, not a healthy/unhealthy pair per poll (performance F12). A refused action delta restores the endpoint's previous contribution so a rejected catalog never stays in the shared union, and the per-poller workflow hash memory is pruned to the accepted catalog. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * fix(serverless): one registration per tenant, per-tenant operation locks and attempt-keyed deliveries The runner now keeps one registration per tenant per process, shared by every unit of the tenant the process owns (performance F6); it opens with the first unit, stays while any unit remains and closes with the last. The links open registrations by tenant alone, so the in-engine worker is named per dispatcher and the shard label is gone. The runner's lock covers its maps only. Each tenant's work (gaining and losing units, maintenance, shutdown) runs under the tenant's own operation lock, so a hung engine on one tenant's open, a slow poller shutdown or a database outage no longer stalls other tenants or the lease tick, and maintenance refreshes tenants with bounded parallelism (performance F9). A gained unit whose tenant fails to load is kept pending and retried by every maintenance pass until it is served, so a claimed unit is never left owned but unserved (correctness F02). Healthchecks take a per-tenant slot inside the process-wide limit, first polls of a gained unit are spread out, and the network slots are released before a catalog is applied under its own deadline (performance F8, security F05). In-flight deliveries are keyed by task and attempt (retry count and durable invocation). A finish removes only its own record, a cancel reaches the attempt it names (or every attempt of the task when it names none in flight), and a duplicate assignment of an attempt in flight is ignored (correctness F09). The endpoint's request timeout is applied before the durable channel is opened, so the in-engine handshake is bounded like the rest of the invocation (correctness F15). Every advertised action, derived from a workflow or listed, is validated with the engine's action rules before the namespace prefix, and a healthcheck may advertise at most MaxWorkflowsPerEndpoint workflows and MaxActionsPerEndpoint actions; a catalog over either cap is refused and the endpoint marked with the error (security F04, F05). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * fix(grpclink): bounded durable send queue, joined callback goroutines, closed evicted clients and tenant checks Durable requests reach the shared listener through a bounded per-hub queue and one pump goroutine, since the listener's own enqueue cannot be cancelled; every channel's Send selects on the invocation's close signal instead, so a full queue (an unreachable engine) no longer holds a closing invocation, its relay teardown or the operator's shutdown (correctness F12). A send refused by the close undoes its pending ack and in-flight slot. Every goroutine a channel starts (ack waits, entry waits, eviction acks) is refused once closing begins and joined before the invocation's pending state is dropped from the listener, and an entry wait that finds the channel closed drops its registration again, so no callback can outlive its invocation on the shared listener (correctness F13). A goleak test covers the ordinary lifecycle. A tenant's cached client is closed when it is replaced by a rotated token or released with the tenant's last registration, through the client's Close when it exposes one (correctness F14, performance F13); pkg/client does not expose its gRPC connection yet, which the regression test reports by skipping. Open refuses a registration the engine authenticated as another tenant than the unit's, closing the session before any action is sent, and refuses a token whose tenant claim names another tenant before a client is built (security F09). The local token exchange records the mtime of every referenced token_file and reloads when any of them changes, and logs reload failures when given a logger (security F10). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * fix(enginelink): hold pre-handshake responses and link initial actions in bulk The dispatcher registers a durable task for routing before the register-worker handshake completes, so a completion restored for a resumed invocation can reach the channel before the ack. openDurable now holds such responses for the invocation under the existing ack-before- entry ordering, bounded by handshakeHoldLimit, instead of rejecting them as an invalid handshake (correctness F21, reproduced at the link level with the dispatcher's ordering). Open creates the worker without actions and links the initial set through the same chunked bulk path a streamed delta takes, before the worker is activated, instead of one action upsert per action inside worker creation (performance F3); both links now build the action set from the same empty seed. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * fix(engine): supervise the in-engine serverless operator The engine's serverless goroutine ran the core once: an initial failure (a database unreachable at startup) ended it for the life of the engine, and its stop function could return that stale error ahead of the rest of the cleanup chain (correctness F16). The core is now supervised: a stop the engine did not ask for is logged and the core restarted with backoff (one second doubling to thirty, reset after a long run), the supervisor exposes whether the core is running for a readiness probe, and stop ends the core, waits for it and never reports earlier failures, so the dispatcher cleanup continues past it. The leaser's initial heartbeat retries with backoff on its own as well. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * test(enginelink): keep the receive helper visible under every build tag link_test.go carries no build constraint while the regression file is excluded under the integration and e2e tags, so the helper both use lives in link_test.go; go vet -tags integration compiles again. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * fix(client): close operator clients and cancel durable sends Client gains Close, which closes the gRPC connection the client owns, so a caller that retires a client (the serverless grpclink when a tenant's token rotates or its last registration closes) releases the connection's goroutines instead of leaving them behind (correctness F14, performance F13). grpclink's closeClient calls it directly and the goleak regression test that had to skip while pkg/client exposed no Close now runs. DurableTaskListener.SendRequest takes a context and returns an error: the request queue is bounded, so a caller blocked on an unreachable engine is released when its context ends or the listener is stopped (Stop closes a per-generation stop signal that Start renews), and the listener's own request helpers drop their pending ack when the send is refused. SendMemoCompleted follows the same shape. grpclink's pump hands requests to the listener under a context closeAll cancels, so a stopped registration no longer holds the pump on a full queue (correctness F12). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * perf(grpclink): index durable channels by task for server-evict lookups The hub indexed open channels by (task, invocation) in one map, so a server-evict notice, which names one task, scanned every open channel of the registration under the hub's mutex. Channels are now indexed by task and then invocation: a notice visits the named task's entries only, a task is dropped from the index when its last channel closes, and closeAll walks the nested index. A test opens channels for two tasks and checks that only the named task's invocations at or below the notice receive the server_evict response. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * fix(serverless): wire relay limits through registration and the engine link The relay's upgrade header budget and queued bytes budget were configurable on the core, the binary and the engine, but the registration's durable.Run call never passed them, so the relay always used its built-in defaults. Registration now passes Config.WSMaxUpgradeHeaderBytes and WSMaxQueuedBytes as Params.MaxUpgradeHeaderBytes and MaxQueuedBytes, and the engine link's ConfigFromServer copies both from the server config, so SERVERLESS_OPERATOR_WS_MAX_UPGRADE_HEADER_BYTES and _WS_MAX_QUEUED_BYTES (and their SERVER_SERVERLESS_OPERATOR_ prefixed forms) are live. The in-engine mode's safeclient sender now takes the server config's HTTPMaxIdleConns, HTTPMaxIdleConnsPerHost and HTTPIdleConnTimeout, as the out-of-process binary already did, and the stop function closes the idle pool once the core has drained (correctness F19, performance F7). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * fix(serverless): delegate name prefixing to the contract namespace rule routing.go's prefixName and namespacePrefix carried their own copy of the namespace rule the durable relay applies to nested references. Both now delegate to contract.ApplyNamespace and contract.NamespacePrefix, and ParseNamespace reads contract.NamespaceSeparator, so what the operator registers and what the relay confines an endpoint to are prefixed by one implementation (security F13 follow-up). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * fix(grpcoperator): expose admission caps in server config The per-operator Listen stream cap and action link budget were options on the operator service with no way to set them from configuration. Runtime now carries GRPCOperatorMaxListenStreamsPerOperator (default 100) and GRPCOperatorMaxActionsPerOperator (default 1,000,000), bound to SERVER_GRPC_OPERATOR_MAX_LISTEN_STREAMS_PER_OPERATOR and SERVER_GRPC_OPERATOR_MAX_ACTIONS_PER_OPERATOR next to SERVER_GRPC_OPERATORS_ENABLED, and both engine run paths pass them to grpcoperator.New through WithMaxListenStreamsPerOperator and WithMaxActionsPerOperator. Zero disables either cap (security F06). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * fix(repository): recompute operator worker action hash from links UpdateOperatorWorkerActions stored the digest of the list it was given while only adding links, so a worker that already held links ended with a hash that named a subset of its linked set, and GetWorkerActionsByWorkerActionHash would then share the wrong action list with other workers of that hash. The call now locks the worker row with the tenant in the predicate (a worker of another tenant is refused and nothing is written), lower-cases and dedupes the ids, links through the tenant-checked LinkActionsToWorkerReturning, and recomputes the hash from the linked rows with ComputeWorkerActionHash inside the same transaction, as the worker repository's delta path does (correctness F17, security F02, F08). Tests check that the stored hash is the canonical digest of the final linked set after an additive call and after a call naming a subset, that it equals the delta path's hash for the same set, and that the tenant is checked. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * perf(repository): index workers by tenant and operator CountOperatorWorkerActions joins the action links of every worker of an operator once per Listen stream, finding the workers by (tenantId, operatorId); "Worker" had no index on that pair. v1_0_156 adds Worker_tenantId_operatorId_idx (a plain CREATE INDEX inside the migration's transaction, the migration is unreleased) and the schema mirrors it. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * fix(engine): report the serverless operator's state through readiness The supervised in-engine core exposed Running for a readiness probe but nothing consulted it. health.Health now accepts named readiness checks (AddReadinessCheck), which /ready evaluates after the queue and the repository and logs by name when one fails; /live is unchanged. startServerlessOperator returns the supervisor's Running alongside stop, and both engine run paths register it as the "serverless-operator" check when the health server is on, so a replica whose core is down and waiting to restart reports not ready until the core is back (correctness F16). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * fix(serverless): bind every operator setting on the binary and the engine Four settings the fix branches added to the engine's ServerlessOperatorConfigFile had no environment binding (HEALTHCHECK_TENANT_CONCURRENCY, HEALTHCHECK_APPLY_TIMEOUT, MAX_WORKFLOWS_PER_ENDPOINT, MAX_ACTIONS_PER_ENDPOINT); they are bound under SERVER_SERVERLESS_OPERATOR_ now. MaintenanceConcurrency existed on the core Config only. It is now a setting of the binary (SERVERLESS_OPERATOR_MAINTENANCE_CONCURRENCY) and the engine (SERVER_SERVERLESS_OPERATOR_MAINTENANCE_CONCURRENCY), default 8. The leaser's MaxClaimPerTick existed on lease.Config only; the core Config carries it as LeaseMaxClaimPerTick (default 1024, the leaser's own default) and passes it to the leaser, and both binaries bind it as LEASE_MAX_CLAIM_PER_TICK. ConfigFromServer copies both, and its test now sets and checks every shared setting so a field added to one side without the other fails. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * test(grpcoperator): synchronise the operator store fake fakeOperatorStore counted GetOperatorById calls and served its map with no synchronisation, while TestListenStreamCapIsPerOperator authorizes several Listen streams concurrently through it, so the race detector failed the package intermittently. The counter is atomic and the map is guarded, and the assertions read the counter through Load. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * test(grpclink): replace the token file atomically in the reload test The test rewrote the tenant token file with a truncate-then-write while the exchange polled its mtime every ten milliseconds, so the poller could reload between the two and parse an empty file as a valid mapping with no tenants, which failed the later lookup intermittently. The helper now writes a temporary file and renames it over the path, the way a rotated secret or a mounted file is swapped, which is what the exchange is documented to follow. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * fix(dispatcher): keep operator stream sends serialised and drains unblockable The shutdown drain ranges the session map and did a blocking send on each stream session's fin channel. A Listen handler that exited concurrently released its session and stopped selecting on fin, so a pointer the drain had already captured blocked it forever. Operator stream sessions now carry a done channel that Release closes, and the drain selects on fin or done through requestFin, so a released session is skipped. Sends on one stream are serialised by the per-worker send lock, but the lock was released by the caller's deferred Release as soon as its context ended, while the SendMsg goroutine was still blocked by flow control. The next caller then started a second SendMsg on the same stream, which gRPC forbids. The lock is now released by the send goroutine once SendMsg exits; a caller whose context ends returns without the lock, and the next caller fails fast with errFlowControlActive after the lock timeout instead of overlapping. AddOperatorStreamSession returns an OperatorStreamSession handle with Fin, Send and Release. Send shares the send serialisation with the action fan-out so an operator service can write its own protocol messages on the stream, and the optional wrap function converts each assigned action into the stream's server message type before it is encoded. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * fix(grpcoperator): acknowledge action deltas and retain them until committed The Listen protocol had no delta acknowledgement. A delta the stream had accepted but the engine had not committed (stream ended before the apply, database error, rejected chunk) was dropped by the client as soon as the send returned, and a resumed worker never learned about it: the client only replayed its desired set for a fresh worker, and repeating the same AddActions was a no-op against the desired set. Listen now returns a stream of OperatorListenResponse, a oneof of the dispatcher's AssignedAction and an OperatorActionsAck. Each delta carries a positive, strictly increasing sequence; the engine applies deltas in stream order and acks a sequence once the change is committed, and an ack for N confirms every lower sequence. A delta that cannot be applied, or whose ack cannot be written, ends the stream instead of leaving it unconfirmed. The dispatcher session wraps actions into the envelope and writes acks through the same per-stream send serialisation as the fan-out. The client keeps every sent chunk until its ack arrives and replays the unacked chunks in order on every reconnect. When the reconnect did not resume the worker it replaces the queue with a snapshot of the desired set, taken under the same lock AddActions and RemoveActions use, so a removal that races the replay is queued relative to the snapshot instead of being coalesced away with an add that was never sent. Replay runs as the reconnecting stream's replay callback, under sendMu before the stream is published, so no chunk is sent in between. A chunk that waits longer than the ack timeout hangs the stream up so the reconnect replays it. Flush now waits for the acks, so a nil Flush means the engine committed the set. Because Flush is called before Actions by the serverless link, the session runs its receive and heartbeat loops from connect until Close; the receive loop demultiplexes acks and actions, buffering actions in an inbox that the consumer Actions attaches drains. Starting the loops and attaching the consumer add to the WaitGroup under the mutex Close takes, which removes the Actions/Close race on the WaitGroup; Actions after Close is refused. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * fix(repository): hash worker action sets canonically and check the tenant on links The worker action hash was the XOR of per-action digests seeded from whatever the row held. A worker created with an initial set stored the sha256 of the ordered list while a worker built by deltas stored the XOR fold, so equal sets hashed differently across the two paths, and the XOR fold is linear, so a chosen set of ids could be solved to collide with any target hash. GetWorkerActionsByWorkerActionHash shares one worker's action list with every worker of the same hash, so both properties matter. hashActions is now the canonical digest: sha256 over the lower-cased, deduplicated ids sorted by byte order, each followed by ";". The delta transactions recompute the same digest in SQL from the linked rows (ComputeWorkerActionHash, sha256 over string_agg ordered with the C collation) while holding the worker's row lock, so the stored hash is always the digest of the links the transaction leaves behind. UpdateWorker, which only adds links, recomputes it the same way. At 100,000 linked actions one add chunk of 1,000 costs about 85 ms, a no-op re-add about 12 ms and a remove about 65 ms on the review database. Deltas no longer upsert the shared Action rows: existing rows are read, the missing ones are inserted in sorted order with ON CONFLICT DO NOTHING, and the resulting row ids are linked in sorted order. Concurrent replays of the same set by different workers therefore take no locks on existing Action rows, write no new row versions, and cannot deadlock. The row lock, the link and the unlink statements all carry the tenant: LockWorkerActionHash finds no row for a worker of another tenant, which is reported as an error wrapping pgx.ErrNoRows before anything is mutated, and the link and unlink statements join Worker and Action on the tenant so a cross-tenant pair affects zero rows. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * fix(grpcoperator): refuse a second register_worker on a DurableTask stream The durable stream wrapper checked worker ownership on the first register_worker only and passed every later message through, so a second register_worker naming another operator's worker in the same tenant reached the dispatcher, which rebinds the named worker's durable dispatcher without an operator check. The SDK's durable listener registers exactly once per stream and opens a new stream to register again, so a second register_worker is a protocol error: the wrapper now refuses it with InvalidArgument, whichever worker it names, and the stream stays bound to the worker it was authorized for. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * fix(grpcoperator): cap Listen streams and action links per operator Public gRPC admission had no cumulative limit: an authenticated tenant token could open any number of Listen streams for one operator, each a live worker exempt from the worker and slot metering, and stream deltas that grow the operator's action links without bound. The service now enforces two per-operator caps, refused with ResourceExhausted. WithMaxListenStreamsPerOperator bounds the Listen streams one operator holds open on this replica (default 100); a stream over the cap is refused before its worker is activated and the slot is returned when the stream ends. WithMaxActionsPerOperator bounds the action links held across all workers of the operator (default 1,000,000). The budget is read once per stream with CountOperatorWorkerActions and then tracked from the deltas the stream applies; the repository enforces it on the links a delta would newly create, inside the delta's transaction, so a delta over the budget links nothing and a delta that repeats actions the worker already holds (a replay after a resumed reconnect) never counts. Zero disables either cap. The caps are options on the service. Wiring them to server configuration (runtime.grpcOperatorMaxListenStreamsPerOperator and runtime.grpcOperatorMaxActionsPerOperator with env bindings, next to grpcOperatorsEnabled) is a change to pkg/config/server and cmd/hatchet-engine outside this branch's scope. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * fix(client): close operator clients and cancel durable sends Client gains Close, which closes the gRPC connection the client owns, so a caller that retires a client (the serverless grpclink when a tenant's token rotates or its last registration closes) releases the connection's goroutines instead of leaving them behind (correctness F14, performance F13). grpclink's closeClient calls it directly and the goleak regression test that had to skip while pkg/client exposed no Close now runs. DurableTaskListener.SendRequest takes a context and returns an error: the request queue is bounded, so a caller blocked on an unreachable engine is released when its context ends or the listener is stopped (Stop closes a per-generation stop signal that Start renews), and the listener's own request helpers drop their pending ack when the send is refused. SendMemoCompleted follows the same shape. grpclink's pump hands requests to the listener under a context closeAll cancels, so a stopped registration no longer holds the pump on a full queue (correctness F12). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * fix(grpcoperator): expose admission caps in server config The per-operator Listen stream cap and action link budget were options on the operator service with no way to set them from configuration. Runtime now carries GRPCOperatorMaxListenStreamsPerOperator (default 100) and GRPCOperatorMaxActionsPerOperator (default 1,000,000), bound to SERVER_GRPC_OPERATOR_MAX_LISTEN_STREAMS_PER_OPERATOR and SERVER_GRPC_OPERATOR_MAX_ACTIONS_PER_OPERATOR next to SERVER_GRPC_OPERATORS_ENABLED, and both engine run paths pass them to grpcoperator.New through WithMaxListenStreamsPerOperator and WithMaxActionsPerOperator. Zero disables either cap (security F06). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * fix(repository): recompute operator worker action hash from links UpdateOperatorWorkerActions stored the digest of the list it was given while only adding links, so a worker that already held links ended with a hash that named a subset of its linked set, and GetWorkerActionsByWorkerActionHash would then share the wrong action list with other workers of that hash. The call now locks the worker row with the tenant in the predicate (a worker of another tenant is refused and nothing is written), lower-cases and dedupes the ids, links through the tenant-checked LinkActionsToWorkerReturning, and recomputes the hash from the linked rows with ComputeWorkerActionHash inside the same transaction, as the worker repository's delta path does (correctness F17, security F02, F08). Tests check that the stored hash is the canonical digest of the final linked set after an additive call and after a call naming a subset, that it equals the delta path's hash for the same set, and that the tenant is checked. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * test(grpcoperator): synchronise the operator store fake fakeOperatorStore counted GetOperatorById calls and served its map with no synchronisation, while TestListenStreamCapIsPerOperator authorizes several Listen streams concurrently through it, so the race detector failed the package intermittently. The counter is atomic and the map is guarded, and the assertions read the counter through Load. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * refactor(grpcoperator): extract operatorsvc The gRPC operator service owns logic that is not gRPC's: upserting the operator row, creating or resuming its worker, activating the worker under a listener fence, applying action set deltas within the per-operator budget, and closing the session again. The in-process operator host needs all of it, and the serverless operator's engine link already re-implements it by hand, with the drift that comes from a copy: no resume, no budget, a separate dispatcher session key. internal/services/operatorsvc now holds that logic behind Service, Session and the two deliveries a session can have. A stream delivery encodes assigned actions onto a gRPC stream and counts against the per-operator stream cap; a handler delivery hands them to an operator running in this process and counts against nothing but the action budget, since it holds no stream. Both get one session id that is the dispatcher's session key and the worker row's listener fence, both notify the scheduler through the same throttle, and both close by pause, drain, deactivate. The gRPC Listen handler closes without the pause: its operator pauses itself before hanging up, and a stream that simply drops must leave the worker assignable for the next connection. internal/services/grpcoperator keeps what is genuinely the protocol: the metadata that names the operator, the start handshake, the message loop, the delta acks, and the durable stream it hands to the dispatcher unchanged. The dispatcher gains AddOperatorSession, the handler-backed twin of AddOperatorStreamSession, keyed on the session id its caller chose, and pkg/operator gains ActionHandler, the one method a session needs to deliver to. The test doubles for the stores and the dispatcher move to operatorsvc/operatorsvctest so the service's tests and the handlers' tests drive the same behaviour. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * feat(operatorsvc): durable invocations with ack-before-entry ordering An operator hosted in the engine drives a durable invocation over the dispatcher's channel-backed session rather than the DurableTask stream, and the handshake and ordering that goes with it is not obvious enough to leave to each host. Session.OpenDurable performs the register_worker handshake itself, so Recv only ever returns invocation traffic, and applies the ordering the operator expects: the engine delivers a completion for an already satisfied entry as soon as the invocation registers, which on a replay is before the operator has sent the wait_for that names it, so an entry is held until the ack carrying its ref has gone out. Responses that arrive during the handshake are held under the same rule, bounded at 256 so an engine that never acks cannot make the session buffer without end. As on the stream, one ack-bearing request is in flight at a time, because the engine keys its pending state by (task, invocation). This is a port of the serverless operator's engine link, which grew the ordering rule as a bug fix and will be deleted in favour of this once the in-process host lands. The gRPC DurableTask handler is unchanged: it still hands the raw stream to the dispatcher after the ownership check. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * feat(grpcoperator): PauseWorker An operator that wants to shut down cleanly needs the scheduler to stop assigning to its worker before it stops answering, and it needs to know the pause is committed before it starts draining. A message on the Listen stream would need an ack of its own and would not work while that stream is being torn down or reconnected, so pausing is its own unary call. OperatorService.PauseWorker takes {worker_id, paused}: the same call pauses and unpauses, which is what a worker that is resumed rather than replaced needs. It is authorised like SendStepActionEvent, by the operator metadata plus worker ownership. Register already clears the pause when it resumes a worker, so an operator that paused to drain and then crashed comes back assignable. The proto is unreleased, so this is an addition with no compatibility shim. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * feat(client): pause-then-drain on session close Closing an operator session used to hang the Listen stream up at once, which deactivates the worker while the actions the consumer is still working on are in flight; the engine only recovers those when the tasks time out. Close is now pause then drain: it pauses the worker through PauseWorker, so the scheduler stops assigning, waits for the task runs already handed to the consumer to be reported, and only then flushes the deltas and ends the stream. The session learns what is in flight from the contract it already has with the caller: a task run is outstanding from the moment it is handed to the Actions channel until the caller reports it completed, failed or cancelled. Cancels carry no work of their own and are never reported, so they are not tracked. The wait is bounded (30 seconds by default, WithDrainTimeout to change it) and WithoutDrain keeps the previous immediate hang-up, which is what a process that is going away anyway wants. Pause and Resume are also on the session, for a caller that drains on its own schedule rather than at close. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * fix(client): count an action in flight before handing it over Two ways the drain on Close could wait for reports that were never coming. An action was recorded as in flight after the send on the Actions channel returned, so a consumer that reported the outcome the instant it took the action cleared an entry that had not been added yet; the entry added afterwards was never removed and every later Close waited out the whole drain timeout. The action is now recorded before the hand-over, and taken back when the hand-over does not happen. The other way is a consumer that walks away: it cancels the context it passed to Actions, and the runs it was already holding will never be reported. Ending delivery now forgets them, so Close hangs up instead of waiting; the engine retries those tasks when they time out, which is what happened to them before this branch too. Also: report the drain timeout only when something really is outstanding, keep the operator name on the Listen handler's log lines, and say plainly that the notifier may publish one notification after stop. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * feat(operator): host contract Add the hosting contract to pkg/operator: Identity (tenant plus an existing operator row or a name and kind to upsert), OpenOpts, Host, Session, Registration, DurableChannel and the sentinel errors. It is what an operator needs from the engine whichever process it runs in; the engine-internal types (Operator, TaskEventWriter, SharedOperator) stay in operator.go and are documented as such. operatorsvc grows what the in-process host needs to implement it: - RegisterOpts.OperatorId registers an already claimed row (name and kind from the row, no upsert) and points the row's worker_id at the new worker, which is how ClaimOperators keeps seeing the assignment. - Session.SendStepActionEvent reports on the tenant-forged context path, filling the session's worker id and refusing another's. - The durable channel takes contexts on Send and Recv, uses the contract's error sentinels, stamps and passes worker_status through (the operator reports what it is blocked on; the worker is the session's), and reads the engine's responses on a pump goroutine so a single-threaded operator that sends while a response is undelivered no longer deadlocks against the engine's sequential delivery. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * feat(operator): in-process host over operatorsvc internal/operator/hostinproc implements pkg/operator.Host inside the dispatcher process. Open registers the operator through operatorsvc.Register (an existing row by id for claimed operators, an upsert by name otherwise), opens a handler-backed engine session whose id is both the dispatcher key and the worker row's listener fence, links the initial action set in engine-sized chunks and adds the worker to the host's heartbeat set. Deltas are applied synchronously, so Flush returns at once; events go through the session's tenant-forged path; OpenDurable is the engine session's handshake and hold buffer; PutWorkflow goes through the admin service and reads the derived action ids from the stored steps. One host-wide ticker writes a bulk UpdateWorkerHeartbeats for every open session, including those draining between Pause and Close. pkg/operator/operatortest carries an echo operator written against the contract only; it is opened here in a unit test and, in a later commit, over gRPC in the operator end-to-end suite. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * feat(operator): gRPC host pkg/operator/hostgrpc implements pkg/operator.Host over pkg/client's OperatorSession, a port of the serverless branch's grpclink onto the contract. It keeps one engine client per tenant (rebuilt when the token rotates, closed when replaced, released or on the host's Close), resolves tokens through a TokenSource (the TenantTokenExchange renamed; LocalExchange keeps the YAML file with referenced-file mtime tracking, StaticExchange the single-token case), refuses a session the engine authenticated as another tenant, and retries once after an Unauthenticated connect. A session runs the deliver loop that hands the client's assigned actions to the handler; a refused start is reported as a retryable failure because there is no requeue over gRPC. Pause is the unary PauseWorker RPC and Close is the client's pause-then-drain. The durable hub (task-indexed, bounded outbound queue, one ack-bearing request in flight, evictions and non-determinism errors reconstructed as responses) is mapped onto DurableChannel with context-aware Send and Recv; worker_status is dropped because the listener reports the awaited entries itself. Identity by operator id, a kind other than GRPC, ResumeWorkerId and WorkerName report ErrNotSupported: OperatorService has no such requests yet. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * refactor(operator): manager becomes a claimer over the in-process host Operator is now the contract's operator: an ActionHandler plus Start(ctx, Session) and Drain(ctx). SharedOperator no longer holds a repository or a worker id; it takes the session from Start and reports events, changes the action set (as a diff into AddActions and RemoveActions) and reads its worker id through it. TaskEventWriter keeps only what exists inside the engine: CancelTaskEvent, RegisterDurableTask, TriggerDAGStep and CancelDAGChildren. The DAG operator keeps its repository and writer and gains the lifecycle: Start registers the tenant's DAG actions on the session and starts the poller, Drain interrupts the runs in flight and waits for them. It is hosted in process only. pkg/operator/manager is replaced by internal/operator/claimer, which polls ClaimOperators as before but opens every claimed row through the in-process host (Identity{OperatorId}) and starts the operator on the session; a row that leaves the claim result is paused, drained and closed, and so is every row at Stop. Worker rows, heartbeats and the dispatcher's routing table are the host's now. It moved under internal/ because it depends on the in-process host and on operatorsvc, both engine-internal, and is constructed only by the engine. The dispatcher stops constructing the manager: listenForOperators, the om field and WithDAGOperatorDefaultSlots are gone, and with them the dispatcher -> manager -> operator import edge. run.go wires the claimer next to the dispatcher in both service sets and stops it before the dispatcher drains, while events can still be reported. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * fix(dagoperator): durable handshake through the session The DAG operator opened its durable invocations through RegisterDurableTask, sent register_worker itself and treated the first response as the ack without checking its type. The engine routes responses to the task as soon as it is registered, before the ack, so a completion restored for a resumed invocation could be taken for the ack and lost (F21, the race the serverless enginelink fixed for itself). The run now opens the invocation through Session.OpenDurable: the host does the handshake and holds any response that arrives before the ack, and delivers entries only behind the ack that names them. The dag drives a DurableChannel instead of the raw channel pair; the channel reads the engine's responses on its own, so the dag's send no longer has to drain responses to avoid deadlocking against the engine's sequential delivery, and awaitResponse's blocked-status report runs off a bounded Recv. The dag keeps its own buffer for an entry that races its ack, which the tests still exercise through an unordered test channel. RegisterDurableTask leaves TaskEventWriter and SharedOperator; the dispatcher still implements it for operatorsvc. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * test(operatore2e): host the echo operator over gRPC The echo operator in pkg/operator/operatortest is written against the contract only; internal/operator/hostinproc opens it in a unit test. This scenario opens the same handler through pkg/operator/hostgrpc against the harness engine: identity from the token's tenant, a workflow put through the session, a run driven to completion, and the host's teardown order (pause, drain, close) observed on the worker row. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * chore(operator): comments describe the hosts as they are The operatorsvc package doc names the in-process host that now calls it, ClaimOperators' doc no longer points at CreateOperatorWorker (the claimer registers workers through the operator service), and the DAG step index conversion says why its gosec finding is suppressed. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * feat(operatorsvc): register SERVERLESS operators by name Register upserted GRPC rows only; every other kind had to come in by the id of a claimed row. The serverless operator registers by name like a gRPC operator but under its own kind, so upsertOperator now picks the statement by kind: UpsertGRPCOperator for GRPC, UpsertServerlessOperator for SERVERLESS, a refusal for the rest. The rows stay unique per (tenant, name, kind), so a serverless and a gRPC operator of one name are two rows. AuthorizeOperator's GRPC-only check is unchanged: only out-of-process operators authenticate through OperatorService. The store interface and its test double gain UpsertServerlessOperator. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * refactor(serverless): open registrations through the operator host The serverless core reaches the engine through pkg/operator's Host, Session and DurableChannel instead of its own link layer. Deps.Link becomes Deps.Host, plus two hosting facts the wiring decides: the operator kind (SERVERLESS in the engine, where the in-process host upserts the row; empty in the binary, where OperatorService registers it as a gRPC operator) and an optional worker name, which only the in-process host accepts. A registration is now the session's ActionHandler. HandleAction replaces the Actions channel and the loop that read it: a START_STEP_RUN starts a delivery, a CANCEL_STEP_RUN interrupts one. Flow control is by blocking, never by refusing: deliveries in flight are capped at the worker's slot count and a start that finds no slot waits on the caller's context, since an error would requeue in process and burn a retry as a FAILED event over gRPC. A handler call that arrives before Open returned (the in-process host links the initial actions first) waits for the session. Teardown is pause, drain, close: the worker is paused before the drain so a task assigned in the drain window goes to a live worker rather than into a delivery the timeout would abort. Session reconnects are the host's now, so the registration no longer detaches itself on stream failure and the reconnect metric goes. The relay drives operator.DurableChannel: Send and Recv take a context, the relay's own, which teardown cancels once the channel is closed. It is not derived from the delivery context on purpose: that context ending is the relay's abort, and a Recv that returned on it first would be reported as an engine failure. ErrSessionEnded from Recv is an engine-side failure and resolves to the retryable outcome. The engine builds the core over the in-process host it already runs for the claimed operators, so one host and one operatorsvc.Service serve the DAG operator and the serverless operator alike; startOperatorClaimer hands the host back for that. ConfigFromServer moves in from enginelink, the AddOperatorSession adapter goes. The binary builds hostgrpc with the token sources that now live there. The e2e harness runs every scenario against both hosts and the shutdown scenario asserts the pause lands, with the worker still active, while a delivery is held at the endpoint. The fake link in the core's tests becomes a fake operator.Host and Session; operatortest.Session does not derive workflow actions or open durable channels, which the core's tests need, so the core keeps its own. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * refactor(serverless): delete the link layer pkg/serverlessoperator/link, link/enginelink and link/grpclink are gone: the host contract replaced the interfaces, hostinproc replaced enginelink and hostgrpc is the promoted grpclink. Their tests already live with their subjects: the enginelink durable ordering tests (an entry never ahead of its ack, the bounded pre-handshake hold, close unblocking a blocked send) are operatorsvc's OpenDurable tests, the open, delta and event tests are hostinproc's, the grpclink hub, client cache and token source tests are hostgrpc's, and ConfigFromServer's test moved with it to the engine. The serverless-specific flow control test is the core's registration test. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * docs(operator.proto): say why the heartbeat rides the Listen stream SDK workers heartbeat out of stream because the runtimes they target can block their event loop while a task runs and would starve an in-stream heartbeat. Operators are written against the Go SDK, which is not subject to that, so the heartbeat stays on the stream it keeps alive. The reasoning now lives on the OperatorHeartbeat message so it travels with the protocol. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * refactor(operator): the engine-internal writer names its tenant TaskEventWriter.CancelTaskEvent becomes CancelTaskEventCustom, named for what it is: the writer behind SendCancelledWithMessage, which reports a cancellation with a custom reason, distinct from the CANCELLED step action event every host offers and not on the gRPC surface. Every writer call now takes the tenant as an argument. The dispatcher's implementation no longer reads the tenant the gRPC auth middleware puts on a request context, so pkg/operator stops forging that context value with a copy of the middleware's key; the DAG step calls already carried the tenant and only pretended to need the context. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * refactor(operator): document the lastActions invariant, drop assertions UpdateWorkerActions keeps diffing against the set it last advertised: the engine would accept the whole set on every poll, but each send is a write, and the removes cannot be derived without the previous set. The map needs no lock because the operator calls the method from one goroutine at a time (its Start, then the poller Start launches once that call returned); the doc now says so instead of leaving it implied. The compile-time interface assertions go: the DAG factory, the hosts' Open and the test operator already return these types as the interfaces, so the compiler checks the same thing where it matters, and the repo does not use the idiom elsewhere. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * refactor(hostinproc): take the admin service and the workflow repository directly The in-process host no longer needs a wrapper that forwards to the admin service and the workflow repository: it asks for each on its own, an AdminService for the put and a WorkflowStore for the stored steps the action ids are read from, both satisfied by what the engine already builds. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * refactor(operator): constructors take variadic options for every argument operatorsvc.New, hostinproc.New and claimer.New follow the repo's New(opts ...Opt) shape: required collaborators arrive through With... options like the optional ones and are validated in New with an error that names the option to use, the way grpcoperator.New and dispatcher.New do. The Deps structs go away; the claimer registers factories one kind at a time through WithFactory. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * feat(repository): apply an operator actions delta in one transaction A delta carries adds and removes under one sequence number, and the ack for that sequence tells the operator the delta is committed, so its two sides must commit together. ApplyWorkerActionsDelta replaces AddWorkerActions, AddWorkerActionsWithinBudget and RemoveWorkerActions: one row lock, adds then removes, the budget check before any remove is applied so a refused delta rolls back whole, and the action hash recomputed once from the resulting set. operatorsvc applies a delta with one call and charges the budget with both counts. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * fix(repository): meter operator workers like SDK workers A gRPC operator's workers are the tenant's workers: CreateNewWorker meters them against the WORKER and WORKER_SLOT limits, and the limit queries count them. The one exception is the DAG operator, whose workers the engine creates for every tenant with DAG workflows as infrastructure; those stay out of both, keyed on the operator's kind (a NOT EXISTS against v1_operator) rather than on operatorId, so what CreateNewWorker meters is exactly what the queries count. CreateWorkerOpts names the operator's kind next to its id for that decision. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * refactor(client): operator client and stream helpers get packages of their own pkg/client/streaming holds the reconnecting stream, the listen loop and the error classifier every listener in pkg/client shares, exported. On top of it, pkg/client/operatorclient is the OperatorService client: Connect, the session, the action delta queue and the durable stream opener, with a New(conn, opts...) constructor that takes the token and metadata it needs. It does not import the legacy package, so client.Operator() stays as a thin accessor built over the client's own connection. hostgrpc now speaks operatorclient. The legacy package is named in one file, for the two things only it has (the dial from a token and the environment, and the durable task listener the Go SDK worker shares), and the host builds and owns the listener since the client session no longer does. hostgrpc.New takes variadic options like the other constructors: WithTokenSource is required, WithLogger and WithClientFactory are not; the local exchange's logger option is WithExchangeLogger. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * docs(self-hosting): document the operator settings SERVER_GRPC_OPERATOR_MAX_LISTEN_STREAMS_PER_OPERATOR, SERVER_GRPC_OPERATOR_MAX_ACTIONS_PER_OPERATOR and SERVER_DAG_OPERATOR_DEFAULT_SLOTS join SERVER_GRPC_OPERATORS_ENABLED in the configuration table. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * feat(operator): pause is a message on the Listen stream The Listen stream is stateful, so the pause belongs on it rather than in a unary RPC. The client sends OperatorListenRequest.pause and waits for the OperatorPauseAck the server answers with once the pause is committed; from that ack on nothing is delivered on the stream. The server commits the pause in this order: the dispatcher session stops delivering first, then the row is written, then the ack goes out, so an action the scheduler assigned before it observed the pause is returned to the queue the way a failed send is (subscribedWorker.paused, honoured for starts and not for cancels, which a draining operator may still need). Resuming lifts the pause in the opposite order and is acknowledged the same way. The pause is stream state: Register clears it when it resumes a worker, so a paused client sends it again on every new stream, right after start and before its deltas. The unary PauseWorker RPC and its messages are gone; pkg/client/operatorclient keeps Pause, Resume and the pause-then-drain Close over the stream, and the in-process host pauses through the same operatorsvc.Session.Pause, so both delivery kinds stop the same way. The e2e pause scenario now asserts that nothing reaches the operator after the pause ack and that the run held back runs once after the resume. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * feat(operator): the contract names entries acked outside it and a session's end Two additions to pkg/operator, each for a review finding fixed on top of it. DurableChannel.ExpectEntry(branchId, nodeId) registers an entry whose completion the operator awaits but which no request on the channel acknowledged. The engine-internal writer creates DAG children directly and returns their refs, so no trigger_runs ack ever names them; the channel's ack-before-entry rule would otherwise hold their completions forever. In process it marks the ref as acknowledged (a held completion is queued at once); over gRPC it awaits the entry on the listener, once per entry. Session.Done and Session.Err surface the end of a session to the caller that owns it: Done closes after Close or when the host gave up on the session, and Err says which. Every implementation closes Done on Close today; the gRPC host's supervision of terminal failures builds on it. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * fix(safeclient): speak HTTP/1.1 only so the idle pool bounds every origin Performance R7. HTTP/2 connections live in net/http's h2 pool, which MaxIdleConns does not bound, so a fleet polling many distinct origins retained one connection per origin for as long as the origin was revisited: 500 origins left 500 connections open. The Sender now speaks HTTP/1.1 only, with ForceAttemptHTTP2 off, an empty TLSNextProto map and ALPN offering http/1.1 alone, so the custom dialer safeurl installs can never see h2 frames. Every retained connection is then in the transport's idle pool, bounded by the new Config.MaxIdleConns, MaxIdleConnsPerHost and IdleConnTimeout (256, 4, 90s by default, the names the serverless branch already uses), and Sender.CloseIdleConnections reaches the transport safeurl actually installed. Regression: TestDeliver_IdlePoolBoundsRetainedConnectionsAcrossOrigins (500 origins, MaxIdleConns 8: retained 500 over h2 before, 8 after) and TestDeliver_NegotiatesHTTP1AgainstHTTP2Origin. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * fix(dagoperator): expect directly triggered children and register one condition at a time Correctness R09 and R10, found by running the interpreter over the engine's own durable channel instead of the raw test channel. R09: a child the interpreter creates through the direct TriggerDAGStep callback has no trigger_runs request on the channel, so no ack ever names its entry and the channel held the child's completion for good: the run never advanced past its first asynchronous child, and the blocked-status recovery resent a completion the channel held again. The interpreter now registers every triggered, not yet satisfied step (and the on-failure step) with DurableChannel.ExpectEntry as soon as the callback returns its ref, which stands in for the ack; a completion that arrived first is released at once. R10: a task with a skip and a cancel condition, or several ready tasks with conditions, sent their wait_for registrations back to back, and the channel admits one ack-bearing request until its ack arrives, so the second was refused with ErrRequestInFlight and the run failed. The emitter now registers one condition per pass: a task whose registration must wait for a pending ack stays pending, the main loop consumes the ack (which is what assigns the registration its node id, in FIFO order), and the next pass registers the next one. Log order is unchanged; the trigger gate on pending acks already existed. The racing-ack test in dag_test.go is adapted: with one registration in flight the skip fires on its own ack, so the wait watch is never sent. Regression tests, over operatorsvc's channel and the dispatcher double: TestDagOverEngineChannelCompletesDirectlyTriggeredChild (failed with context deadline exceeded) and TestDagOverEngineChannelRegistersConditionsOneAtATime (failed with ErrRequestInFlight). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * fix(operatorsvc): bound what the durable pump retains and deliver a completion once Correctness R07. After the register-worker ack the in-process channel's pump queued every engine response for Recv, and held every completion whose ref no ack had named, with no bound: 10,000 completions were accepted with no Recv, inside the engine process. RetainedResponseLimit (four handshake holds, 1024) now bounds queued plus held responses for the life of the invocation. Past it the channel fails: what it retained is dropped, Send and Recv return ErrChannelClosed, and the pump keeps discarding until the operator closes the channel, so the engine's delivery goroutine is never left blocked on a failed channel. The DAG operator reports such a run as a retryable failure. A completion is delivered once. The engine resends satisfied entries the operator reports as awaited, and a repeat used to be held forever under the ack-before-entry rule; delivered refs are remembered and a repeat is dropped, which also keeps expectations for delivered entries inert, as the contract says. The gRPC channel's ExpectEntry awaits an entry once however often it is registered, since a second listener callback would strand the first. Regression tests: TestOpenDurableRetainedResponsesAreBounded (Recv timed out against the old channel), TestOpenDurableDeliversACompletionOnce, TestOpenDurableExpectEntryStandsInForTheAck and, over gRPC, TestDurableChannelExpectEntryAwaitsTheEntry. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * fix(repository): one action budget per operator, an unambiguous hash refreshed per sequence, a fenced pause Four findings on the worker action set, all in the delta transaction and the session around it. Correctness R02 and R11, security R01: the per-operator action cap was a per-session snapshot tracked from that session's own deltas, read and written without synchronization from concurrent AddActions (a Go data race under the in-process host) and never shared: N sessions of one operator could each consume the whole cap. The budget is now one transactional check. "Worker"."actionCount" (new column, an invariant every link-mutating path maintains: the delta settles it by added minus removed under the worker row lock, CreateNewWorker sets the initial size, UpdateWorker recounts) makes the operator's total a sum over its workers, O(workers) never O(links). ApplyWorkerActionsDelta takes the operator-wide cap, locks the v1_operator row after the worker row (always in that order), sums the counts and rolls the whole delta back past the cap with *ActionBudgetError carrying the real totals. The session keeps no budget state at all; the fake store mirrors the operator-wide check. Correctness R03: the pause a closing session wrote, and Session.Pause itself, were not fenced on the listener session id, so closing a superseded session paused its successor. SetWorkerPausedForListener writes only while the session is the one on the row; a superseded write is a debug line, the newer session owns the worker's scheduling state. The explicit PauseWorker RPC path stays unfenced. Security R02: the canonical hash joined ids with ";" while an id may contain ";", so two different sets could hash equal and the scheduler, which treats equal hashes as equal sets, could give a worker another set's actions. Go and SQL now hash each sorted id as a 4-byte big-endian length prefix followed by its bytes. The migration recomputes the hash of active workers in place and clears it on inactive ones, which refresh when a session resumes them (OpenSession refreshes a NULL hash before activation, so a worker whose previous session died between a delta and its refresh comes back correct). Performance R2: every delta chunk re-sorted and rehashed all of the worker's links, 500 million row visits for a million-action replay, which timed out at 265,000. The delta now clears the hash and the session refreshes it once per delta sequence: in the throttled notifier before the scheduler notification that ends the window (only when a delta changed the links, on a detached bounded context), and on Close when a refresh is still owed. The scheduler already reads a worker without a hash through the join, which now also covers a pending refresh. Measured on the reviewer's dense fixture: 100k replay 10.3s to 3.1s (one refresh 0.23s); 1M replay from a 30s timeout at 265k to 26.0s (one refresh 2.3s cold, the link insert itself is 15.7s of it); one removal on a 1M worker from 715ms to 1ms plus a 1.0s refresh. The single 1M refresh is still over a second, so the alternative is recorded here: a (workerId, actionSetVersion) cache key, bumped by the settle statement, removes the full-set read entirely at the price of one cache entry per worker instead of per identical set. UpdateOperatorWorkerActions had no caller and is deleted with its test. The e2e hash reads wait for the end-of-sequence refresh. Regression tests: TestSessionActionBudgetSpansSessions, TestConcurrentDeltasThroughHost (race), TestOperatorActionBudgetSpansWorkers, TestOperatorActionBudgetUnderConcurrentDeltas; TestSessionSupersededCloseDoesNotPauseSuccessor, TestSupersededCloseLeavesSuccessorAssignable, TestPauseWorkerForListenerIsFenced; TestHashActionsIsCanonical, TestWorkerActionHashEncodingIsUnambiguous; TestWorkerActionHashRefreshFollowsDeltas, TestSessionRefreshesActionHashPerWindow, TestOpenSessionRefreshesPendingHash. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * fix(operatorclient): a chunk staged before a replay is never sent after it Correctness R01. The flusher stages a chunk (moves it from pending to unacked) before it takes the stream's send lock. A reconnect that owned the lock meanwhile replayed the stream: a fresh worker got a snapshot of the desired set, then the stale staged chunk went out behind it and undid part of the snapshot (wire order [2 1]; the worker lost an action while Flush reported success), and a resumed worker got the chunk twice, once from the replay and once from the flusher. The queue now carries a stream generation, bumped under its mutex by every replay. A chunk is staged with the generation it saw, and the flusher's send, which runs under the same send lock as the replay, checks the generation first: if a replay ran in between, the chunk is not sent and counts as done, because the replay either resent it from unacked (resumed) or dropped it for the snapshot (fresh). Regression tests: TestOperatorSessionFreshReplayDropsStagedRemoval and TestOperatorSessionFreshReplayDropsStagedAdd (the server set diverged from the desired set and the wire carried [2 1]) and TestOperatorSessionResumedReplaySendsStagedChunkOnce. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * fix(hostgrpc): supervise the client session and keep tenant clients referenced Four findings on the out-of-process host. Correctness R05: a terminal failure of the client session (a revoked token, an Unauthenticated Register on reconnect) ended the client's receive loop and the host only logged it, leaving the tenant registered and unserved for good; the token source was never asked again. The host session now supervises its client session: when the client's action stream ends while the session is not closing, it closes the durable hub (open channels fail and their invocations retry), closes the old client session and reconnects through the host with jittered backoff (1s to 30s), which asks the source for the tenant's current token, verifies the tenant, restores the action set the session advertised and resumes delivery under the new registration. It gives up only on a failure no retry fixes (ErrNoToken, a tenant mismatch, ErrNotSupported), reported through the contract's Done and Err. Correctness R04: ReleaseTenant, token rotation and the host's Close closed the tenant's cached client unconditionally, so an old registration's teardown closed the connection a newer registration of the same tenant had just opened over. Cached clients are reference counted: a session holds one reference from connect to Close (or to giving up); eviction only stops the client being handed out, and an evicted client is closed by its last release. Host.Close still closes everything. Correctness R06: the session reported outcomes under the worker its first Open registered, even after a fresh reconnect gave the client a new worker because the engine no longer had the old one, so the engine refused the reports. Registration follows the client's current worker; the worker current at each delivered start is recorded per task run, and a report is stamped with it while it is still the current worker and with the current worker otherwise, since the engine registers a fresh worker exactly when it cannot resume the previous one. A report naming a worker the session used to be is rewritten the same way; a foreign id passes through for the engine to refuse. Performance R8: delivery called the handler serially straight off the client's action channel, so a start that blocked on the operator's flow control held every later message, cancels included, while the client's inbox grew without bound. Delivery now reads the channel eagerly: cancels go to the handler on their own goroutine at once, starts go through a bounded queue (twice the worker's slot units, at least 64) served by one goroutine, and a start that finds the queue full is refused with a retryable failure report instead of being held. Regression tests: TestSessionReopensAfterTerminalFailure, TestSessionGivesUpWithoutToken, TestSessionCloseDuringBackoff and, over a real listener, TestLoopbackSessionRecoversWithRotatedToken; TestReleaseTenantWaitsForOpenSessions; TestSessionReportsUnderCurrentWorker and TestLoopbackFreshReconnectReportsUnderNewWorker; TestCancelsBypassBlockedStarts. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * fix(lease): count and claim only units with endpoints, sample the backlog per tick Correctness R08 and performance R5, both in the claimable count. R08: the count sampled the first countLimit unowned units in key order, empties included. A window of empty units (shard growth, deleted endpoints) ahead of populated ones reported a claimable weight of zero, so a process holding anything computed a weight budget of zero and never claimed; the window never advanced and the populated units stayed unowned for good. Empty units are now neither counted nor claimed: they have nothing to poll, and a unit becomes claimable when an endpoint lands on it. The claimable index covers exactly those units (migration v1_0_158), and the claim's abandoned walk applies the same rule. Progress then follows from the count: a claimable population raises the fair share above what the live processes hold, so at least one of them gets a positive budget. R5: the count's window was MaxClaimPerTick times the live process count, so every replica walked the whole backlog each tick during a cold start or a large takeover, and the abandoned side's limit applied per dead owner, so 101 dead owners returned 91,024 rows under a limit of 1,024. The window is now MaxClaimPerTick, whatever the fleet's size, and the abandoned side is bounded as a whole. A full sample means the backlog is at least a tick's worth: the process takes a full MaxClaimPerTick and the next tick's shed settles any overshoot once the counts are exact, instead of every replica paying for an exact fleet-wide count. Regression tests: TestZeroWeightWindowDoesNotStrandBacklog (the reviewer's TestRound2ZeroWeightWindowStrandsBacklog over the production leaser and SQL, on DATABASE_URL), TestTickClaimsPopulatedUnitsBehindEmptyOnes, TestSaturatedSampleClaimsAFullTick, and the repository subtests "empty units are neither counted nor claimed" and "abandoned count is bounded as a whole". Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * fix(serverless): shed units of a busy tenant without stopping its deliveries Performance R4. InFlight reported the tenant registration's whole delivery count for every one of the tenant's units, and pickShed excluded any unit with a positive count, so one long-running delivery (a single durable invocation was enough) pinned every unit of the tenant to an overloaded process while new replicas had nothing to take. The count was also a walk of the in-flight map per unit considered, O(owned units times deliveries) per shed attempt. Deliveries belong to the registration, which stays open while the tenant owns any unit: losing one of several units stops that unit's pollers and leaves the registration and its deliveries where they are. A unit now reports no deliveries while the tenant owns others, so it is shed like an idle unit and the delivery finishes on the registration it started on. Only the tenant's last unit reports the count, since losing it closes the registration (drained for DrainTimeout, then aborted). The leaser orders such units after the idle ones instead of excluding them: a busy unit is shed when the idle ones do not cover the excess, so a delivery that never ends cannot keep a process overloaded for good. The count is a counter kept beside the map, read in O(1). Regression tests: TestInFlightIsReportedOnTheTenantsLastUnitOnly and TestTickShedsBusyUnitWhenIdleOnesDoNotCover (the busy unit was never shed before). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * fix(serverless): gain the tenants of one lease batch concurrently Performance R6. The leaser hands UnitsGained one batch for a whole tick, and the runner gained its tenants in series: each gain is a load and a Host.Open, so a startup or a takeover of many tenants held the lease tick through every tenant's load and open, 30 seconds per hung open, while the process's heartbeat stayed healthy and no other replica could take the units over. The per-tenant locks had removed the global lock, not the serial loop. Gains now run on a pool bounded by MaintenanceConcurrency, the same bound the maintenance pass uses; each tenant's gain still runs under its own opMu, so no tenant is gained twice at once. The maintenance retry of pending units goes through the same pool. Regression test: TestGainedTenantsOpenConcurrently (six tenants in one call against a host that waits for four concurrent opens: one at a time and six seconds before, four at once now). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * perf(serverless): sync a changed union as its delta, never as the sorted union Performance R1. syncActions called ActionUnion before it read the delta log, and ActionUnion rebuilt and sorted the whole tenant union under the routing cache's write lock on every new revision: a one-action change on a tenant of a million actions copied and sorted a million ids (196 ms and 16 MB per change, with every route on the tenant blocked for the sort), for a delta of one id. The sync path never materializes the union now. DeltasSince returns the coalesced delta together with the revision it ends at, read under one lock, and that delta is what the session gets. The registration keeps the set the engine holds as the base slice it opened with (the cache's shared sorted slice, never modified) plus the deltas it pushed since, and materializes a set only in two O(set) cases that sort nothing: the fallback, when the log no longer reaches the advertised revision, which diffs the set against the union through DiffAgainst and rebases; and compaction, when the applied deltas carry more ids than the base. The sorted union is built only for an open, and ActionUnion now copies the keys under the read lock and sorts outside any lock, publishing the result if the revision is still current. BenchmarkOneActionChangeSync, one healthcheck adding an action followed by the owner's sync, ten actions per endpoint: 1k endpoints (10k actions): 1.26 ms, 165 KB -> 2.0 us, 1.8 KB 10k endpoints (100k actions): 13.3 ms, 1.6 MB -> 2.4 us, 1.8 KB 100k endpoints (1M actions): 196 ms, 16 MB -> 2.4 us, 1.8 KB Regression tests: TestChangedUnionSyncIsIncremental (100 one-action changes over 100k actions under 4 MB, against 160 MB before) and TestSyncFallsBackToDiffAndCompacts (the engine's set equals the union through the fallback and the compaction). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * perf(serverless): resolve routing misses by namespace and reconcile the cache by version Performance R3. Two paths reloaded a served tenant's every endpoint row (configuration, ciphertext and all registered actions, disabled rows included): the anti-entropy full reload every RoutingFullReloadInterval, which also built a configuration per row before finding it unchanged and walked the cache under its write lock; and every routing miss, each on its own, so a burst of assignments to an absent namespace multiplied the tenant's rows over the wire and the serialized application work. At a million endpoints that is at least 530 MB per reload per process. A miss now resolves the one endpoint its namespace names (GetServerlessEndpointByNamespace, through the namespace's unique index and scoped to the tenant); concurrent misses on a namespace share one lookup, and a namespace found absent is remembered for five seconds in a bounded negative cache, so the tenant is never reloaded for a miss. The periodic pass is an anti-entropy reconcile: it pages the tenant's (id, version) pairs through the covering version index (ListServerlessEndpointVersions), fetches in full only the rows whose version the cache does not hold (ListServerlessEndpointsByIds, 500 at a time), stamps what it listed with the pass's generation and drops what it did not, publishing at most one revision for the drops. An unchanged tenant transfers its ids and versions and allocates nothing; deleted endpoints and rows whose commit landed behind a refresh's read are still caught. Each tenant's interval is spread by up to ten percent so the served tenants do not all list on one tick. The zero watermark of a first page is sent as a real time: as NULL it made the keyset comparison NULL, which also affected ListUpdatedSince on a tenant the cache had never seen a row of. Measured on the budget fixture, an unchanged reconcile of 10k endpoints (100k actions): 0 rows read, 0 bytes allocated, 4.7 ms under the race detector; the full reload it replaces read every row and allocated a configuration for each. Regression tests: TestRoutingCacheRouteAndMissRefresh (a miss looks up one namespace, never the tenant, and an absent namespace is not looked up again), TestRoutingCacheCoalescesConcurrentMisses, TestRoutingCacheReconcileFetchesChangedRowsOnly, TestUnchangedReconcileReadsNoRows and the repository subtest "routing lookups by namespace, version listing and fetch by id". Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * fix(safeclient): word policy failures for the tenant, keep the cause for the log Security R03. mapSafeurlError wrapped safeurl's own error into the typed policy error, and safeurl's message carries the resolved address and the whole request URL, query string included, so a tenant-visible task error or endpoint status read "destination blocked by SSRF policy: Post "https://host/path?credential=...": dial tcp [::1]:443: ipv6 blocked". The websocket dialer's resolver already redacted its branch; the HTTP branch and the relay's socket errors did not. A policy failure's public error now names the configured host and the policy category only (scheme, port, userinfo, or an address the policy does not allow), still matchable with errors.Is for classification; a URL that does not parse is refused without echoing it. The cause goes to the operator log through logCause, which drops a *url.Error's URL and keeps its operation and inner error under the host logged beside it. The durable relay maps its dial and socket errors through safeclient.PublicError the same way, so a read error names the endpoint host and the stage, never the address the socket was connected to. Regression tests: TestDeliver_PolicyErrorsAreRedacted (the query marker and the loopback address were in the message before), TestLogCauseStripsTheURL, TestTransportMessageRedactsAddresses. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * fix(serverless): prune the remembered puts on a rejected catalog too Security R06. applyChange remembered each accepted workflow's hash as soon as its put succeeded and pruned the map to the catalog only after the whole catalog was accepted, so a run of rejected catalogs, each with a fresh accepted workflow ahead of the rejected one, grew the per-endpoint map without bound: 250 two-workflow catalogs retained 250 hashes under a catalog cap of 200. The prune now runs whatever the outcome, deferred at the top of applyChange: the remembered puts are those of the catalog just applied, so an accepted early workflow is still not re-put on the retry, while names from earlier catalogs are dropped and the map never holds more than one catalog's worth. Regression test: TestRejectedCatalogsDoNotAccumulatePutHashes (250 retained before, at most 2 now). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * fix(serverless): refuse unstorable actions before the union and reverse a refused delta Security R05. An advertised action with a NUL code point passed the engine's action id rules and the catalog caps, entered the tenant's union and the session's desired set, and only then failed, at the database (text refuses NUL). The routing rollback restored the union, but the registration's advertised revision was unchanged by the failure and the cache's log cancelled the add against the rollback, so no removal was ever sent: over gRPC the client kept the refused id in the desired set it restores on every new client session, so one bad endpoint disrupted the shared tenant registration. Two changes. The parser validates normalized actions against storage before anything sees the catalog: a NUL code point, invalid UTF-8 and ids over maxActionIdBytes (1024, a documented cap where the column has none) are refused with the endpoint's other catalog errors. And a delta the engine refuses is taken back out of the session: syncActions pushes the inverse delta on its own bounded context, so what the session desires is again what the registration advertises, and the next sync, whose log delta coalesces to nothing, has nothing to send. The inverse is idempotent on the engine; its own failure is logged and the forward delta is retried by the next sync as before. A queued chunk the gRPC client has already sent and holds for a resumed replay is the client's to reject or clear; that is on the operator client, not here. Regression tests: TestHealthcheckRejectsUnstorableActions (the NUL action was accepted before) and TestRefusedDeltaIsReversedOnTheSession (the session still desired the refused action before). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * test(serverless): cancel the durable delivery only once the endpoint holds the socket TestDurableDeliveryCancelSendsNoSecondEvent delivered the cancel as soon as STARTED was reported, and STARTED goes out before the socket is dialed: a cancel that landed first aborted the dial, no socket was ever closed and the test timed out waiting for the close code, once in about twelve runs under the race detector. The endpoint now signals the upgrade and the test waits for it before cancelling. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * fix(serverless): sign the upgrade for its endpoint and keep accepted nonces for their window Security R04, in two parts. The upgrade signature covered timestamp, nonce, task id and invocation, so where two endpoints share a signing secret a signature made for one verified at the other; the endpoint id was only an unsigned header compared to configuration. contract.UpgradeSigningPayload now takes the endpoint id first: endpoint_id "." timestamp "." nonce "." task_id "." invocation. The relay signs with the id it dials for, and the example Worker verifies the id it serves against the signed one. The proto is unaffected; the headers already carried the id. Every verifier of the upgrade must move with it: the Go test endpoints and the e2e fake here, and the TypeScript package's verifier on belanger/serverless-ts-core. The example's NonceSet evicted its oldest entry at capacity even inside the five minute window, so under enough upgrades a captured upgrade was accepted again. It now keeps every accepted nonce until its window has passed and reports "full" when it holds capacity live nonces; the verifier then refuses the upgrade with 503 rather than forget a nonce, and the operator retries later. The per-isolate limitation stands and is documented as before. Regression tests: TestSignedUpgradeHeadersAreBoundToTheEndpoint (Go), and the example verifier exercised over Node WebCrypto: an immediate replay is refused, a full set refuses new upgrades and keeps its live nonces, expired entries free it, and the same signature under another endpoint id is a bad signature (both accepted before). The example typechecks with tsc --noEmit. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * test(repository): do not assume the order of two random endpoint ids The routing lookup subtest expected the fetch by id to return the endpoint written second in second place, but ids are random and the rows come back ordered by id, so the expectation held by luck. The subtest now checks the order property and finds each row by its id. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * fix(repository): keep the action hash encoding and drop the hash migration The worker action hash keeps the per-id framing main has always used: sha256 over the sorted, deduplicated, lower-cased action ids, each followed by ";". hashActions in Go and ComputeWorkerActionHash in SQL compute it byte for byte the same way, so a worker created with an initial set and a worker built by deltas hash equal for the same set. The length-prefixed encoding is dropped, and with it the two "actionHash" rewrites in migration v1_0_154, which now only adds "actionCount" and backfills it for operator workers. A rolling deploy needs no hash step: a worker registered by an older engine keeps a hash the new engine computes the same way, and any hash the scheduler does not recognise is a group of its own, so distinct hashes never misroute. The same-tenant ambiguity of a ";" inside an id is closed at validation instead, in the commit that follows. The unambiguous-encoding test becomes a Go/SQL parity test over a fixed set of action sets, and the unit test pins the digest of a sorted set to sha256 of "svc:a;svc:b;". Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * fix(validator): reject semicolons in action ids ParseActionID refuses a ";" anywhere in an action id. The worker action hash frames each id with a semicolon, so an id carrying one could make two different action sets of the same tenant hash equal, and the scheduler treats equal hashes as equal sets. ParseActionID is the one function every registration path goes through: the actionId validator tag (worker registration and update, operator action deltas, workflow step actions and the worker list filter) calls it, and the v0 and v1 PutWorkflow handlers call it directly on each step action before the step is stored. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * refactor(repository): rename "actionCount" to "operatorActionCount" The column added by migration v1_0_154 counts the "_ActionToWorker" rows a worker holds so the per-operator action budget can be summed over the operator's workers instead of counting their links. Only operator workers are read and only they are backfilled; an SDK worker's count is zero until it registers and nothing reads it. The name says so. Semantics are unchanged: every path that links or unlinks actions maintains it, and the budget check sums it under the operator's lock. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * chore(generate): format the configuration table and align the operator protoc stamp The configuration-options table added on this branch was not prettier formatted, so the docs generator reflowed its column widths and both the generate diff check and the docs lint job failed on it. The table is now written the way prettier writes it. operator_grpc.pb.go carried a protoc v5.29.3 stamp from a local protoc; CI generates with v5.29.6, like every other generated file here. The stamp line is aligned by hand rather than regenerated with the local toolchain. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * fix(operatorsvc): admit orchestrator action ids on DAG operator sessions The DAG operator registers the orchestrator action ids of the tenant's DAG workflows, "<workflow>_orchestrator". They are not "service:verb" action ids on purpose: without a ":" no SDK or GRPC worker can register one. On main the operator wrote them through a dedicated repository method that never ran the actionId validator. On this branch the in-process host routes them through Session.ApplyDelta, which validated every delta with the actionId tag, so every refresh failed with InvalidArgument from the first one at engine start, the DAG operator's worker never held an action, and no multi-task workflow of a tenant with dag_operator enabled ever started. That is the failing python test (3.14, true, true) cell: test_cancellation, and the durable spawn-DAG tests, hang until the job times out. Delta validation is now scoped by the operator's kind. A DAG operator session admits orchestrator ids and nothing else; every other kind admits action ids and nothing else. A DAG operator is hosted in process only and its row is never upserted through a registration, so the kind is the engine's own claim and a remote operator cannot hold an orchestrator action. The id form lives in one place, repository.DAGOrchestratorActionId, which the workflow store builds with and the service checks against; the form has no ";" either. The hostinproc tests that seeded DAG rows with "dag:a" style ids had hidden the mismatch; they use orchestrator ids now, and an operatorsvc test reproduces the failure on both kinds. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * feat(repository): add operator leasing and worker limit exemption columns What an operator is and who keeps it alive are separate axes. v1_operator.kind stays what it is (DAG, the engine-internal DAG operator; GRPC, a contract operator hostable in process or out of process) and a new v1_operator.leasing enum says who keeps the row alive: MANAGED rows are assigned to a dispatcher by ClaimOperators and built from a factory inside the engine, SELF rows keep themselves alive through a Listen stream out of process or their own leaser in process. SERVERLESS is retired: the serverless operator is GRPC with SELF in both modes, and the migration folds any SERVERLESS rows into GRPC (the enum value stays; databases migrated by the serverless branch carry it). Rows are unique per (tenant, name, kind) for every kind, replacing the two partial indexes that only covered the kinds upserted by name. Duplicates are renamed, not deleted, before the index is built. Limit exemption is a hosting fact decided at worker creation rather than an attribute of the operator row, so "Worker"."exemptFromLimits" is added, backfilled for the DAG operator's workers, which the limit queries left out before. The queries switch to the column in a following commit. The migration is v1_0_159: belanger/serverless-operator, which stacks on this branch, owns v1_0_155 to v1_0_158. It runs in goose's transaction: the enum type is created here rather than extended, and only a value added by ALTER TYPE ... ADD VALUE is unusable in the transaction that added it. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * refactor(repository): exempt workers from limits by flag Whether a worker counts against the tenant's WORKER and WORKER_SLOT limits used to be derived from the kind of the operator it backs (DAG workers were exempt). That tied metering to the operator row, which no longer says who hosts the operator. It is now an explicit CreateWorkerOpts.ExemptFromLimits, stored on the row, that only the in-process operator host sets: every worker it creates is engine infrastructure that runs whether or not the tenant runs workers of its own, whatever kind or leasing its row has. Wire-registered workers, over OperatorService or from an SDK, are metered. The limit queries in workers.sql and tenant_limits.sql read the flag off the worker instead of joining v1_operator on kind, so what CreateNewWorker meters is what they count. CreateWorkerOpts.OperatorKind had no other reader and is removed. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * feat(operator): separate operator kind from leasing Registration and claiming now run on the leasing axis rather than the kind list. ClaimOperators claims every MANAGED row, whatever its kind; the claimer resolves the factory by kind, except for GRPC rows, which it resolves by the row's name through WithNamedFactory, since the kind only says the row is a contract operator and the name says which one the engine should build. A claimed row with no factory is logged once and left alone. UpsertGRPCOperator is replaced by UpsertOperator(tenant, name, kind, leasing) over the plain unique index; a repeat registration takes the leasing it names, so a row the engine was leasing that registers itself leaves the claim set on the next poll, and the other way round. operatorsvc.RegisterOpts carries Leasing; a named registration must say who keeps the row alive. Wire registration (grpcoperator Register, hostgrpc) is GRPC and SELF, and AuthorizeOperator, behind the hatchet-operator-id metadata, requires kind GRPC and leasing SELF: an engine-leased row can never be driven over the wire. operator.Identity gains Leasing, defaulting to SELF; hostinproc passes it through and marks every worker it creates exempt from limits. The DAG operator's row is created MANAGED, and a race creating it is tolerated now that the name is unique. The unused CreateOperatorWorker query and repository method, which predate the claimer registering workers through the operator service, are removed. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * refactor(serverless): register as a self-leased GRPC operator The serverless operator no longer has a kind of its own. With kind and leasing split, what it is (a contract operator written against pkg/operator) and who keeps it alive (itself, through the sessions it holds, in both modes) are said separately: every registration opens its session with kind GRPC and leasing SELF, in the binary over hostgrpc and inside the engine over the in-process host. The claimer never sees the rows, since it claims MANAGED rows only, and the in-process host still exempts the workers it creates from the tenant's limits while out-of-process registrations are metered like any other wire operator. Deps.OperatorKind goes: the identity is fixed, not a hosting choice, so the wiring no longer picks it. The generated V1OperatorKindSERVERLESS constant is gone with the enum value the merged schema no longer declares. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * refactor(operator): name the leasing manager The v1_operator column that says who keeps an operator alive is now leasing_manager, of type v1_operator_leasing_manager with the values SELF and DISPATCHER. DISPATCHER replaces MANAGED: the dispatcher claims the row through ClaimOperators and builds the operator from a factory, a SELF row keeps itself alive. The Go side follows: sqlcv1.V1OperatorLeasingManager, the LeasingManager field on Identity, RegisterOpts, CreateOperatorOpts and UpsertOperatorOpts, and the comments that described managed rows. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * chore(migrate): one migration for the gRPC operator The three migrations this branch added (the GRPC enum value and its partial index, operatorActionCount, then leasing and exemptFromLimits) collapse into 202609111300000_v1_0_154.sql. The file runs with NO TRANSACTION so the GRPC value is committed before anything uses it; the partial GRPC index is never created, only the (tenant, name, kind) unique index; and the SERVERLESS rewrite is gone, since that value never exists on this branch. The goose id is fifteen digits: main's 202609101223115_v1_0_153.sql is, and goose refuses a version that sorts below one already applied. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * chore(migrate): one migration for the serverless operator The four migrations this branch added (the SERVERLESS enum value, the serverless tables, the index rework, the claimable index filter) collapse into 202609111400000_v1_0_155.sql, a plain transactional file: every type it uses is created in it. SERVERLESS is never added, since the serverless operator is a GRPC operator with leasing manager SELF, and its partial index is gone with it; the tables and indexes are created in their final shape, so the orphaned-unit release of the old 156 has nothing to do. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * test(serverless): lift the barrier fake once barrierHost closed its release channel whenever the count of Opens in progress reached want. The concurrency test gains six tenants with a MaintenanceConcurrency of four: the first four Opens lift the barrier, one of them returns, a fifth tenant's Open enters, the count reaches four again and the channel is closed a second time, which panics the package. The runner opened each of the six tenants exactly once; only the fake was wrong. The barrier now lifts under a sync.Once, and the condition is >= want so the most-concurrent measurement still records the first full batch. Opens that arrive after the barrier lifted pass straight through, which is what the test's assertion on MaintenanceConcurrency expects. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * chore(sdks): regenerate the Python and Ruby REST clients for serverless endpoints The serverless endpoint and tenant settings routes were added to the OpenAPI contract without regenerating the Python and Ruby REST clients, so the sdk-python lint (3.14) and sdk-ruby lint jobs, which regenerate every client from the spec and diff, failed on the missing ServerlessApi and its models. Generated with sdks/python/generate.sh (openapi-generator-cli 7.12.0) and sdks/ruby/generate.sh (openapi-generator-cli 7.14.0), the same scripts CI runs; the pre-commit fixers ran on the Ruby output as the script does. No SDK version bump or changelog entry: the clients only mirror the contract on this feature branch. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * chore(ts-sdk): format the generated REST contracts The V1CreateWebhookRequest union in data-contracts.ts was left in the generator's pre-prettier form when main was merged into this branch; main and the parent branch carry the single-line form prettier produces, so the TypeScript lint jobs (pnpm and bun) failed prettier --list-different on the file. Regenerating with pnpm run generate-api and running the project's prettier yields exactly this hunk, so the sdk-typescript generate check and the lint check now agree on the file. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * Renumber the operator migration to v1_0_155 with a fourteen-digit goose id. Main remapped v1_0_153 to a fourteen-digit id and added v1_0_154, so this file moves to 155 and drops the note about matching a fifteen-digit version. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * Renumber the serverless migration to v1_0_156 with a fourteen-digit goose id. Follows main's remap of v1_0_153 and the operator migration's move to v1_0_155. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * Renumber the serverless migration to v1_0_157. Main added its own v1_0_155, which moved the operator migration to v1_0_156. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * Address review comments on the operator migration, queries and naming. Migration: the "Worker" columns are added in one re-runnable statement at the top so the table lock is released before the backfills, the header states that as the reason for running without a transaction, and the comments are cut down. Queries: the action hash is recomputed and stored by one RefreshWorkerActionHash query, and the fenced pause goes through UpdateWorker with an optional listener session id instead of its own query. UpdateWorker now also scopes by tenant. Naming: isExemptFromLimits, is_paused on OperatorPause and OperatorPauseAck, CancelTaskWithReason, LinkNewActionsToWorker and UnlinkActionsFromWorker, and the durable goroutines are a receive loop (engine side) and a send loop (client hub) instead of a pump. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * Realign struct literals after the isExemptFromLimits rename. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * Constrain DAG operators to dispatcher leasing and make the down migration atomic. v1_operator gets a check that a DAG row is leased by the dispatcher. The down migration is sent as one query, so Postgres runs it as a single transaction and the drops need no IF EXISTS. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * Serve the operator service over ConnectRPC. The Listen and DurableTask handlers take connect's bidi streams and run over a transport-free half so their tests keep driving fake streams; the operator id is read from the request header through the handler call info; errors are connect errors with the same codes. The dispatcher's operator stream session takes the stream's guarded sender and wraps assigned actions itself, replacing the wrap hook, and the durable session is entered through the dispatcher's DurableTaskWithReceive so the ownership check on the first message stays in the operator service. The operator client stays on grpc-go, and the operator e2e suite passes against the connect server. Also fixes the two admin trigger paths that main left on the removed status package (#4755 landed beside #4987). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * Fix the load-tagged partition test for the two-value CreatePartitions. Main's #5002 changed the query to :one and left this test on the old signature. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * Add stream frames and a per-task streams flag to the serverless contract A serverless task that needs an engine stream (child run results, run events, durable events) cannot hold one from a fetch-based runtime, so the operator opens the stream on the task's behalf and relays it over the invocation websocket. The healthcheck catalog gains per-task options with a streams flag so the SDK can ask for a socket for a non-durable task, and the frame oneof gains stream_open, stream_message and stream_close. Existing frames and field numbers are unchanged; the TypeScript bindings are regenerated in the serverless SDK phase. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * Let an operator session open the engine's run observation streams The serverless relay needs to open SubscribeToWorkflowRuns, SubscribeToWorkflowEvents and ListenForDurableEvent on behalf of a tenant's task without handing the endpoint a token, so the host contract gains Session.OpenRunStream: one method over a small RunStream (Send, Recv, Close) keyed by RunStreamKind, which also names the procedure and the message types so a relay can decode protojson frames without knowing the transport. The gRPC host opens the streams on the operator client's connection with the session's bearer token. The in-process host needs the dispatcher to serve the same handlers over channels the way RegisterDurableTask does for DurableTask: the run and durable event handlers now take their receive half as a function, and RegisterRunStream runs them over a request channel and a bounded response channel with the tenant on the context. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * Relay engine streams over the serverless invocation websocket A task that needs a stream gets a socket: a non-durable task the catalog flags with streams is invoked over the same signed upgrade and first frame as a durable one (invocation 1, no durable channel) and finishes with a done frame, while unflagged tasks keep the POST byte for byte. The relay grows a stream table per socket: stream_open decodes the procedure's first message, checks the runs it names against one run filter (any run of the tenant today, a namespace later without a frame change), and opens the engine stream through the session; stream_message flows both ways; stream_close from either side tears the engine stream down; every stream ends with the socket. Procedures outside the allow-list, undecodable requests, engine refusals and the per-socket cap (WS_MAX_STREAMS, 16) are answered with a stream_close carrying a connect code, and stream frames count against the existing queued-bytes budget. SubscribeToWorkflowRuns is multiplexed per tenant registration: one engine stream per process, opened on the first subscriber, reopened with backoff with the awaited runs subscribed again, and its events fanned out to every socket that asked for the run, the way the SDK's child listener shares one stream. The other kinds are one engine stream per socket. The endpoint row records which actions asked for a socket (stream_actions, migration v1_0_158) next to registered_actions, so every process routing the tenant delivers them the same way. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * Cover a flagged serverless task awaiting a child over the socket end to end The scenario runs on both hosts: a parent task the catalog flags with streams is invoked over the websocket, triggers an unflagged child through the tenant's unary client, opens a SubscribeToWorkflowRuns stream for it over the socket, and completes with the child's output once the terminal event arrives, while the child itself is posted. The fake endpoint advertises task options in its healthcheck and runs a stream script on the socket of a non-durable task. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * Fold the stream_actions column into the serverless migration. One migration per PR: #4912 is unmerged, so the column is created with the table rather than added by a second file. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * Regenerate the serverless contract bindings for the stream frames. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * Remove the Cloudflare Workers example from the operator branch. The example is an SDK artifact and belongs with the TypeScript serverless package PR, next to the code it exercises. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * Drop the serverless readiness check from the engine health probe. The operator initiates every connection it holds, so nothing routes to a replica on the core's account and its state says nothing useful about the replica's readiness. The health package goes back to its shape on main; the supervisor keeps Running for its tests only. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * Generalize the hosted-operator comments and reword RegisterRunStream. The engine comments described the DAG operator where they mean any claimed operator; RegisterRunStream now says what it proxies and why. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * Name the contract timestamps by their unit. ServerlessHealthcheckRequest and ServerlessTriggerRequest carry the build time as Unix seconds; the field name now says so and the proto and header comments state the unit and the 5 minute freshness window, so an endpoint SDK cannot mistake the value for milliseconds. The field numbers are unchanged. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * Register serverless endpoints under their plain names. Each endpoint had a namespace UUID the operator prefixed onto every workflow, action and event it registered, carried in the healthcheck, trigger and first-frame bodies and confining the durable relay. The per-endpoint auth it was the hook for needs a larger refactor of namespace-scoped tokens on worker endpoints, so the prefixing goes: workflows and actions register exactly as a worker's do. Two endpoints of one tenant declaring the same action both serve it, like two workers: the routing cache keeps the enabled endpoints per action and picks one per delivery uniformly at random among those not known unhealthy, falling back to any enabled one, and a miss fails the task retryable without a repository lookup. The column, the proto fields (numbers reserved), the REST field, the miss query and the relay's confinement are removed, and the SDK REST clients follow. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * Take the serverless shard count from the operator configuration. The shard count is a backend sizing choice, not something a tenant sets over the API, so the tenant-settings endpoints and their schemas go. SERVER_SERVERLESS_OPERATOR_SHARD_COUNT (1 to 64, default 1) is written into a tenant's v1_serverless_tenant row when the API server creates it with the tenant's first endpoint; an existing row keeps its count, so a change applies only to tenants first seen after it, and endpoints keep the shard they were inserted with. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * Move the serverless operator binary's config into the config packages. The binary's environment struct and its bindings live in pkg/config/serverlessoperator and the loaders for its repository, encryption service and token exchange in pkg/config/loader, the way the other binaries are wired. The token exchange mode is explicit: SERVERLESS_OPERATOR_TOKEN_EXCHANGE=static reads HATCHET_CLIENT_TOKEN for one tenant, local reads the token file, each mode requires its own variable and refuses the other mode's, so a stray token cannot be carried along unnoticed; the single-tenant warning stays. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * Gate the serverless operator on a tenant entitlement. tenant_entitlement.serverless_operator gates the feature the way dag_operator does: endpoint creation over the API answers 403 for a tenant without it, and the operator, in either hosting mode, counts and claims only lease units of entitled tenants and releases the units it holds of a tenant whose entitlement is switched off on its next tick, so the runner stops serving that tenant. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * Page the routing cache's full tenant load. ListServerlessEndpointsForTenant is the cache's full load, so a tenant of any size read it in one statement. It is keyset-paged by id in the repository, 500 rows at a time, and the cache applies each page under its own lock and revision, so routing on the tenant is never held for the whole load. The versioned incremental refresh is unchanged. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * State the run stream filter's tenant-wide decision. allowRun admits any run of the tenant: the socket is authorized for one task run of the tenant and the operator's engine calls carry the operator's own tenant-wide credentials. A narrower token scope is checked at the same point later, without a frame change. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * Write registered_actions before the session delta. Other processes read an endpoint's action set from its row, so the delta reached the engine before the row held it: a failed write left the owner's session advertising a set nobody else could see. The row is written first; a failed write changes nothing and the next poll retries the whole change, and a delta refused after a successful write restores the cached contribution and is retried without another write. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * Soft-delete serverless endpoints so caches see the deletion. A deleted row was absent from the incremental refresh, so a routing cache kept delivering to the endpoint, with its cached signing secret, until the next full reconcile. The delete API sets deleted_at with updated_at, so the versioned refresh returns the row and every cache drops the endpoint on its next pass; every other read filters deleted rows out, names are unique among live endpoints, and the operator's sweep purges rows once the grace period has passed. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * Bump the SDKs whose REST clients gained the serverless endpoints API. TypeScript 1.34.0-alpha.3, Python 1.41.2 and Ruby 0.8.1 carry the regenerated REST clients; each changelog describes the addition and the Ruby example lock pin follows the gem version. The docs changelog mirrors are regenerated by the pre-commit hook. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * Wait for the worker's action set in the e2e registration helper. registered_actions is written before the session delta, so the row alone says nothing about what the engine can assign yet; the helper also waits for an active worker of the tenant to advertise the action. The trigger record's name field is the job name. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * Regenerate the TypeScript serverless bindings without the namespace. The bindings follow api-contracts/v1/serverless.proto: the reserved namespace fields are gone from the healthcheck, trigger and first frame messages and the comments describe plain-name registration. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * Rename serverless shards to endpoint partitions. The lease unit a serverless operator process owns is now (tenant, partition): v1_serverless_tenant.endpoint_partition_count, the endpoint_partition column on v1_serverless_endpoint and v1_serverless_lease, the SERVER_SERVERLESS_OPERATOR_ENDPOINT_PARTITION_COUNT and SERVERLESS_OPERATOR_ENDPOINT_PARTITION_COUNT settings, and the matching Go, sqlc, and docs names. The migration is edited in place; sqlc output is regenerated. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * Renumber the serverless migration to v1_0_161. Main gained v1_0_158 through v1_0_160 since this branch was cut, so the serverless migration now sorts after the newest applied file and carries the next free label. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * Document the exported serverless API handlers. Every exported type, constructor and handler method in the serverless v1 handler package and the serverless transformers now carries a godoc comment. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * Declare the serverless endpoint operations as payload-free. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * Let concurrent registrations of a new workflow name both succeed. PutWorkflowVersion read the name and then inserted the row, so two registrations racing on a new name left one with a unique violation. The insert now yields to a concurrent creator and the loser continues as a new version of that row. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * Back off only on engine refusals of a serverless catalog. A PutWorkflow failure without a refusal code (engine unavailable, a transient database error) marks the endpoint unhealthy and is retried on the next poll instead of entering the rejected-catalog backoff. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * Constrain the concurrent workflow create test like its helpers. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * Answer unparseable and cyclic workflow registrations with InvalidArgument. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * Renumber the serverless operator migration after main's latest label. Main's chain now ends at 20261013191212_v1_0_173, so the serverless operator migration moves to 20261013200000_v1_0_174 with its content unchanged. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * Sign serverless requests through pkg/encryption. Sign and Verify live in the serverless contract package and build on encryption.NewHMAC, so the FIPS key-length check applies to signing secrets. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * Refuse to run the serverless operator under fips140=only. The durable relay dials endpoints over RFC 6455 websockets, whose upgrade handshake is specified over SHA-1, and Go refuses SHA-1 in that mode. The engine and the standalone binary now fail at startup with a clear error instead of panicking on the first durable delivery. The tests that upgrade a websocket skip there, and the test signing secrets meet the FIPS minimum key length. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * Renumber the serverless operator migration after main's v1_0_174. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * Cancel run streams before waiting on a blocked send. A gRPC run stream's Send holds sendMu across the transport write, which flow control can block; Close took the same mutex to half-close the send side before cancelling the stream context, so invocation teardown and registration shutdown could wait behind a stalled engine stream. Close now marks the stream closed and cancels its context without touching sendMu; the cancellation is what ends the stream on the transport and releases the blocked send. The half-close is dropped, since the engine treats a cancelled stream the same as a half-closed one. sendMu stays with Send alone, where it serialises senders and re-checks the closed flag under the lock. The tenant run mux had the same ordering: install replayed every awaited run while holding mu, and close took mu before cancelling. install now snapshots the awaited runs, sends outside the lock, and loops until no run subscribed during the replay is left unsent before publishing the stream; close cancels the mux context before taking the lock. mu is no longer held across any engine send. Regression tests drive a Send blocked until its context is cancelled and check that Close from another goroutine returns within a second, for the gRPC stream and for the mux's replay; both fail on the previous ordering. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * Bound and release the run interests of a serverless stream. An endpoint's SubscribeToWorkflowRuns stream over an invocation socket could name any number of run ids, and the tenant mux kept every one in engine state for the registration's lifetime: the dispatcher has no unsubscribe message, it polls every id it was told of once a second until the stream ends, and a handle's close only edited the mux's local maps while the shared engine stream stayed open. A handle now awaits at most WSMaxRunsPerStream distinct runs at once (SERVER_SERVERLESS_OPERATOR_WS_MAX_RUNS_PER_STREAM and SERVERLESS_OPERATOR_WS_MAX_RUNS_PER_STREAM, default 1024); a run already awaited is not counted again and a finished run frees its place. A subscription past the cap fails with ResourceExhausted, which the relay turns into a stream_close with code 8 while the socket stays up, the way the stream cap is answered. When a handle closes while it still awaits runs no other handle holds, the mux retires the engine stream: it is closed, the serve loop finds it retired rather than failed, and the next stream is opened at once with only the runs still awaited, so the engine drops the orphaned ids with the old stream. Closes that overlap while a reopen is in flight share it, and a replay that sent a run whose last holder closed during the replay retires that stream before publishing it. The mux stays one engine stream per registration rather than one per handle: a stream per handle would multiply the dispatcher's per-stream tenant event subscription and poll goroutine by the number of live invocation sockets and lose the mux's transparent reconnect, while a retire-and-reopen costs one open per handle close that leaves orphans, which in normal operation is rare since an invocation awaits its runs to completion. The dispatcher is unchanged: its acknowledgement set is per stream and is dropped when the stream ends, so retiring the operator's stream is what releases the engine-side state and poll work, and the cap bounds what one handle can place there. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * Frame the catalog hash by length and refuse control characters in action ids. The catalog hash joined action ids and stream-action ids with newlines, so two catalogs whose ids contained a newline could encode alike and a stream-flag change between them would be skipped as unchanged. The digest now frames the workflow encodings, the action set and the stream-action set as tagged sections with length-prefixed elements, so it is a function of the lists themselves. The healthcheck storage check also refuses any control character in an action id, alongside NUL, invalid UTF-8 and oversized ids. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * Reconcile action deltas made while a client session is replaced. A recovery snapshotted the desired action set, flushed it to the replacement client session and installed the replacement without looking at the desired set again. A delta made in between updated the desired set but was routed to the previous client, which was being closed, so the replacement never advertised it and a later Flush reported success for an action set it did not hold. The session now reads the desired set and the client a delta goes to under one hold of its lock, the lock the replacement is installed under, and the install queues the difference between the restored snapshot and the desired set on the replacement before releasing it. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * Tell endpoints a fixed message per code when an engine stream fails. A stream_close frame carried the text of the engine-side error that ended the stream, so an endpoint could read engine hostnames, ports and other transport diagnostics. The frame now carries the connect code with a fixed message for that code, and the error itself is logged at debug level under the task, invocation and stream id. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * Word a websocket ping failure for the tenant like a write failure. The ping path reported the raw transport error, while the dial and write paths report the endpoint host and the stage that failed through transportMessage. The ping path now uses the same form. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * Release a run stream replay blocked on flow control when the mux closes. The run mux replays subscriptions on a fresh engine stream before publishing it, and grpc-go releases a send blocked on flow control only when the stream's own context ends. Closing the mux cancelled a context the stream did not share and closed only the published stream, so a close during a blocked replay waited on the loop forever. OpenRunStream now cancels the stream when the context it received ends, the same as Close does, and the mux holds the stream it is replaying on as opening from the moment open returns, so close reaches it whether or not the replay has finished. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * Make a listed healthcheck action set the exact set the endpoint serves. The TypeScript package sends the served set in actions, which may leave out tasks of a declared workflow, while the operator read the field as ids served in addition to the derived ones. It registered and routed tasks the endpoint refuses. A non-empty actions is now the served set as stated; an empty one serves every action derived from workflows. Workflows are still registered whole, and stream options must name a served action. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * Keep durable tasks off endpoints that do not support the durable protocol. A healthcheck that reports no durable support leaves the workflow's durable tasks out of the endpoint's served actions, so the engine never assigns it one, and the capability is written to the endpoint row as durable_supported so every process's routing cache sees it. A durable assignment that still reaches such an endpoint is refused before the dial as a retryable failure. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * Leave the SDK versions and changelogs untouched. The serverless REST endpoints ride with the next regular SDK releases. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * Serve a gained unit only while its lease is still pending. A maintenance pass copied the pending units and gained them afterwards, so a unit the leaser lost in between was loaded, added to the tenant and polled, and no later lease tick removed it. A gained unit now stays pending until the gain that loaded its tenant claims it, UnitsLost clears it from pending before it waits on the tenant, and a gain serves only the units it claims; a tenant left with nothing to serve is forgotten. The direct gain and the retry share the rule. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * Keep a written catalog over a refresh row read before the write. The poller wrote the endpoint's action set and put it in the cache without a version, so a refresh that had read the row before the write applied the row's older set over it, and the union lost the new catalog until the next refresh. The write now returns the row's updated_at, the poller stamps the cached set with it, and a refresh row versioned before that stamp keeps the cached set and version while its other fields are taken as read, the way a status this process wrote is kept. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * Send run subscriptions from a writer goroutine, never from the subscriber. A subscriber sent its subscription on the shared engine stream inline, and the gRPC stream's Send is released only by the stream's close, so when the engine connection stalled a subscriber whose socket had timed out stayed blocked in Send, and the relay's teardown waited on it with the task slot occupied. Each published stream now has a writer that owns its sends from a bounded queue; a subscriber records its interest and enqueues the run id without touching the transport, and a full queue retires the stalled stream so the reopen replays every awaited run. The writer exits when its stream is retired or the mux closes, and a handle's Close never waits on a send. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * Stamp the cached catalog separately from the row version. The routing cache keeps the updated_at of the write that recorded an endpoint's catalog apart from the row version it last read, so a refresh row older than the catalog keeps the catalog while every other field and the row version are taken as read; settings saved before the write apply with the next row and the row at the write's version is not skipped. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
| Commit: | 93409a3 | |
|---|---|---|
| Author: | Alexander Belanger | |
Make a listed healthcheck action set the exact set the endpoint serves. The TypeScript package sends the served set in actions, which may leave out tasks of a declared workflow, while the operator read the field as ids served in addition to the derived ones. It registered and routed tasks the endpoint refuses. A non-empty actions is now the served set as stated; an empty one serves every action derived from workflows. Workflows are still registered whole, and stream options must name a served action. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK
| Commit: | 0ead286 | |
|---|---|---|
| Author: | Alexander Belanger | |
Merge origin/main into belanger/serverless-operator Resolves the merge against main's FIPS, durable streams and entitlement changes: the serverless signing helper now builds on encryption.NewHMAC, the serverless endpoint handler checks the entitlement through HasEntitlement with the new EntitlementServerlessOperator, and the local encryption envelope keeps main's FIPS key checks while NewLocalDataEncryption shares it. Generated OAS, REST client and proto output is regenerated, and the SDK versions move above main's releases (TS 1.39.0-alpha.0, Python 1.42.2, Ruby 0.10.1). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK
| Commit: | a22623e | |
|---|---|---|
| Author: | Julius Park | |
| Committer: | Julius Park | |
sdk changes
| Commit: | e2e6644 | |
|---|---|---|
| Author: | Julius Park | |
| Committer: | GitHub | |
feat(durable-streams): add Reset method for durable streams (#5194)
| Commit: | 7adc8bb | |
|---|---|---|
| Author: | Julius Park | |
| Committer: | GitHub | |
feat: add RetryAfter(Exception|Error) to SDKs and engine (#5162) * initial commit * fix issues * lint python * always warn * address comments * lint etc * Update frontend/docs/content/docs/reference/go/runnables.mdx Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * fix generate * move migration * fix fractional stuff * trim down comments --------- Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
| Commit: | 0855599 | |
|---|---|---|
| Author: | Julius Park | |
| Committer: | Julius Park | |
initial commit
| Commit: | 9aded23 | |
|---|---|---|
| Author: | Julius Park | |
| Committer: | GitHub | |
feat(durable-streams): large messages, metadata (#5139)
| Commit: | 109ac5c | |
|---|---|---|
| Author: | Julius Park | |
initial commit
| Commit: | 65c7d76 | |
|---|---|---|
| Author: | Alexander Belanger | |
Merge remote-tracking branch 'origin/main' into belanger/serverless-operator
| Commit: | 1925dc7 | |
|---|---|---|
| Author: | Julius Park | |
| Committer: | GitHub | |
feat: durable streams (#5072) * initial wip * fix tests, add default cursor behavior * rabbit publishes must be confirmed otherwise grpc error occurs, needs deslopping * forgot batch files * a variety of fixes/deslopping * some more fixes (first message race, tighten up sdk failures) * small generation fixes * fix lint * more linting and remove test that keeps failing * generate * refactors, etc * consolidate all pg reads into topic poller * retention stuff * more granular fixes * make partitions by tenantid so they can be dropped based on tenant retention * fix ordering bug * some validation changes etc * rabbit-ectomy, use direct to pg for grpc handlers * forgot some files * rework partitions etc * rework offset stuff * move migration * add feature flag * retention management * consolidate tests * clean up comments and fix load test * add changelog * add test files that were untracked * autovac settings * use default autovac for messages table * add reconnection * fix generate * fix v0config issue
| Commit: | 9678dd5 | |
|---|---|---|
| Author: | Alexander Belanger | |
Register serverless endpoints under their plain names. Each endpoint had a namespace UUID the operator prefixed onto every workflow, action and event it registered, carried in the healthcheck, trigger and first-frame bodies and confining the durable relay. The per-endpoint auth it was the hook for needs a larger refactor of namespace-scoped tokens on worker endpoints, so the prefixing goes: workflows and actions register exactly as a worker's do. Two endpoints of one tenant declaring the same action both serve it, like two workers: the routing cache keeps the enabled endpoints per action and picks one per delivery uniformly at random among those not known unhealthy, falling back to any enabled one, and a miss fails the task retryable without a repository lookup. The column, the proto fields (numbers reserved), the REST field, the miss query and the relay's confinement are removed, and the SDK REST clients follow. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK
| Commit: | 793dc97 | |
|---|---|---|
| Author: | Alexander Belanger | |
Name the contract timestamps by their unit. ServerlessHealthcheckRequest and ServerlessTriggerRequest carry the build time as Unix seconds; the field name now says so and the proto and header comments state the unit and the 5 minute freshness window, so an endpoint SDK cannot mistake the value for milliseconds. The field numbers are unchanged. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK
| Commit: | 2d799d9 | |
|---|---|---|
| Author: | Julius Park | |
merge
| Commit: | b4a480f | |
|---|---|---|
| Author: | Julius Park | |
| Committer: | Julius Park | |
initial wip
| Commit: | 8d1b4e3 | |
|---|---|---|
| Author: | Alexander Belanger | |
Merge remote-tracking branch 'origin/main' into belanger/serverless-operator Main added its own v1_0_157, so the serverless migration moves to v1_0_158. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK
| Commit: | 073f45f | |
|---|---|---|
| Author: | Gabe Ruttner | |
| Committer: | GitHub | |
fix(engine): look up satisfied durable entries by primary key (#5026) * fix(engine): look up satisfied durable entries by primary key The old join filtered branch and node after reading every entry for the task. * fix(engine): prune durable entry lookups to the task partition Pass each task's inserted_at into ListSatisfiedEntries and plan with those values so daily partitions that do not contain them are not locked. * fix(engine): bound durable entry lookups by task inserted_at range Replace the timestamp array with min/max bounds so the btree does not re-sort the array on every index rescan, and return to the plain primary-key join with materialized inputs instead of a lateral subquery. * simple feedback * drop plan stuff * fix(engine): pass task inserted_at on the operator action The DAG operator reads inserted_at from the assigned action instead of looking the task up. Keep the satisfied-entry external id typed as uuid so sqlc still emits uuid.UUID. * fix: signatures
| Commit: | 1ae390a | |
|---|---|---|
| Author: | Alexander Belanger | |
Add stream frames and a per-task streams flag to the serverless contract A serverless task that needs an engine stream (child run results, run events, durable events) cannot hold one from a fetch-based runtime, so the operator opens the stream on the task's behalf and relays it over the invocation websocket. The healthcheck catalog gains per-task options with a streams flag so the SDK can ask for a socket for a non-durable task, and the frame oneof gains stream_open, stream_message and stream_close. Existing frames and field numbers are unchanged; the TypeScript bindings are regenerated in the serverless SDK phase. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK
| Commit: | e990e1e | |
|---|---|---|
| Author: | abelanger5 | |
| Committer: | GitHub | |
feat(engine): engine-side gRPC operator support for out-of-process operators (#4910) * feat(engine): gRPC operator service for out-of-process operators Adds v1.OperatorService so an operator can run outside the engine and talk to it over gRPC with a tenant-scoped token: - Register (unary) upserts the operator by (tenant, name, kind = GRPC) and creates the worker for the connection, or resumes a previous worker of the same operator. Registration carries no workflows and no actions. - Listen (bidi) activates the worker for the lifetime of the stream: the first message is start{worker_id}, then heartbeats and action deltas flow in and AssignedAction messages flow out straight from the dispatcher fan-out. Deltas are applied incrementally (at most 1000 ids per message) and the scheduler is notified at most once per second while they arrive. - SendStepActionEvent and DurableTask delegate to the dispatcher; the durable stream is wrapped so the first register_worker message is checked for worker ownership before the dispatcher sees it. - Every RPC after Register, including Listen, carries hatchet-operator-id metadata validated against the token's tenant through a cached lookup. Repository: CreateWorkerOpts.OperatorId flows into the existing CreateWorker query and skips WORKER / WORKER_SLOT metering, since operator workers are excluded from those limit counts. AddWorkerActions / RemoveWorkerActions apply deltas in one transaction under the worker row lock and maintain the action hash as the XOR of per-action sha256 digests, so equal sets hash equal in any order and a delta costs O(chunk). The proto keeps importing the package-less dispatcher.proto for the shared action types (no SDK regeneration). SERVER_GRPC_OPERATORS_ENABLED (default false) gates service registration. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * feat(client): gRPC operator client with streamed action deltas Go client for v1.OperatorService in pkg/client, on the existing reconnecting stream machinery: Connect calls Register and opens Listen with the start message; heartbeats and action deltas go through retrySend so a dead stream reconnects and resumes the worker. AddActions / RemoveActions never block: a flusher coalesces pending ops (an add followed by a remove of an id that was never sent cancels out), chunks them to 1000 ids and sends every 250ms or as soon as a chunk is full. Flush waits for the queue to drain. The session keeps the desired action set and replays it as chunked deltas when a reconnect's Register did not resume the previous worker. PutWorkflow is a thin helper over the admin service that returns the derived action ids for the caller to AddActions. Harness-based e2e suite covering echo, durable memo, deactivation on close, bulk deltas with hash consistency, resume without replay, non-resume replay, and the operator id and worker ownership checks. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * fix(dispatcher): keep operator stream sends serialised and drains unblockable The shutdown drain ranges the session map and did a blocking send on each stream session's fin channel. A Listen handler that exited concurrently released its session and stopped selecting on fin, so a pointer the drain had already captured blocked it forever. Operator stream sessions now carry a done channel that Release closes, and the drain selects on fin or done through requestFin, so a released session is skipped. Sends on one stream are serialised by the per-worker send lock, but the lock was released by the caller's deferred Release as soon as its context ended, while the SendMsg goroutine was still blocked by flow control. The next caller then started a second SendMsg on the same stream, which gRPC forbids. The lock is now released by the send goroutine once SendMsg exits; a caller whose context ends returns without the lock, and the next caller fails fast with errFlowControlActive after the lock timeout instead of overlapping. AddOperatorStreamSession returns an OperatorStreamSession handle with Fin, Send and Release. Send shares the send serialisation with the action fan-out so an operator service can write its own protocol messages on the stream, and the optional wrap function converts each assigned action into the stream's server message type before it is encoded. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * fix(grpcoperator): acknowledge action deltas and retain them until committed The Listen protocol had no delta acknowledgement. A delta the stream had accepted but the engine had not committed (stream ended before the apply, database error, rejected chunk) was dropped by the client as soon as the send returned, and a resumed worker never learned about it: the client only replayed its desired set for a fresh worker, and repeating the same AddActions was a no-op against the desired set. Listen now returns a stream of OperatorListenResponse, a oneof of the dispatcher's AssignedAction and an OperatorActionsAck. Each delta carries a positive, strictly increasing sequence; the engine applies deltas in stream order and acks a sequence once the change is committed, and an ack for N confirms every lower sequence. A delta that cannot be applied, or whose ack cannot be written, ends the stream instead of leaving it unconfirmed. The dispatcher session wraps actions into the envelope and writes acks through the same per-stream send serialisation as the fan-out. The client keeps every sent chunk until its ack arrives and replays the unacked chunks in order on every reconnect. When the reconnect did not resume the worker it replaces the queue with a snapshot of the desired set, taken under the same lock AddActions and RemoveActions use, so a removal that races the replay is queued relative to the snapshot instead of being coalesced away with an add that was never sent. Replay runs as the reconnecting stream's replay callback, under sendMu before the stream is published, so no chunk is sent in between. A chunk that waits longer than the ack timeout hangs the stream up so the reconnect replays it. Flush now waits for the acks, so a nil Flush means the engine committed the set. Because Flush is called before Actions by the serverless link, the session runs its receive and heartbeat loops from connect until Close; the receive loop demultiplexes acks and actions, buffering actions in an inbox that the consumer Actions attaches drains. Starting the loops and attaching the consumer add to the WaitGroup under the mutex Close takes, which removes the Actions/Close race on the WaitGroup; Actions after Close is refused. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * fix(repository): hash worker action sets canonically and check the tenant on links The worker action hash was the XOR of per-action digests seeded from whatever the row held. A worker created with an initial set stored the sha256 of the ordered list while a worker built by deltas stored the XOR fold, so equal sets hashed differently across the two paths, and the XOR fold is linear, so a chosen set of ids could be solved to collide with any target hash. GetWorkerActionsByWorkerActionHash shares one worker's action list with every worker of the same hash, so both properties matter. hashActions is now the canonical digest: sha256 over the lower-cased, deduplicated ids sorted by byte order, each followed by ";". The delta transactions recompute the same digest in SQL from the linked rows (ComputeWorkerActionHash, sha256 over string_agg ordered with the C collation) while holding the worker's row lock, so the stored hash is always the digest of the links the transaction leaves behind. UpdateWorker, which only adds links, recomputes it the same way. At 100,000 linked actions one add chunk of 1,000 costs about 85 ms, a no-op re-add about 12 ms and a remove about 65 ms on the review database. Deltas no longer upsert the shared Action rows: existing rows are read, the missing ones are inserted in sorted order with ON CONFLICT DO NOTHING, and the resulting row ids are linked in sorted order. Concurrent replays of the same set by different workers therefore take no locks on existing Action rows, write no new row versions, and cannot deadlock. The row lock, the link and the unlink statements all carry the tenant: LockWorkerActionHash finds no row for a worker of another tenant, which is reported as an error wrapping pgx.ErrNoRows before anything is mutated, and the link and unlink statements join Worker and Action on the tenant so a cross-tenant pair affects zero rows. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * fix(grpcoperator): refuse a second register_worker on a DurableTask stream The durable stream wrapper checked worker ownership on the first register_worker only and passed every later message through, so a second register_worker naming another operator's worker in the same tenant reached the dispatcher, which rebinds the named worker's durable dispatcher without an operator check. The SDK's durable listener registers exactly once per stream and opens a new stream to register again, so a second register_worker is a protocol error: the wrapper now refuses it with InvalidArgument, whichever worker it names, and the stream stays bound to the worker it was authorized for. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * fix(grpcoperator): cap Listen streams and action links per operator Public gRPC admission had no cumulative limit: an authenticated tenant token could open any number of Listen streams for one operator, each a live worker exempt from the worker and slot metering, and stream deltas that grow the operator's action links without bound. The service now enforces two per-operator caps, refused with ResourceExhausted. WithMaxListenStreamsPerOperator bounds the Listen streams one operator holds open on this replica (default 100); a stream over the cap is refused before its worker is activated and the slot is returned when the stream ends. WithMaxActionsPerOperator bounds the action links held across all workers of the operator (default 1,000,000). The budget is read once per stream with CountOperatorWorkerActions and then tracked from the deltas the stream applies; the repository enforces it on the links a delta would newly create, inside the delta's transaction, so a delta over the budget links nothing and a delta that repeats actions the worker already holds (a replay after a resumed reconnect) never counts. Zero disables either cap. The caps are options on the service. Wiring them to server configuration (runtime.grpcOperatorMaxListenStreamsPerOperator and runtime.grpcOperatorMaxActionsPerOperator with env bindings, next to grpcOperatorsEnabled) is a change to pkg/config/server and cmd/hatchet-engine outside this branch's scope. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * fix(client): close operator clients and cancel durable sends Client gains Close, which closes the gRPC connection the client owns, so a caller that retires a client (the serverless grpclink when a tenant's token rotates or its last registration closes) releases the connection's goroutines instead of leaving them behind (correctness F14, performance F13). grpclink's closeClient calls it directly and the goleak regression test that had to skip while pkg/client exposed no Close now runs. DurableTaskListener.SendRequest takes a context and returns an error: the request queue is bounded, so a caller blocked on an unreachable engine is released when its context ends or the listener is stopped (Stop closes a per-generation stop signal that Start renews), and the listener's own request helpers drop their pending ack when the send is refused. SendMemoCompleted follows the same shape. grpclink's pump hands requests to the listener under a context closeAll cancels, so a stopped registration no longer holds the pump on a full queue (correctness F12). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * fix(grpcoperator): expose admission caps in server config The per-operator Listen stream cap and action link budget were options on the operator service with no way to set them from configuration. Runtime now carries GRPCOperatorMaxListenStreamsPerOperator (default 100) and GRPCOperatorMaxActionsPerOperator (default 1,000,000), bound to SERVER_GRPC_OPERATOR_MAX_LISTEN_STREAMS_PER_OPERATOR and SERVER_GRPC_OPERATOR_MAX_ACTIONS_PER_OPERATOR next to SERVER_GRPC_OPERATORS_ENABLED, and both engine run paths pass them to grpcoperator.New through WithMaxListenStreamsPerOperator and WithMaxActionsPerOperator. Zero disables either cap (security F06). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * fix(repository): recompute operator worker action hash from links UpdateOperatorWorkerActions stored the digest of the list it was given while only adding links, so a worker that already held links ended with a hash that named a subset of its linked set, and GetWorkerActionsByWorkerActionHash would then share the wrong action list with other workers of that hash. The call now locks the worker row with the tenant in the predicate (a worker of another tenant is refused and nothing is written), lower-cases and dedupes the ids, links through the tenant-checked LinkActionsToWorkerReturning, and recomputes the hash from the linked rows with ComputeWorkerActionHash inside the same transaction, as the worker repository's delta path does (correctness F17, security F02, F08). Tests check that the stored hash is the canonical digest of the final linked set after an additive call and after a call naming a subset, that it equals the delta path's hash for the same set, and that the tenant is checked. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * test(grpcoperator): synchronise the operator store fake fakeOperatorStore counted GetOperatorById calls and served its map with no synchronisation, while TestListenStreamCapIsPerOperator authorizes several Listen streams concurrently through it, so the race detector failed the package intermittently. The counter is atomic and the map is guarded, and the assertions read the counter through Load. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * refactor(grpcoperator): extract operatorsvc The gRPC operator service owns logic that is not gRPC's: upserting the operator row, creating or resuming its worker, activating the worker under a listener fence, applying action set deltas within the per-operator budget, and closing the session again. The in-process operator host needs all of it, and the serverless operator's engine link already re-implements it by hand, with the drift that comes from a copy: no resume, no budget, a separate dispatcher session key. internal/services/operatorsvc now holds that logic behind Service, Session and the two deliveries a session can have. A stream delivery encodes assigned actions onto a gRPC stream and counts against the per-operator stream cap; a handler delivery hands them to an operator running in this process and counts against nothing but the action budget, since it holds no stream. Both get one session id that is the dispatcher's session key and the worker row's listener fence, both notify the scheduler through the same throttle, and both close by pause, drain, deactivate. The gRPC Listen handler closes without the pause: its operator pauses itself before hanging up, and a stream that simply drops must leave the worker assignable for the next connection. internal/services/grpcoperator keeps what is genuinely the protocol: the metadata that names the operator, the start handshake, the message loop, the delta acks, and the durable stream it hands to the dispatcher unchanged. The dispatcher gains AddOperatorSession, the handler-backed twin of AddOperatorStreamSession, keyed on the session id its caller chose, and pkg/operator gains ActionHandler, the one method a session needs to deliver to. The test doubles for the stores and the dispatcher move to operatorsvc/operatorsvctest so the service's tests and the handlers' tests drive the same behaviour. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * feat(operatorsvc): durable invocations with ack-before-entry ordering An operator hosted in the engine drives a durable invocation over the dispatcher's channel-backed session rather than the DurableTask stream, and the handshake and ordering that goes with it is not obvious enough to leave to each host. Session.OpenDurable performs the register_worker handshake itself, so Recv only ever returns invocation traffic, and applies the ordering the operator expects: the engine delivers a completion for an already satisfied entry as soon as the invocation registers, which on a replay is before the operator has sent the wait_for that names it, so an entry is held until the ack carrying its ref has gone out. Responses that arrive during the handshake are held under the same rule, bounded at 256 so an engine that never acks cannot make the session buffer without end. As on the stream, one ack-bearing request is in flight at a time, because the engine keys its pending state by (task, invocation). This is a port of the serverless operator's engine link, which grew the ordering rule as a bug fix and will be deleted in favour of this once the in-process host lands. The gRPC DurableTask handler is unchanged: it still hands the raw stream to the dispatcher after the ownership check. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * feat(grpcoperator): PauseWorker An operator that wants to shut down cleanly needs the scheduler to stop assigning to its worker before it stops answering, and it needs to know the pause is committed before it starts draining. A message on the Listen stream would need an ack of its own and would not work while that stream is being torn down or reconnected, so pausing is its own unary call. OperatorService.PauseWorker takes {worker_id, paused}: the same call pauses and unpauses, which is what a worker that is resumed rather than replaced needs. It is authorised like SendStepActionEvent, by the operator metadata plus worker ownership. Register already clears the pause when it resumes a worker, so an operator that paused to drain and then crashed comes back assignable. The proto is unreleased, so this is an addition with no compatibility shim. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * feat(client): pause-then-drain on session close Closing an operator session used to hang the Listen stream up at once, which deactivates the worker while the actions the consumer is still working on are in flight; the engine only recovers those when the tasks time out. Close is now pause then drain: it pauses the worker through PauseWorker, so the scheduler stops assigning, waits for the task runs already handed to the consumer to be reported, and only then flushes the deltas and ends the stream. The session learns what is in flight from the contract it already has with the caller: a task run is outstanding from the moment it is handed to the Actions channel until the caller reports it completed, failed or cancelled. Cancels carry no work of their own and are never reported, so they are not tracked. The wait is bounded (30 seconds by default, WithDrainTimeout to change it) and WithoutDrain keeps the previous immediate hang-up, which is what a process that is going away anyway wants. Pause and Resume are also on the session, for a caller that drains on its own schedule rather than at close. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * fix(client): count an action in flight before handing it over Two ways the drain on Close could wait for reports that were never coming. An action was recorded as in flight after the send on the Actions channel returned, so a consumer that reported the outcome the instant it took the action cleared an entry that had not been added yet; the entry added afterwards was never removed and every later Close waited out the whole drain timeout. The action is now recorded before the hand-over, and taken back when the hand-over does not happen. The other way is a consumer that walks away: it cancels the context it passed to Actions, and the runs it was already holding will never be reported. Ending delivery now forgets them, so Close hangs up instead of waiting; the engine retries those tasks when they time out, which is what happened to them before this branch too. Also: report the drain timeout only when something really is outstanding, keep the operator name on the Listen handler's log lines, and say plainly that the notifier may publish one notification after stop. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * feat(operator): host contract Add the hosting contract to pkg/operator: Identity (tenant plus an existing operator row or a name and kind to upsert), OpenOpts, Host, Session, Registration, DurableChannel and the sentinel errors. It is what an operator needs from the engine whichever process it runs in; the engine-internal types (Operator, TaskEventWriter, SharedOperator) stay in operator.go and are documented as such. operatorsvc grows what the in-process host needs to implement it: - RegisterOpts.OperatorId registers an already claimed row (name and kind from the row, no upsert) and points the row's worker_id at the new worker, which is how ClaimOperators keeps seeing the assignment. - Session.SendStepActionEvent reports on the tenant-forged context path, filling the session's worker id and refusing another's. - The durable channel takes contexts on Send and Recv, uses the contract's error sentinels, stamps and passes worker_status through (the operator reports what it is blocked on; the worker is the session's), and reads the engine's responses on a pump goroutine so a single-threaded operator that sends while a response is undelivered no longer deadlocks against the engine's sequential delivery. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * feat(operator): in-process host over operatorsvc internal/operator/hostinproc implements pkg/operator.Host inside the dispatcher process. Open registers the operator through operatorsvc.Register (an existing row by id for claimed operators, an upsert by name otherwise), opens a handler-backed engine session whose id is both the dispatcher key and the worker row's listener fence, links the initial action set in engine-sized chunks and adds the worker to the host's heartbeat set. Deltas are applied synchronously, so Flush returns at once; events go through the session's tenant-forged path; OpenDurable is the engine session's handshake and hold buffer; PutWorkflow goes through the admin service and reads the derived action ids from the stored steps. One host-wide ticker writes a bulk UpdateWorkerHeartbeats for every open session, including those draining between Pause and Close. pkg/operator/operatortest carries an echo operator written against the contract only; it is opened here in a unit test and, in a later commit, over gRPC in the operator end-to-end suite. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * feat(operator): gRPC host pkg/operator/hostgrpc implements pkg/operator.Host over pkg/client's OperatorSession, a port of the serverless branch's grpclink onto the contract. It keeps one engine client per tenant (rebuilt when the token rotates, closed when replaced, released or on the host's Close), resolves tokens through a TokenSource (the TenantTokenExchange renamed; LocalExchange keeps the YAML file with referenced-file mtime tracking, StaticExchange the single-token case), refuses a session the engine authenticated as another tenant, and retries once after an Unauthenticated connect. A session runs the deliver loop that hands the client's assigned actions to the handler; a refused start is reported as a retryable failure because there is no requeue over gRPC. Pause is the unary PauseWorker RPC and Close is the client's pause-then-drain. The durable hub (task-indexed, bounded outbound queue, one ack-bearing request in flight, evictions and non-determinism errors reconstructed as responses) is mapped onto DurableChannel with context-aware Send and Recv; worker_status is dropped because the listener reports the awaited entries itself. Identity by operator id, a kind other than GRPC, ResumeWorkerId and WorkerName report ErrNotSupported: OperatorService has no such requests yet. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * refactor(operator): manager becomes a claimer over the in-process host Operator is now the contract's operator: an ActionHandler plus Start(ctx, Session) and Drain(ctx). SharedOperator no longer holds a repository or a worker id; it takes the session from Start and reports events, changes the action set (as a diff into AddActions and RemoveActions) and reads its worker id through it. TaskEventWriter keeps only what exists inside the engine: CancelTaskEvent, RegisterDurableTask, TriggerDAGStep and CancelDAGChildren. The DAG operator keeps its repository and writer and gains the lifecycle: Start registers the tenant's DAG actions on the session and starts the poller, Drain interrupts the runs in flight and waits for them. It is hosted in process only. pkg/operator/manager is replaced by internal/operator/claimer, which polls ClaimOperators as before but opens every claimed row through the in-process host (Identity{OperatorId}) and starts the operator on the session; a row that leaves the claim result is paused, drained and closed, and so is every row at Stop. Worker rows, heartbeats and the dispatcher's routing table are the host's now. It moved under internal/ because it depends on the in-process host and on operatorsvc, both engine-internal, and is constructed only by the engine. The dispatcher stops constructing the manager: listenForOperators, the om field and WithDAGOperatorDefaultSlots are gone, and with them the dispatcher -> manager -> operator import edge. run.go wires the claimer next to the dispatcher in both service sets and stops it before the dispatcher drains, while events can still be reported. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * fix(dagoperator): durable handshake through the session The DAG operator opened its durable invocations through RegisterDurableTask, sent register_worker itself and treated the first response as the ack without checking its type. The engine routes responses to the task as soon as it is registered, before the ack, so a completion restored for a resumed invocation could be taken for the ack and lost (F21, the race the serverless enginelink fixed for itself). The run now opens the invocation through Session.OpenDurable: the host does the handshake and holds any response that arrives before the ack, and delivers entries only behind the ack that names them. The dag drives a DurableChannel instead of the raw channel pair; the channel reads the engine's responses on its own, so the dag's send no longer has to drain responses to avoid deadlocking against the engine's sequential delivery, and awaitResponse's blocked-status report runs off a bounded Recv. The dag keeps its own buffer for an entry that races its ack, which the tests still exercise through an unordered test channel. RegisterDurableTask leaves TaskEventWriter and SharedOperator; the dispatcher still implements it for operatorsvc. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * test(operatore2e): host the echo operator over gRPC The echo operator in pkg/operator/operatortest is written against the contract only; internal/operator/hostinproc opens it in a unit test. This scenario opens the same handler through pkg/operator/hostgrpc against the harness engine: identity from the token's tenant, a workflow put through the session, a run driven to completion, and the host's teardown order (pause, drain, close) observed on the worker row. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * chore(operator): comments describe the hosts as they are The operatorsvc package doc names the in-process host that now calls it, ClaimOperators' doc no longer points at CreateOperatorWorker (the claimer registers workers through the operator service), and the DAG step index conversion says why its gosec finding is suppressed. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * docs(operator.proto): say why the heartbeat rides the Listen stream SDK workers heartbeat out of stream because the runtimes they target can block their event loop while a task runs and would starve an in-stream heartbeat. Operators are written against the Go SDK, which is not subject to that, so the heartbeat stays on the stream it keeps alive. The reasoning now lives on the OperatorHeartbeat message so it travels with the protocol. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * refactor(operator): the engine-internal writer names its tenant TaskEventWriter.CancelTaskEvent becomes CancelTaskEventCustom, named for what it is: the writer behind SendCancelledWithMessage, which reports a cancellation with a custom reason, distinct from the CANCELLED step action event every host offers and not on the gRPC surface. Every writer call now takes the tenant as an argument. The dispatcher's implementation no longer reads the tenant the gRPC auth middleware puts on a request context, so pkg/operator stops forging that context value with a copy of the middleware's key; the DAG step calls already carried the tenant and only pretended to need the context. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * refactor(operator): document the lastActions invariant, drop assertions UpdateWorkerActions keeps diffing against the set it last advertised: the engine would accept the whole set on every poll, but each send is a write, and the removes cannot be derived without the previous set. The map needs no lock because the operator calls the method from one goroutine at a time (its Start, then the poller Start launches once that call returned); the doc now says so instead of leaving it implied. The compile-time interface assertions go: the DAG factory, the hosts' Open and the test operator already return these types as the interfaces, so the compiler checks the same thing where it matters, and the repo does not use the idiom elsewhere. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * refactor(hostinproc): take the admin service and the workflow repository directly The in-process host no longer needs a wrapper that forwards to the admin service and the workflow repository: it asks for each on its own, an AdminService for the put and a WorkflowStore for the stored steps the action ids are read from, both satisfied by what the engine already builds. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * refactor(operator): constructors take variadic options for every argument operatorsvc.New, hostinproc.New and claimer.New follow the repo's New(opts ...Opt) shape: required collaborators arrive through With... options like the optional ones and are validated in New with an error that names the option to use, the way grpcoperator.New and dispatcher.New do. The Deps structs go away; the claimer registers factories one kind at a time through WithFactory. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * feat(repository): apply an operator actions delta in one transaction A delta carries adds and removes under one sequence number, and the ack for that sequence tells the operator the delta is committed, so its two sides must commit together. ApplyWorkerActionsDelta replaces AddWorkerActions, AddWorkerActionsWithinBudget and RemoveWorkerActions: one row lock, adds then removes, the budget check before any remove is applied so a refused delta rolls back whole, and the action hash recomputed once from the resulting set. operatorsvc applies a delta with one call and charges the budget with both counts. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * fix(repository): meter operator workers like SDK workers A gRPC operator's workers are the tenant's workers: CreateNewWorker meters them against the WORKER and WORKER_SLOT limits, and the limit queries count them. The one exception is the DAG operator, whose workers the engine creates for every tenant with DAG workflows as infrastructure; those stay out of both, keyed on the operator's kind (a NOT EXISTS against v1_operator) rather than on operatorId, so what CreateNewWorker meters is exactly what the queries count. CreateWorkerOpts names the operator's kind next to its id for that decision. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * refactor(client): operator client and stream helpers get packages of their own pkg/client/streaming holds the reconnecting stream, the listen loop and the error classifier every listener in pkg/client shares, exported. On top of it, pkg/client/operatorclient is the OperatorService client: Connect, the session, the action delta queue and the durable stream opener, with a New(conn, opts...) constructor that takes the token and metadata it needs. It does not import the legacy package, so client.Operator() stays as a thin accessor built over the client's own connection. hostgrpc now speaks operatorclient. The legacy package is named in one file, for the two things only it has (the dial from a token and the environment, and the durable task listener the Go SDK worker shares), and the host builds and owns the listener since the client session no longer does. hostgrpc.New takes variadic options like the other constructors: WithTokenSource is required, WithLogger and WithClientFactory are not; the local exchange's logger option is WithExchangeLogger. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * docs(self-hosting): document the operator settings SERVER_GRPC_OPERATOR_MAX_LISTEN_STREAMS_PER_OPERATOR, SERVER_GRPC_OPERATOR_MAX_ACTIONS_PER_OPERATOR and SERVER_DAG_OPERATOR_DEFAULT_SLOTS join SERVER_GRPC_OPERATORS_ENABLED in the configuration table. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * feat(operator): pause is a message on the Listen stream The Listen stream is stateful, so the pause belongs on it rather than in a unary RPC. The client sends OperatorListenRequest.pause and waits for the OperatorPauseAck the server answers with once the pause is committed; from that ack on nothing is delivered on the stream. The server commits the pause in this order: the dispatcher session stops delivering first, then the row is written, then the ack goes out, so an action the scheduler assigned before it observed the pause is returned to the queue the way a failed send is (subscribedWorker.paused, honoured for starts and not for cancels, which a draining operator may still need). Resuming lifts the pause in the opposite order and is acknowledged the same way. The pause is stream state: Register clears it when it resumes a worker, so a paused client sends it again on every new stream, right after start and before its deltas. The unary PauseWorker RPC and its messages are gone; pkg/client/operatorclient keeps Pause, Resume and the pause-then-drain Close over the stream, and the in-process host pauses through the same operatorsvc.Session.Pause, so both delivery kinds stop the same way. The e2e pause scenario now asserts that nothing reaches the operator after the pause ack and that the run held back runs once after the resume. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * feat(operator): the contract names entries acked outside it and a session's end Two additions to pkg/operator, each for a review finding fixed on top of it. DurableChannel.ExpectEntry(branchId, nodeId) registers an entry whose completion the operator awaits but which no request on the channel acknowledged. The engine-internal writer creates DAG children directly and returns their refs, so no trigger_runs ack ever names them; the channel's ack-before-entry rule would otherwise hold their completions forever. In process it marks the ref as acknowledged (a held completion is queued at once); over gRPC it awaits the entry on the listener, once per entry. Session.Done and Session.Err surface the end of a session to the caller that owns it: Done closes after Close or when the host gave up on the session, and Err says which. Every implementation closes Done on Close today; the gRPC host's supervision of terminal failures builds on it. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * fix(safeclient): speak HTTP/1.1 only so the idle pool bounds every origin Performance R7. HTTP/2 connections live in net/http's h2 pool, which MaxIdleConns does not bound, so a fleet polling many distinct origins retained one connection per origin for as long as the origin was revisited: 500 origins left 500 connections open. The Sender now speaks HTTP/1.1 only, with ForceAttemptHTTP2 off, an empty TLSNextProto map and ALPN offering http/1.1 alone, so the custom dialer safeurl installs can never see h2 frames. Every retained connection is then in the transport's idle pool, bounded by the new Config.MaxIdleConns, MaxIdleConnsPerHost and IdleConnTimeout (256, 4, 90s by default, the names the serverless branch already uses), and Sender.CloseIdleConnections reaches the transport safeurl actually installed. Regression: TestDeliver_IdlePoolBoundsRetainedConnectionsAcrossOrigins (500 origins, MaxIdleConns 8: retained 500 over h2 before, 8 after) and TestDeliver_NegotiatesHTTP1AgainstHTTP2Origin. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * fix(dagoperator): expect directly triggered children and register one condition at a time Correctness R09 and R10, found by running the interpreter over the engine's own durable channel instead of the raw test channel. R09: a child the interpreter creates through the direct TriggerDAGStep callback has no trigger_runs request on the channel, so no ack ever names its entry and the channel held the child's completion for good: the run never advanced past its first asynchronous child, and the blocked-status recovery resent a completion the channel held again. The interpreter now registers every triggered, not yet satisfied step (and the on-failure step) with DurableChannel.ExpectEntry as soon as the callback returns its ref, which stands in for the ack; a completion that arrived first is released at once. R10: a task with a skip and a cancel condition, or several ready tasks with conditions, sent their wait_for registrations back to back, and the channel admits one ack-bearing request until its ack arrives, so the second was refused with ErrRequestInFlight and the run failed. The emitter now registers one condition per pass: a task whose registration must wait for a pending ack stays pending, the main loop consumes the ack (which is what assigns the registration its node id, in FIFO order), and the next pass registers the next one. Log order is unchanged; the trigger gate on pending acks already existed. The racing-ack test in dag_test.go is adapted: with one registration in flight the skip fires on its own ack, so the wait watch is never sent. Regression tests, over operatorsvc's channel and the dispatcher double: TestDagOverEngineChannelCompletesDirectlyTriggeredChild (failed with context deadline exceeded) and TestDagOverEngineChannelRegistersConditionsOneAtATime (failed with ErrRequestInFlight). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * fix(operatorsvc): bound what the durable pump retains and deliver a completion once Correctness R07. After the register-worker ack the in-process channel's pump queued every engine response for Recv, and held every completion whose ref no ack had named, with no bound: 10,000 completions were accepted with no Recv, inside the engine process. RetainedResponseLimit (four handshake holds, 1024) now bounds queued plus held responses for the life of the invocation. Past it the channel fails: what it retained is dropped, Send and Recv return ErrChannelClosed, and the pump keeps discarding until the operator closes the channel, so the engine's delivery goroutine is never left blocked on a failed channel. The DAG operator reports such a run as a retryable failure. A completion is delivered once. The engine resends satisfied entries the operator reports as awaited, and a repeat used to be held forever under the ack-before-entry rule; delivered refs are remembered and a repeat is dropped, which also keeps expectations for delivered entries inert, as the contract says. The gRPC channel's ExpectEntry awaits an entry once however often it is registered, since a second listener callback would strand the first. Regression tests: TestOpenDurableRetainedResponsesAreBounded (Recv timed out against the old channel), TestOpenDurableDeliversACompletionOnce, TestOpenDurableExpectEntryStandsInForTheAck and, over gRPC, TestDurableChannelExpectEntryAwaitsTheEntry. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * fix(repository): one action budget per operator, an unambiguous hash refreshed per sequence, a fenced pause Four findings on the worker action set, all in the delta transaction and the session around it. Correctness R02 and R11, security R01: the per-operator action cap was a per-session snapshot tracked from that session's own deltas, read and written without synchronization from concurrent AddActions (a Go data race under the in-process host) and never shared: N sessions of one operator could each consume the whole cap. The budget is now one transactional check. "Worker"."actionCount" (new column, an invariant every link-mutating path maintains: the delta settles it by added minus removed under the worker row lock, CreateNewWorker sets the initial size, UpdateWorker recounts) makes the operator's total a sum over its workers, O(workers) never O(links). ApplyWorkerActionsDelta takes the operator-wide cap, locks the v1_operator row after the worker row (always in that order), sums the counts and rolls the whole delta back past the cap with *ActionBudgetError carrying the real totals. The session keeps no budget state at all; the fake store mirrors the operator-wide check. Correctness R03: the pause a closing session wrote, and Session.Pause itself, were not fenced on the listener session id, so closing a superseded session paused its successor. SetWorkerPausedForListener writes only while the session is the one on the row; a superseded write is a debug line, the newer session owns the worker's scheduling state. The explicit PauseWorker RPC path stays unfenced. Security R02: the canonical hash joined ids with ";" while an id may contain ";", so two different sets could hash equal and the scheduler, which treats equal hashes as equal sets, could give a worker another set's actions. Go and SQL now hash each sorted id as a 4-byte big-endian length prefix followed by its bytes. The migration recomputes the hash of active workers in place and clears it on inactive ones, which refresh when a session resumes them (OpenSession refreshes a NULL hash before activation, so a worker whose previous session died between a delta and its refresh comes back correct). Performance R2: every delta chunk re-sorted and rehashed all of the worker's links, 500 million row visits for a million-action replay, which timed out at 265,000. The delta now clears the hash and the session refreshes it once per delta sequence: in the throttled notifier before the scheduler notification that ends the window (only when a delta changed the links, on a detached bounded context), and on Close when a refresh is still owed. The scheduler already reads a worker without a hash through the join, which now also covers a pending refresh. Measured on the reviewer's dense fixture: 100k replay 10.3s to 3.1s (one refresh 0.23s); 1M replay from a 30s timeout at 265k to 26.0s (one refresh 2.3s cold, the link insert itself is 15.7s of it); one removal on a 1M worker from 715ms to 1ms plus a 1.0s refresh. The single 1M refresh is still over a second, so the alternative is recorded here: a (workerId, actionSetVersion) cache key, bumped by the settle statement, removes the full-set read entirely at the price of one cache entry per worker instead of per identical set. UpdateOperatorWorkerActions had no caller and is deleted with its test. The e2e hash reads wait for the end-of-sequence refresh. Regression tests: TestSessionActionBudgetSpansSessions, TestConcurrentDeltasThroughHost (race), TestOperatorActionBudgetSpansWorkers, TestOperatorActionBudgetUnderConcurrentDeltas; TestSessionSupersededCloseDoesNotPauseSuccessor, TestSupersededCloseLeavesSuccessorAssignable, TestPauseWorkerForListenerIsFenced; TestHashActionsIsCanonical, TestWorkerActionHashEncodingIsUnambiguous; TestWorkerActionHashRefreshFollowsDeltas, TestSessionRefreshesActionHashPerWindow, TestOpenSessionRefreshesPendingHash. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * fix(operatorclient): a chunk staged before a replay is never sent after it Correctness R01. The flusher stages a chunk (moves it from pending to unacked) before it takes the stream's send lock. A reconnect that owned the lock meanwhile replayed the stream: a fresh worker got a snapshot of the desired set, then the stale staged chunk went out behind it and undid part of the snapshot (wire order [2 1]; the worker lost an action while Flush reported success), and a resumed worker got the chunk twice, once from the replay and once from the flusher. The queue now carries a stream generation, bumped under its mutex by every replay. A chunk is staged with the generation it saw, and the flusher's send, which runs under the same send lock as the replay, checks the generation first: if a replay ran in between, the chunk is not sent and counts as done, because the replay either resent it from unacked (resumed) or dropped it for the snapshot (fresh). Regression tests: TestOperatorSessionFreshReplayDropsStagedRemoval and TestOperatorSessionFreshReplayDropsStagedAdd (the server set diverged from the desired set and the wire carried [2 1]) and TestOperatorSessionResumedReplaySendsStagedChunkOnce. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * fix(hostgrpc): supervise the client session and keep tenant clients referenced Four findings on the out-of-process host. Correctness R05: a terminal failure of the client session (a revoked token, an Unauthenticated Register on reconnect) ended the client's receive loop and the host only logged it, leaving the tenant registered and unserved for good; the token source was never asked again. The host session now supervises its client session: when the client's action stream ends while the session is not closing, it closes the durable hub (open channels fail and their invocations retry), closes the old client session and reconnects through the host with jittered backoff (1s to 30s), which asks the source for the tenant's current token, verifies the tenant, restores the action set the session advertised and resumes delivery under the new registration. It gives up only on a failure no retry fixes (ErrNoToken, a tenant mismatch, ErrNotSupported), reported through the contract's Done and Err. Correctness R04: ReleaseTenant, token rotation and the host's Close closed the tenant's cached client unconditionally, so an old registration's teardown closed the connection a newer registration of the same tenant had just opened over. Cached clients are reference counted: a session holds one reference from connect to Close (or to giving up); eviction only stops the client being handed out, and an evicted client is closed by its last release. Host.Close still closes everything. Correctness R06: the session reported outcomes under the worker its first Open registered, even after a fresh reconnect gave the client a new worker because the engine no longer had the old one, so the engine refused the reports. Registration follows the client's current worker; the worker current at each delivered start is recorded per task run, and a report is stamped with it while it is still the current worker and with the current worker otherwise, since the engine registers a fresh worker exactly when it cannot resume the previous one. A report naming a worker the session used to be is rewritten the same way; a foreign id passes through for the engine to refuse. Performance R8: delivery called the handler serially straight off the client's action channel, so a start that blocked on the operator's flow control held every later message, cancels included, while the client's inbox grew without bound. Delivery now reads the channel eagerly: cancels go to the handler on their own goroutine at once, starts go through a bounded queue (twice the worker's slot units, at least 64) served by one goroutine, and a start that finds the queue full is refused with a retryable failure report instead of being held. Regression tests: TestSessionReopensAfterTerminalFailure, TestSessionGivesUpWithoutToken, TestSessionCloseDuringBackoff and, over a real listener, TestLoopbackSessionRecoversWithRotatedToken; TestReleaseTenantWaitsForOpenSessions; TestSessionReportsUnderCurrentWorker and TestLoopbackFreshReconnectReportsUnderNewWorker; TestCancelsBypassBlockedStarts. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * fix(repository): keep the action hash encoding and drop the hash migration The worker action hash keeps the per-id framing main has always used: sha256 over the sorted, deduplicated, lower-cased action ids, each followed by ";". hashActions in Go and ComputeWorkerActionHash in SQL compute it byte for byte the same way, so a worker created with an initial set and a worker built by deltas hash equal for the same set. The length-prefixed encoding is dropped, and with it the two "actionHash" rewrites in migration v1_0_154, which now only adds "actionCount" and backfills it for operator workers. A rolling deploy needs no hash step: a worker registered by an older engine keeps a hash the new engine computes the same way, and any hash the scheduler does not recognise is a group of its own, so distinct hashes never misroute. The same-tenant ambiguity of a ";" inside an id is closed at validation instead, in the commit that follows. The unambiguous-encoding test becomes a Go/SQL parity test over a fixed set of action sets, and the unit test pins the digest of a sorted set to sha256 of "svc:a;svc:b;". Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * fix(validator): reject semicolons in action ids ParseActionID refuses a ";" anywhere in an action id. The worker action hash frames each id with a semicolon, so an id carrying one could make two different action sets of the same tenant hash equal, and the scheduler treats equal hashes as equal sets. ParseActionID is the one function every registration path goes through: the actionId validator tag (worker registration and update, operator action deltas, workflow step actions and the worker list filter) calls it, and the v0 and v1 PutWorkflow handlers call it directly on each step action before the step is stored. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * refactor(repository): rename "actionCount" to "operatorActionCount" The column added by migration v1_0_154 counts the "_ActionToWorker" rows a worker holds so the per-operator action budget can be summed over the operator's workers instead of counting their links. Only operator workers are read and only they are backfilled; an SDK worker's count is zero until it registers and nothing reads it. The name says so. Semantics are unchanged: every path that links or unlinks actions maintains it, and the budget check sums it under the operator's lock. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * chore(generate): format the configuration table and align the operator protoc stamp The configuration-options table added on this branch was not prettier formatted, so the docs generator reflowed its column widths and both the generate diff check and the docs lint job failed on it. The table is now written the way prettier writes it. operator_grpc.pb.go carried a protoc v5.29.3 stamp from a local protoc; CI generates with v5.29.6, like every other generated file here. The stamp line is aligned by hand rather than regenerated with the local toolchain. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * fix(operatorsvc): admit orchestrator action ids on DAG operator sessions The DAG operator registers the orchestrator action ids of the tenant's DAG workflows, "<workflow>_orchestrator". They are not "service:verb" action ids on purpose: without a ":" no SDK or GRPC worker can register one. On main the operator wrote them through a dedicated repository method that never ran the actionId validator. On this branch the in-process host routes them through Session.ApplyDelta, which validated every delta with the actionId tag, so every refresh failed with InvalidArgument from the first one at engine start, the DAG operator's worker never held an action, and no multi-task workflow of a tenant with dag_operator enabled ever started. That is the failing python test (3.14, true, true) cell: test_cancellation, and the durable spawn-DAG tests, hang until the job times out. Delta validation is now scoped by the operator's kind. A DAG operator session admits orchestrator ids and nothing else; every other kind admits action ids and nothing else. A DAG operator is hosted in process only and its row is never upserted through a registration, so the kind is the engine's own claim and a remote operator cannot hold an orchestrator action. The id form lives in one place, repository.DAGOrchestratorActionId, which the workflow store builds with and the service checks against; the form has no ";" either. The hostinproc tests that seeded DAG rows with "dag:a" style ids had hidden the mismatch; they use orchestrator ids now, and an operatorsvc test reproduces the failure on both kinds. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * feat(repository): add operator leasing and worker limit exemption columns What an operator is and who keeps it alive are separate axes. v1_operator.kind stays what it is (DAG, the engine-internal DAG operator; GRPC, a contract operator hostable in process or out of process) and a new v1_operator.leasing enum says who keeps the row alive: MANAGED rows are assigned to a dispatcher by ClaimOperators and built from a factory inside the engine, SELF rows keep themselves alive through a Listen stream out of process or their own leaser in process. SERVERLESS is retired: the serverless operator is GRPC with SELF in both modes, and the migration folds any SERVERLESS rows into GRPC (the enum value stays; databases migrated by the serverless branch carry it). Rows are unique per (tenant, name, kind) for every kind, replacing the two partial indexes that only covered the kinds upserted by name. Duplicates are renamed, not deleted, before the index is built. Limit exemption is a hosting fact decided at worker creation rather than an attribute of the operator row, so "Worker"."exemptFromLimits" is added, backfilled for the DAG operator's workers, which the limit queries left out before. The queries switch to the column in a following commit. The migration is v1_0_159: belanger/serverless-operator, which stacks on this branch, owns v1_0_155 to v1_0_158. It runs in goose's transaction: the enum type is created here rather than extended, and only a value added by ALTER TYPE ... ADD VALUE is unusable in the transaction that added it. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * refactor(repository): exempt workers from limits by flag Whether a worker counts against the tenant's WORKER and WORKER_SLOT limits used to be derived from the kind of the operator it backs (DAG workers were exempt). That tied metering to the operator row, which no longer says who hosts the operator. It is now an explicit CreateWorkerOpts.ExemptFromLimits, stored on the row, that only the in-process operator host sets: every worker it creates is engine infrastructure that runs whether or not the tenant runs workers of its own, whatever kind or leasing its row has. Wire-registered workers, over OperatorService or from an SDK, are metered. The limit queries in workers.sql and tenant_limits.sql read the flag off the worker instead of joining v1_operator on kind, so what CreateNewWorker meters is what they count. CreateWorkerOpts.OperatorKind had no other reader and is removed. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * feat(operator): separate operator kind from leasing Registration and claiming now run on the leasing axis rather than the kind list. ClaimOperators claims every MANAGED row, whatever its kind; the claimer resolves the factory by kind, except for GRPC rows, which it resolves by the row's name through WithNamedFactory, since the kind only says the row is a contract operator and the name says which one the engine should build. A claimed row with no factory is logged once and left alone. UpsertGRPCOperator is replaced by UpsertOperator(tenant, name, kind, leasing) over the plain unique index; a repeat registration takes the leasing it names, so a row the engine was leasing that registers itself leaves the claim set on the next poll, and the other way round. operatorsvc.RegisterOpts carries Leasing; a named registration must say who keeps the row alive. Wire registration (grpcoperator Register, hostgrpc) is GRPC and SELF, and AuthorizeOperator, behind the hatchet-operator-id metadata, requires kind GRPC and leasing SELF: an engine-leased row can never be driven over the wire. operator.Identity gains Leasing, defaulting to SELF; hostinproc passes it through and marks every worker it creates exempt from limits. The DAG operator's row is created MANAGED, and a race creating it is tolerated now that the name is unique. The unused CreateOperatorWorker query and repository method, which predate the claimer registering workers through the operator service, are removed. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * refactor(operator): name the leasing manager The v1_operator column that says who keeps an operator alive is now leasing_manager, of type v1_operator_leasing_manager with the values SELF and DISPATCHER. DISPATCHER replaces MANAGED: the dispatcher claims the row through ClaimOperators and builds the operator from a factory, a SELF row keeps itself alive. The Go side follows: sqlcv1.V1OperatorLeasingManager, the LeasingManager field on Identity, RegisterOpts, CreateOperatorOpts and UpsertOperatorOpts, and the comments that described managed rows. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * chore(migrate): one migration for the gRPC operator The three migrations this branch added (the GRPC enum value and its partial index, operatorActionCount, then leasing and exemptFromLimits) collapse into 202609111300000_v1_0_154.sql. The file runs with NO TRANSACTION so the GRPC value is committed before anything uses it; the partial GRPC index is never created, only the (tenant, name, kind) unique index; and the SERVERLESS rewrite is gone, since that value never exists on this branch. The goose id is fifteen digits: main's 202609101223115_v1_0_153.sql is, and goose refuses a version that sorts below one already applied. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * Renumber the operator migration to v1_0_155 with a fourteen-digit goose id. Main remapped v1_0_153 to a fourteen-digit id and added v1_0_154, so this file moves to 155 and drops the note about matching a fifteen-digit version. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * Address review comments on the operator migration, queries and naming. Migration: the "Worker" columns are added in one re-runnable statement at the top so the table lock is released before the backfills, the header states that as the reason for running without a transaction, and the comments are cut down. Queries: the action hash is recomputed and stored by one RefreshWorkerActionHash query, and the fenced pause goes through UpdateWorker with an optional listener session id instead of its own query. UpdateWorker now also scopes by tenant. Naming: isExemptFromLimits, is_paused on OperatorPause and OperatorPauseAck, CancelTaskWithReason, LinkNewActionsToWorker and UnlinkActionsFromWorker, and the durable goroutines are a receive loop (engine side) and a send loop (client hub) instead of a pump. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * Realign struct literals after the isExemptFromLimits rename. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * Constrain DAG operators to dispatcher leasing and make the down migration atomic. v1_operator gets a check that a DAG row is leased by the dispatcher. The down migration is sent as one query, so Postgres runs it as a single transaction and the drops need no IF EXISTS. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * Serve the operator service over ConnectRPC. The Listen and DurableTask handlers take connect's bidi streams and run over a transport-free half so their tests keep driving fake streams; the operator id is read from the request header through the handler call info; errors are connect errors with the same codes. The dispatcher's operator stream session takes the stream's guarded sender and wraps assigned actions itself, replacing the wrap hook, and the durable session is entered through the dispatcher's DurableTaskWithReceive so the ownership check on the first message stays in the operator service. The operator client stays on grpc-go, and the operator e2e suite passes against the connect server. Also fixes the two admin trigger paths that main left on the removed status package (#4755 landed beside #4987). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK * Fix the load-tagged partition test for the two-value CreatePartitions. Main's #5002 changed the query to :one and left this test on the old signature. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
| Commit: | 376470c | |
|---|---|---|
| Author: | Alexander Belanger | |
Merge branch 'belanger/grpc-operator' into belanger/serverless-operator
| Commit: | d3ac62a | |
|---|---|---|
| Author: | Alexander Belanger | |
Address review comments on the operator migration, queries and naming. Migration: the "Worker" columns are added in one re-runnable statement at the top so the table lock is released before the backfills, the header states that as the reason for running without a transaction, and the comments are cut down. Queries: the action hash is recomputed and stored by one RefreshWorkerActionHash query, and the fenced pause goes through UpdateWorker with an optional listener session id instead of its own query. UpdateWorker now also scopes by tenant. Naming: isExemptFromLimits, is_paused on OperatorPause and OperatorPauseAck, CancelTaskWithReason, LinkNewActionsToWorker and UnlinkActionsFromWorker, and the durable goroutines are a receive loop (engine side) and a send loop (client hub) instead of a pump. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK
| Commit: | e640cb0 | |
|---|---|---|
| Author: | Alexander Belanger | |
Merge branch 'belanger/grpc-operator' Brings the ten #4910 review fixes and the seven round-2 fixes of the gRPC operator branch onto the serverless operator. Conflicts and adaptations: - pkg/operator/safeclient/safeclient.go: this branch's newTransport() and withDefaults() plumbing (shared by the SSRF and InsecureDestinations senders) with grpc-operator's HTTP/1.1-only transport settings (ForceAttemptHTTP2 off, empty TLSNextProto, ALPN http/1.1 alone, TLS 1.2 minimum) and its transport_test.go verbatim. - configuration-options.mdx: both branches' settings tables, the gRPC operator's row replacing this branch's shorter one. - cmd/hatchet-serverless-operator: hostgrpc.WithExchangeLogger for the local exchange, hostgrpc.New(WithTokenSource, WithLogger) returning an error. - Migration 20260911100000_v1_0_154.sql is renamed v1_0_157: goose orders it after this branch's 20260908110000..130000 (v1_0_154..156) by timestamp already, the rename keeps the version labels unique and in chain order. Its version number is unchanged. - CreateWorkerOpts.OperatorKind's oneof gains SERVERLESS, the kind the in-engine host registers under; grpc-operator's validator knew only HTTP_API, DAG and GRPC. - The serverless fakes implement Session.Done/Err and DurableChannel.ExpectEntry; the e2e process test builds the gRPC host through its options. - The runner supervises each registration's session: Done closing with an error means the host gave up for good (ErrNoToken, a tenant mismatch, ErrNotSupported; transient failures and token rotation the gRPC host recovers itself), so the registration is detached and closed and a new one opened while the tenant still owns units, with maintenance retrying an open that fails. Regression: TestRegistrationReopensAfterHostGivesUp. Metering: operator workers are now metered like SDK workers with only kind DAG exempt, so SERVERLESS workers count against WORKER and WORKER_SLOT in both hosting modes. Kept as merged; to be revisited. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK
| Commit: | 7e84168 | |
|---|---|---|
| Author: | Alexander Belanger | |
feat(operator): pause is a message on the Listen stream The Listen stream is stateful, so the pause belongs on it rather than in a unary RPC. The client sends OperatorListenRequest.pause and waits for the OperatorPauseAck the server answers with once the pause is committed; from that ack on nothing is delivered on the stream. The server commits the pause in this order: the dispatcher session stops delivering first, then the row is written, then the ack goes out, so an action the scheduler assigned before it observed the pause is returned to the queue the way a failed send is (subscribedWorker.paused, honoured for starts and not for cancels, which a draining operator may still need). Resuming lifts the pause in the opposite order and is acknowledged the same way. The pause is stream state: Register clears it when it resumes a worker, so a paused client sends it again on every new stream, right after start and before its deltas. The unary PauseWorker RPC and its messages are gone; pkg/client/operatorclient keeps Pause, Resume and the pause-then-drain Close over the stream, and the in-process host pauses through the same operatorsvc.Session.Pause, so both delivery kinds stop the same way. The e2e pause scenario now asserts that nothing reaches the operator after the pause ack and that the run held back runs once after the resume. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK
| Commit: | 579bb34 | |
|---|---|---|
| Author: | Alexander Belanger | |
docs(operator.proto): say why the heartbeat rides the Listen stream SDK workers heartbeat out of stream because the runtimes they target can block their event loop while a task runs and would starve an in-stream heartbeat. Operators are written against the Go SDK, which is not subject to that, so the heartbeat stays on the stream it keeps alive. The reasoning now lives on the OperatorHeartbeat message so it travels with the protocol. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK
| Commit: | 73267fa | |
|---|---|---|
| Author: | Alexander Belanger | |
Merge branch 'belanger/grpc-operator' into belanger/serverless-operator # Conflicts: # internal/services/dispatcher/operator_sessions.go # internal/services/dispatcher/operator_sessions_test.go
| Commit: | ed37b94 | |
|---|---|---|
| Author: | Alexander Belanger | |
feat(grpcoperator): PauseWorker An operator that wants to shut down cleanly needs the scheduler to stop assigning to its worker before it stops answering, and it needs to know the pause is committed before it starts draining. A message on the Listen stream would need an ack of its own and would not work while that stream is being torn down or reconnected, so pausing is its own unary call. OperatorService.PauseWorker takes {worker_id, paused}: the same call pauses and unpauses, which is what a worker that is resumed rather than replaced needs. It is authorised like SendStepActionEvent, by the operator metadata plus worker ownership. Register already clears the pause when it resumes a worker, so an operator that paused to drain and then crashed comes back assignable. The proto is unreleased, so this is an addition with no compatibility shim. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK
| Commit: | e97d912 | |
|---|---|---|
| Author: | Alexander Belanger | |
| Committer: | Alexander Belanger | |
fix(grpcoperator): acknowledge action deltas and retain them until committed The Listen protocol had no delta acknowledgement. A delta the stream had accepted but the engine had not committed (stream ended before the apply, database error, rejected chunk) was dropped by the client as soon as the send returned, and a resumed worker never learned about it: the client only replayed its desired set for a fresh worker, and repeating the same AddActions was a no-op against the desired set. Listen now returns a stream of OperatorListenResponse, a oneof of the dispatcher's AssignedAction and an OperatorActionsAck. Each delta carries a positive, strictly increasing sequence; the engine applies deltas in stream order and acks a sequence once the change is committed, and an ack for N confirms every lower sequence. A delta that cannot be applied, or whose ack cannot be written, ends the stream instead of leaving it unconfirmed. The dispatcher session wraps actions into the envelope and writes acks through the same per-stream send serialisation as the fan-out. The client keeps every sent chunk until its ack arrives and replays the unacked chunks in order on every reconnect. When the reconnect did not resume the worker it replaces the queue with a snapshot of the desired set, taken under the same lock AddActions and RemoveActions use, so a removal that races the replay is queued relative to the snapshot instead of being coalesced away with an add that was never sent. Replay runs as the reconnecting stream's replay callback, under sendMu before the stream is published, so no chunk is sent in between. A chunk that waits longer than the ack timeout hangs the stream up so the reconnect replays it. Flush now waits for the acks, so a nil Flush means the engine committed the set. Because Flush is called before Actions by the serverless link, the session runs its receive and heartbeat loops from connect until Close; the receive loop demultiplexes acks and actions, buffering actions in an inbox that the consumer Actions attaches drains. Starting the loops and attaching the consumer add to the WaitGroup under the mutex Close takes, which removes the Actions/Close race on the WaitGroup; Actions after Close is refused. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK
| Commit: | 417f8c3 | |
|---|---|---|
| Author: | Alexander Belanger | |
| Committer: | Alexander Belanger | |
fix(grpcoperator): acknowledge action deltas and retain them until committed The Listen protocol had no delta acknowledgement. A delta the stream had accepted but the engine had not committed (stream ended before the apply, database error, rejected chunk) was dropped by the client as soon as the send returned, and a resumed worker never learned about it: the client only replayed its desired set for a fresh worker, and repeating the same AddActions was a no-op against the desired set. Listen now returns a stream of OperatorListenResponse, a oneof of the dispatcher's AssignedAction and an OperatorActionsAck. Each delta carries a positive, strictly increasing sequence; the engine applies deltas in stream order and acks a sequence once the change is committed, and an ack for N confirms every lower sequence. A delta that cannot be applied, or whose ack cannot be written, ends the stream instead of leaving it unconfirmed. The dispatcher session wraps actions into the envelope and writes acks through the same per-stream send serialisation as the fan-out. The client keeps every sent chunk until its ack arrives and replays the unacked chunks in order on every reconnect. When the reconnect did not resume the worker it replaces the queue with a snapshot of the desired set, taken under the same lock AddActions and RemoveActions use, so a removal that races the replay is queued relative to the snapshot instead of being coalesced away with an add that was never sent. Replay runs as the reconnecting stream's replay callback, under sendMu before the stream is published, so no chunk is sent in between. A chunk that waits longer than the ack timeout hangs the stream up so the reconnect replays it. Flush now waits for the acks, so a nil Flush means the engine committed the set. Because Flush is called before Actions by the serverless link, the session runs its receive and heartbeat loops from connect until Close; the receive loop demultiplexes acks and actions, buffering actions in an inbox that the consumer Actions attaches drains. Starting the loops and attaching the consumer add to the WaitGroup under the mutex Close takes, which removes the Actions/Close race on the WaitGroup; Actions after Close is refused. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK
| Commit: | 301dfab | |
|---|---|---|
| Author: | Alexander Belanger | |
feat(serverless): define the endpoint wire contract in protobuf api-contracts/v1/serverless.proto is the single source for every message the serverless operator exchanges with an endpoint: the healthcheck request and response, the non-durable trigger request and its error override, and the durable websocket frame (a oneof of first, request, response, error and done). Messages travel as protojson, so the Go operator and the TypeScript package read and write the same encoding from the same definitions. The Go bindings are generated into internal/services/shared/proto/v1 with the same dual proto path as operator.proto, since both reference .AssignedAction from the package-less dispatcher.proto. The TypeScript bindings live in sdks/typescript-serverless, the future @hatchet-dev/serverless package, which for now holds only the generation script (ts-proto pinned to the SDK's version and options) and its output under src/generated/proto. The SDK's own protoc script skips serverless.proto the way it skips operator.proto. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK
| Commit: | 7cb0b07 | |
|---|---|---|
| Author: | Alexander Belanger | |
| Committer: | abelanger5 | |
feat(engine): gRPC operator service for out-of-process operators Adds v1.OperatorService so an operator can run outside the engine and talk to it over gRPC with a tenant-scoped token: - Register (unary) upserts the operator by (tenant, name, kind = GRPC) and creates the worker for the connection, or resumes a previous worker of the same operator. Registration carries no workflows and no actions. - Listen (bidi) activates the worker for the lifetime of the stream: the first message is start{worker_id}, then heartbeats and action deltas flow in and AssignedAction messages flow out straight from the dispatcher fan-out. Deltas are applied incrementally (at most 1000 ids per message) and the scheduler is notified at most once per second while they arrive. - SendStepActionEvent and DurableTask delegate to the dispatcher; the durable stream is wrapped so the first register_worker message is checked for worker ownership before the dispatcher sees it. - Every RPC after Register, including Listen, carries hatchet-operator-id metadata validated against the token's tenant through a cached lookup. Repository: CreateWorkerOpts.OperatorId flows into the existing CreateWorker query and skips WORKER / WORKER_SLOT metering, since operator workers are excluded from those limit counts. AddWorkerActions / RemoveWorkerActions apply deltas in one transaction under the worker row lock and maintain the action hash as the XOR of per-action sha256 digests, so equal sets hash equal in any order and a delta costs O(chunk). The proto keeps importing the package-less dispatcher.proto for the shared action types (no SDK regeneration). SERVER_GRPC_OPERATORS_ENABLED (default false) gates service registration. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK
| Commit: | 44864ac | |
|---|---|---|
| Author: | Alexander Belanger | |
| Committer: | Alexander Belanger | |
feat(engine): gRPC operator service for out-of-process operators Adds v1.OperatorService so an operator can run outside the engine and talk to it over gRPC with a tenant-scoped token: - Register (unary) upserts the operator by (tenant, name, kind = GRPC) and creates the worker for the connection, or resumes a previous worker of the same operator. Registration carries no workflows and no actions. - Listen (bidi) activates the worker for the lifetime of the stream: the first message is start{worker_id}, then heartbeats and action deltas flow in and AssignedAction messages flow out straight from the dispatcher fan-out. Deltas are applied incrementally (at most 1000 ids per message) and the scheduler is notified at most once per second while they arrive. - SendStepActionEvent and DurableTask delegate to the dispatcher; the durable stream is wrapped so the first register_worker message is checked for worker ownership before the dispatcher sees it. - Every RPC after Register, including Listen, carries hatchet-operator-id metadata validated against the token's tenant through a cached lookup. Repository: CreateWorkerOpts.OperatorId flows into the existing CreateWorker query and skips WORKER / WORKER_SLOT metering, since operator workers are excluded from those limit counts. AddWorkerActions / RemoveWorkerActions apply deltas in one transaction under the worker row lock and maintain the action hash as the XOR of per-action sha256 digests, so equal sets hash equal in any order and a delta costs O(chunk). The proto keeps importing the package-less dispatcher.proto for the shared action types (no SDK regeneration). SERVER_GRPC_OPERATORS_ENABLED (default false) gates service registration. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK
| Commit: | 5ba3d7a | |
|---|---|---|
| Author: | mrkaye97 | |
feat: add cancellation reason
| Commit: | f188d07 | |
|---|---|---|
| Author: | Julius Park | |
initial commit
| Commit: | 67b18fc | |
|---|---|---|
| Author: | abelanger5 | |
| Committer: | GitHub | |
feat(engine): dynamic per-group max runs via CEL expression (#4863)
| Commit: | d6a6d5f | |
|---|---|---|
| Author: | Alexander Belanger | |
| Committer: | Alexander Belanger | |
feat: dynamic per-group max runs via CEL expression Adds Concurrency.max_runs_expression: a CEL expression over task input that computes the max runs for that task's concurrency group, so different groups (e.g. pricing tiers) get different limits from one strategy. Supported only by the in-memory concurrency index; dynamic strategies always take that path. The expression evaluates at task-insert time next to the key expression (the scheduler never sees task input) and the resulting value rides on the concurrency slot through the insert triggers, WAL payloads, and hydration. A group's effective limit is the value evaluated for its most recently created task: the in-memory subQueue applies INSERT observations guarded by task_inserted_at, so replays and retry re-inserts of older tasks cannot regress a newer task's limit. The subQueue's begin/rollback scope snapshots the limit so a failed flush restores it with the membership. Evaluation failures (non-integer or non-positive results) fail the task like key-expression failures. Registration statically rejects expressions that can never return an integer and workflow-level entries on the old parent/child DAG path. Tenant-scoped strategies carry the expression through the definition-sync trigger, checksum stripping, and strategyDiffers-driven manager rebuilds. All schema changes live in their own migration; SDK surfaces ship on a separate branch (this change carries only the regenerated contract bindings). Covered by unit tests on the subQueue guard and decide behavior under raised/lowered limits, plus integration tests through the real triggers and outbox, including the replay-cannot-regress path and chained-slot hand-off. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0174j12vY8XiKEkkoZqNnC89
| Commit: | 67461cf | |
|---|---|---|
| Author: | abelanger5 | |
| Committer: | GitHub | |
feat(engine): shared concurrency strategies (#4845)
| Commit: | 02796b0 | |
|---|---|---|
| Author: | Alexander Belanger | |
| Committer: | Alexander Belanger | |
feat: dynamic per-group max runs via CEL expression Adds Concurrency.max_runs_expression: a CEL expression over task input that computes the max runs for that task's concurrency group, so different groups (e.g. pricing tiers) get different limits from one strategy. Supported only by the in-memory concurrency index; dynamic strategies always take that path. The expression evaluates at task-insert time next to the key expression (the scheduler never sees task input) and the resulting value rides on the concurrency slot through the insert triggers, WAL payloads, and hydration. A group's effective limit is the value evaluated for its most recently created task: the in-memory subQueue applies INSERT observations guarded by task_inserted_at, so replays and retry re-inserts of older tasks cannot regress a newer task's limit. The subQueue's begin/rollback scope snapshots the limit so a failed flush restores it with the membership. Evaluation failures (non-integer or non-positive results) fail the task like key-expression failures. Registration statically rejects expressions that can never return an integer and workflow-level entries on the old parent/child DAG path. Tenant-scoped strategies carry the expression through the definition-sync trigger, checksum stripping, and strategyDiffers-driven manager rebuilds. All schema changes live in their own migration; SDK surfaces ship on a separate branch (this change carries only the regenerated contract bindings). Covered by unit tests on the subQueue guard and decide behavior under raised/lowered limits, plus integration tests through the real triggers and outbox, including the replay-cannot-regress path and chained-slot hand-off. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0174j12vY8XiKEkkoZqNnC89
| Commit: | 7133e73 | |
|---|---|---|
| Author: | Alexander Belanger | |
refactor: address review feedback on the backend surface - rename the proto discriminator to is_tenant_scoped and regenerate bindings - rename the unique constraint to the _uq convention - upsert a workflow's tenant strategy definitions in one statement instead of a per-definition loop (row locks are taken in name order inside the statement) - ORDER BY in the tenant stale sweep for deterministic lock acquisition - clarify the 25-hour staleness window and the non-CONCURRENTLY index build - drop a stray blank line in the proto Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0174j12vY8XiKEkkoZqNnC89
| Commit: | fd9af62 | |
|---|---|---|
| Author: | Alexander Belanger | |
| Committer: | Alexander Belanger | |
feat: dynamic per-group max runs via CEL expression Adds Concurrency.max_runs_expression: a CEL expression over task input that computes the max runs for that task's concurrency group, so different groups (e.g. pricing tiers) get different limits from one strategy. Supported only by the in-memory concurrency index; dynamic strategies always take that path. The expression evaluates at task-insert time next to the key expression (the scheduler never sees task input) and the resulting value rides on the concurrency slot through the insert triggers, WAL payloads, and hydration. A group's effective limit is the value evaluated for its most recently created task: the in-memory subQueue applies INSERT observations guarded by task_inserted_at, so replays and retry re-inserts of older tasks cannot regress a newer task's limit. The subQueue's begin/rollback scope snapshots the limit so a failed flush restores it with the membership. Evaluation failures (non-integer or non-positive results) fail the task like key-expression failures. Registration statically rejects expressions that can never return an integer and workflow-level entries on the old parent/child DAG path. Tenant-scoped strategies carry the expression through the definition-sync trigger, checksum stripping, and strategyDiffers-driven manager rebuilds. Covered by unit tests on the subQueue guard and decide behavior under raised/lowered limits, integration tests through the real triggers and outbox (including the replay-cannot-regress path), and Go e2e tests against a live engine. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0174j12vY8XiKEkkoZqNnC89
| Commit: | ccf6d8a | |
|---|---|---|
| Author: | Alexander Belanger | |
feat: explicit tenant_scoped flag and ordering-conflict rejection Concurrency entries gain a tenant_scoped boolean as the explicit discriminator: name alone no longer implies tenant scope (leaving names usable for workflow-scoped entries later), and expression is required on every entry. Reference-only entries are gone; every task declaring a tenant-scoped strategy carries its full definition, upserted in place. Registration now rejects chains that order the same tenant-scoped strategies inconsistently, since tasks hold earlier slots in their chain while queued on later ones and conflicting orders can deadlock. The per-step ordered chains already exist implicitly as the referencing rows' creation order, so the check aggregates them per latest workflow version (excluding the workflow being re-registered, so reorders stay possible), combines them with the incoming chains, and runs cycle detection over the pairwise precedence graph, which also catches cycles spread across more than two chains. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0174j12vY8XiKEkkoZqNnC89
| Commit: | c6dd784 | |
|---|---|---|
| Author: | Alexander Belanger | |
refactor: tenant-scoped concurrency as ordered Concurrency entries Replace the dedicated shared-concurrency proto surface with an optional name on the Concurrency message: entries in CreateTaskOpts.concurrency and CreateWorkflowVersionRequest.concurrency_arr are processed in array order, so a tenant-scoped entry may come before or after a workflow-scoped one. A named entry with an expression defines (or updates in place) the tenant strategy as part of registration; a name-only entry references one defined elsewhere. The shared_concurrency field, shared_concurrency_defs, and the SharedConcurrencyDef message are removed. Ordering: strategy rows are created in array order, and the per-step chain follows creation order (ascending row id); rows referencing a tenant strategy resolve to the tenant strategy id only after sorting, so the tenant strategy's own id never perturbs the declared order. Workflow-level tenant-scoped entries are supported on the new DAGs-as-durable-tasks path by attaching them to the DAG orchestrator task's concurrency (workflow-scoped entries keep the parent fan-out, which always gates first); single-task workflows inherit them via the existing merge. On the old parent/child path they are rejected loudly. The workflow version checksum hashes tenant-scoped entries by name and position only, so re-registering an unchanged workflow with a changed definition reuses the version while still updating the strategy in place (definitions are upserted before the checksum early-return). SDKs: types.Concurrency (Go) and the Concurrency options (TS) gain an optional name; Python's SharedConcurrency now maps onto the Concurrency proto. All three SDKs pass a single ordered concurrency list through. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0174j12vY8XiKEkkoZqNnC89
| Commit: | be0aede | |
|---|---|---|
| Author: | Alexander Belanger | |
| Committer: | Alexander Belanger | |
feat: tenant-scoped shared concurrency strategies Allow v1 concurrency strategies that are not workflow-scoped: a shared strategy is defined per tenant (unique name) in the new v1_tenant_concurrency table and referenced by tasks across different workflows, so they consume the same concurrency limit. Only the in-memory concurrency index supports these strategies. Schema: - new v1_tenant_concurrency table drawing ids from v1_step_concurrency's identity sequence, so strategy ids stay unique across both tables (slots, outbox topics, leases, and advisory locks key on the bare id); the borrow is documented at both definition sites since Postgres records no dependency for it - v1_step_concurrency keeps its shape and gains a nullable tenant_strategy_id: a row referencing a tenant strategy carries a point-in-time copy of the definition, but reads always resolve through the referenced row - the slot-insert reactivation trigger gains a tenant-strategy branch Engine: - strategy definitions travel as SharedConcurrencyDef entries on CreateWorkflowVersionRequest and are upserted inside the workflow-put transaction before steps are created; there is no standalone registration RPC - definitions are excluded from the workflow version checksum: they are tenant-level state, not workflow shape, so changing one updates the strategy in place without minting new workflow versions (the names steps reference remain part of the hash) - referencing rows never get a ConcurrencyManager: the lease listing returns the tenant strategy instead (zero-uuid workflow columns mark tenant scope in the shared descriptor type) - tenant strategies always use the in-memory index, skip the workflow-version active check, retire via a tenant stale sweep, and their managers are rebuilt when a definition changes, so re-registering with new settings propagates within one lease-poll cycle - per-step strategy lookup resolves the tenant strategy's id and definition, so task slots carry the shared id - fix a latent scheduler panic: RunConcurrencyStrategy returned a nil result for strategies with no runnable branch (e.g. NONE), which notifyAfterConcurrency dereferenced SDKs (Go, Python, TypeScript): - shared strategies are declared in task-level concurrency options; definitions collected from tasks ride in the workflow put request, and name-only entries reference strategies registered elsewhere - e2e tests in all three SDKs cover cross-workflow serialization, both directions of the mixed inline+shared case, and an overlap control Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0174j12vY8XiKEkkoZqNnC89
| Commit: | 388746c | |
|---|---|---|
| Author: | Alexander Belanger | |
| Committer: | Alexander Belanger | |
feat: tenant-scoped shared concurrency strategies Allow v1 concurrency strategies that are not workflow-scoped: a shared strategy is registered per tenant (unique name) in the new v1_tenant_concurrency table and referenced by tasks across different workflows, so they consume the same concurrency limit. Only the in-memory concurrency index supports these strategies. Schema: - new v1_tenant_concurrency table drawing ids from v1_step_concurrency's identity sequence, so strategy ids stay unique across both tables (slots, outbox topics, leases, and advisory locks key on the bare id) - v1_step_concurrency keeps its shape and gains a nullable tenant_strategy_id: a row referencing a tenant strategy carries a point-in-time copy of the definition, but reads always resolve through the referenced row - the slot-insert reactivation trigger gains a tenant-strategy branch Engine: - referencing rows never get a ConcurrencyManager: the lease listing returns the tenant strategy instead (zero-uuid workflow columns mark tenant scope in the shared descriptor type) - tenant strategies always use the in-memory index, skip the workflow-version active check, retire via a tenant stale sweep, and their managers are rebuilt when a definition changes, so re-registering with new settings propagates within one lease-poll cycle - per-step strategy lookup resolves the tenant strategy's id and definition, so task slots carry the shared id - strategy definitions travel on CreateWorkflowVersionRequest and are upserted inside the workflow-put transaction before steps are created; the standalone PutSharedConcurrencyStrategy RPC remains for tuning limits without a deploy - fix a latent scheduler panic: RunConcurrencyStrategy returned a nil result for strategies with no runnable branch (e.g. NONE), which notifyAfterConcurrency dereferenced SDKs (Go, Python, TypeScript): - shared strategies are declared in task-level concurrency options; definitions collected from tasks ride in the workflow put request, and name-only entries reference strategies registered elsewhere - e2e tests in all three SDKs cover cross-workflow serialization, both directions of the mixed inline+shared case, and an overlap control Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0174j12vY8XiKEkkoZqNnC89
| Commit: | e32f645 | |
|---|---|---|
| Author: | Julius Park | |
| Committer: | GitHub | |
feat: `CANCEL_QUEUED_EXCEPT_NEWEST`/`CANCEL_QUEUED_EXCEPT_OLDEST` concurrency strategies (#4793)
| Commit: | 7bccc19 | |
|---|---|---|
| Author: | Julius Park | |
initial commit
| Commit: | 208bbd1 | |
|---|---|---|
| Author: | mrkaye97 | |
| Committer: | mrkaye97 | |
fix: claude attempt at a fix
| Commit: | 914d9e4 | |
|---|---|---|
| Author: | mrkaye97 | |
chore: clean up sdk changes
| Commit: | 8b34cfb | |
|---|---|---|
| Author: | mrkaye97 | |
Merge branch 'main' into fix--durable-satisfied-callback-order-gurantees
| Commit: | e667a4e | |
|---|---|---|
| Author: | Julius Park | |
| Committer: | GitHub | |
feat: Batch flush (#3975)
| Commit: | 3c53ad3 | |
|---|---|---|
| Author: | Julius Park | |
| Committer: | Julius Park | |
Merge branch 'main' into feat--batch-flush-redux
| Commit: | 939ae47 | |
|---|---|---|
| Author: | matt | |
| Committer: | GitHub | |
feat(engine): terminal status-based idempotency key release (#4460) * feat: queue idempotency release * feat: col in db * feat: proto * feat: wire it all up * fix: query * feat: python impl * feat: e2e test * fix: improve query * fix: cron input test * fix: small sleep for flakiness * fix: test * test: extend idempotency tests * feat: other sdks * docs: add notes on idempotency strategies * docs: more status details * fix: move release later * fix: clean up query * test: python tests * fix: don't need to join to Step anymore * chore: gen * chore: gen / i hate this * chore: gen
| Commit: | 4c9f96e | |
|---|---|---|
| Author: | mrkaye97 | |
feat: python impl
| Commit: | 63496ef | |
|---|---|---|
| Author: | mrkaye97 | |
feat: wire it all up
| Commit: | 3ac5746 | |
|---|---|---|
| Author: | mrkaye97 | |
feat: proto
| Commit: | ba9baec | |
|---|---|---|
| Author: | Julius Park | |
merge
| Commit: | 34c8b92 | |
|---|---|---|
| Author: | matt | |
| Committer: | GitHub | |
feat(engine): idempotency (#4045) * fix: small optimization, no need to query if there are no claims * feat: add idempotency key expr to proto, workflow version * feat: wiring * fix: wiring to db * feat: add ttl col * chore: gen a whole bunch of python * feat: wire up idempotency config on the sdk * fix: engine wiring * fix: py type * chore: revert worker changes * feat: new example * feat: fix migration, add cols * feat: wiring * feat: wire up idempotency to the triggerTuple * feat: initial engine wiring * fix: len * fix: types * feat: add test * chore: rm unused deps * fix: use the v1 trigger endpoint on python * fix: make the python test do a real thing * feat: first pass bypassing mq fallback for idempotency * fix: remove todo, return the same id * feat: wire up existing run ids * fix: use test run id to dedupe across test runs * fix: improve test a bit * chore: lock * fix: add grpcio types * fix: drive by fix for a flaky test caused by event condition race * fix: not found behavior on the get_details method * fix: revert python changes * fix: handle collisions in the v1 ingest endpoint :facepalm: * chore: gen * chore: add back some comments to shrink diff * chore: python version, changelog * fix: use the correct tx * fix: couple query bugs * fix: couple more event changes * feat: idempotency test with events * fix: type issue * fix: tests, lint * fix: test cleanup * fix: handle deduplication in the query * fix: test, gen * fix: conflict * fix: add prefix to idempotency key claim, clean up a tiny bit * fix: migration name, add idempotency key to cel eval failure source enum * fix: pr comments * chore: gen * feat: snippets, initial doc work * chore: vibe code other sdks * chore: vendor the status proto for js * fix: go docs * fix: ruby simplification * chore: changelogs * fix: ruby ci * fix: more test fixes * chore: gen * chore: gen again * fix: tests * chore: gen * chore: changelogs * chore: lockfile * chore: lint * chore: appease the cop * chore: fix copilot comments * chore: gen * chore: migration version * chore: migration version * fix: ordering to prevent deadlocks * fix: add validation, add skeleton for handling collisions on bulk trigger * feat: add key to v1_task * feat: wiring * feat: add idempotency keys to runs, dags, etc. tables * feat: wire up sending idempotency back over the api to the dash * feat: wire up idempotency filter and column on the fe * feat: return partial success on bulk trigger via an error * feat: wrap bulk triggers * feat: add test * feat: go migration for indexes * fix: order * fix: down * fix: weird diff * fix: always create trigger writer * chore: gen other sdks * fix: queries * fix: param ordering * chore: gen * chore: prettier * fix: merge * fix: migration * chore: lint * chore: bundle * chore: gen changelogs * feat: add TTL-based idempotency config + ABC to allow for better extensibility for other strategies * chore: other sdks, gen * chore: lint * fix: rubocop * fix: rm generic * chore: schema
| Commit: | 08ffdf6 | |
|---|---|---|
| Author: | mrkaye97 | |
feat: return partial success on bulk trigger via an error
| Commit: | fed9fa4 | |
|---|---|---|
| Author: | Julius Park | |
refactor so that messages are sent in one shot
| Commit: | 94f1c92 | |
|---|---|---|
| Author: | mrkaye97 | |
Merge branch 'main' into mk/feat-idempotency
| Commit: | a441554 | |
|---|---|---|
| Author: | mrkaye97 | |
chore: rm unused proto change
The documentation is generated from this commit.
| Commit: | 02f6fd0 | |
|---|---|---|
| Author: | Gabe Ruttner | |
chore order of things
| Commit: | 115306a | |
|---|---|---|
| Author: | Gabe Ruttner | |
comments
| Commit: | e520b06 | |
|---|---|---|
| Author: | Gabe Ruttner | |
refactor: more feedback and clarified naming
| Commit: | e8d2ba7 | |
|---|---|---|
| Author: | Gabe Ruttner | |
ack
| Commit: | a1a84b4 | |
|---|---|---|
| Author: | Gabe Ruttner | |
chore: refactor rate limit stuff
| Commit: | 10301e1 | |
|---|---|---|
| Author: | Gabe Ruttner | |
feat: separate stream proto
| Commit: | 91d1343 | |
|---|---|---|
| Author: | Gabe Ruttner | |
chore: easy feedback
| Commit: | 151fd44 | |
|---|---|---|
| Author: | Gabe Ruttner | |
Merge branch 'main' into feat--hatchet-v2-grpc-api
| Commit: | ac147cb | |
|---|---|---|
| Author: | mrkaye97 | |
feat: sql, move some logic over from my old branch
| Commit: | 91549f1 | |
|---|---|---|
| Author: | Gabe Ruttner | |
fix change og
| Commit: | bd482af | |
|---|---|---|
| Author: | Gabe Ruttner | |
Merge branch 'main' into fix--durable-satisfied-callback-order-gurantees
| Commit: | 1963828 | |
|---|---|---|
| Author: | Julius Park | |
Merge branch 'main' into feat--batch-flush-redux
| Commit: | 15addec | |
|---|---|---|
| Author: | matt | |
| Committer: | GitHub | |
fix(engine): correctly raise error in durable parent when child fails (#4154) * fix: propagate error back to the sdk * fix: python sdk * feat: add is_failure and error_message to the proto * feat: engine changes for wiring failure flag + error msg through * chore: other sdks * chore: cop * chore: nullif on write * fix: feedback * feat: e2e test * feat: tests * fix: tests * chore: lint * fix: test * chore: gen * chore: changelog sync * chore: lint ugh I hate our ci * fix: add tasks to worker * chore: gen * chore: versions, changelogs * chore: bundle * fix: rework test to fail on cloud * fix: autocomplete :facepalm: * chore: black * chore: gen
The documentation is generated from this commit.
| Commit: | 2e0f991 | |
|---|---|---|
| Author: | Gabe Ruttner | |
fix: track satisfied order
| Commit: | 8d37fa8 | |
|---|---|---|
| Author: | mrkaye97 | |
fix: move graph creation to the engine
| Commit: | f29a397 | |
|---|---|---|
| Author: | mrkaye97 | |
feat: almost working e2e PoC
| Commit: | 552a47e | |
|---|---|---|
| Author: | Gabe Ruttner | |
separate register and stream
| Commit: | ab7d578 | |
|---|---|---|
| Author: | Gabe Ruttner | |
feat: proposed protos
| Commit: | 0fb5bf6 | |
|---|---|---|
| Author: | Julius Park | |
add result broadcasting
| Commit: | 0bdfc1c | |
|---|---|---|
| Author: | mrkaye97 | |
chore: vendor the status proto for js
| Commit: | 17e5cbc | |
|---|---|---|
| Author: | Julius Park | |
rename proto fields
| Commit: | 4942ad4 | |
|---|---|---|
| Author: | mrkaye97 | |
feat: wire up existing run ids
| Commit: | 564d867 | |
|---|---|---|
| Author: | Julius Park | |
| Committer: | Julius Park | |
wip merge
| Commit: | 247dd6c | |
|---|---|---|
| Author: | mrkaye97 | |
feat: add ttl col
| Commit: | 09b749e | |
|---|---|---|
| Author: | mrkaye97 | |
feat: add idempotency key expr to proto, workflow version