Proto commits in sozu-proxy/sozu

These commits are when the Protocol Buffers files have changed: (only the last 100 relevant commits are shown)

Commit:2c3b43e
Author:Florentin DUBOIS
Committer:GitHub

feat(http): opt-in HTTP/2 prior knowledge (h2c) on HTTP listeners (#1917) `HttpListener` refused cleartext HTTP/2: a client opening with the `PRI * HTTP/2.0` preface got a 400, so cleartext gRPC (e.g. the Kubernetes Gateway API `GRPCRoute` conformance suite, which dials with insecure credentials) could not be served. Add a per-listener `h2c_prior_knowledge` option (HttpListenerConfig field 38, UpdateHttpListenerConfig field 44, TOML key, `listener http add/update` CLI flags), off by default. When on, a new session first goes through an `H2cPrefaceSniff` state (`lib/src/protocol/h2c.rs`) that peeks up to 24 bytes without consuming them: the full preface makes it an HTTP/2 mux frontend with the listener's H2 settings, as an ALPN `h2` connection is on HTTPS; the first diverging byte makes it HTTP/1.x; a strict prefix waits within `request_timeout`. The mux reads the peeked bytes itself, and the upgrade re-arms READABLE and WRITABLE since peeking raises no new edge. It composes with `expect_proxy` (PROXY header, then sniff). The scheme stays `http` and `tls_alpn` stays unset. `Upgrade: h2c` stays unsupported. Such a listener is held to the H2 minimum `buffer_size` at configuration load, like an HTTPS listener advertising `h2`. New metrics `http.h2c.prior_knowledge` and `http.upgrade.h2c.failed`. Closes #1911 Signed-off-by: Florentin Dubois <florentin.dubois@clever.cloud>

Commit:660ce27
Author:Florentin DUBOIS
Committer:GitHub

feat(router): add a REGEX_PATH rule matched against the path alone (#1916) * feat(router): add a REGEX_PATH rule matched against the path alone A `REGEX` path rule is matched against the request path together with its query string, so `/api/.+` matches `/api/?x=1` and `^/api/v2$` does not match `/api/v2?x=1`. There was no way to match a regex against the path only, which is what the Gateway API HTTPRoute `RegularExpression` path match (and Traefik `PathRegexp`, Envoy `safe_regex`) expects. Add a new path rule kind, `REGEX_PATH` (`PathRuleKind::REGEX_PATH = 3`, TOML `path_type = "REGEX_PATH"`, CLI `--path-regex-path`). It is anchored and compiled exactly like `REGEX` and ranks as `REGEX` does, but it is tested against the request-target up to its first `?`. `REGEX`, `PREFIX` and `EQUALS` are unchanged. With `rewrite_path`, a `REGEX_PATH` frontend's `$PATH[n]` captures, `$PATH[0]` included, come from the path alone. The original `?query` is appended to the rewritten path unless that path carries a `?` of its own, and a redirect `Location` follows it. The frontend gets its own `ConfigState` key letter (`Q`), so a `REGEX` and a `REGEX_PATH` frontend on the same pattern don't collide. Library API: `PathRule::from_cli_options` takes a fourth argument, and `PathRule` (sozu-lib) and `PathRuleType` (sozu-command-lib) gain a `RegexPath` variant. A worker that predates the kind refuses such a frontend with `RouterError::InvalidPathRule`. Fixes #1909 Signed-off-by: Florentin Dubois <florentin.dubois@clever.cloud> * docs(router): drop a public doc link to the private split_query `RUSTDOCFLAGS=-D warnings cargo doc --all-features` (CI msrv-full) refused the intra-doc link from `PathRule::RegexPath` to the private `split_query` helper. Name it as code instead. Signed-off-by: Florentin Dubois <florentin.dubois@clever.cloud>

Commit:ae37afc
Author:Florentin Dubois

Merge feat/regex-path-only into feat/h2c-prior-knowledge Stacks this branch on #1916, #1918 and #1915, up to date with main after #1919 (lockfile) and #1914, so CI runs on the tree this PR lands. Keeps every CHANGELOG slot, each followed by a blank line. Signed-off-by: Florentin Dubois <florentin.dubois@clever.cloud>

Commit:cabdabf
Author:Florentin Dubois

feat(http): opt-in HTTP/2 prior knowledge (h2c) on HTTP listeners `HttpListener` refused cleartext HTTP/2: a client opening with the `PRI * HTTP/2.0` preface got a 400, so cleartext gRPC (e.g. the Kubernetes Gateway API `GRPCRoute` conformance suite, which dials with insecure credentials) could not be served. Add a per-listener `h2c_prior_knowledge` option (HttpListenerConfig field 38, UpdateHttpListenerConfig field 44, TOML key, `listener http add/update` CLI flags), off by default. When on, a new session first goes through an `H2cPrefaceSniff` state (`lib/src/protocol/h2c.rs`) that peeks up to 24 bytes without consuming them: the full preface makes it an HTTP/2 mux frontend with the listener's H2 settings, as an ALPN `h2` connection is on HTTPS; the first diverging byte makes it HTTP/1.x; a strict prefix waits within `request_timeout`. The mux reads the peeked bytes itself, and the upgrade re-arms READABLE and WRITABLE since peeking raises no new edge. It composes with `expect_proxy` (PROXY header, then sniff). The scheme stays `http` and `tls_alpn` stays unset. `Upgrade: h2c` stays unsupported. Such a listener is held to the H2 minimum `buffer_size` at configuration load, like an HTTPS listener advertising `h2`. New metrics `http.h2c.prior_knowledge` and `http.upgrade.h2c.failed`. Closes #1911 Signed-off-by: Florentin Dubois <florentin.dubois@clever.cloud>

Commit:368f541
Author:Florentin Dubois

feat(router): add a REGEX_PATH rule matched against the path alone A `REGEX` path rule is matched against the request path together with its query string, so `/api/.+` matches `/api/?x=1` and `^/api/v2$` does not match `/api/v2?x=1`. There was no way to match a regex against the path only, which is what the Gateway API HTTPRoute `RegularExpression` path match (and Traefik `PathRegexp`, Envoy `safe_regex`) expects. Add a new path rule kind, `REGEX_PATH` (`PathRuleKind::REGEX_PATH = 3`, TOML `path_type = "REGEX_PATH"`, CLI `--path-regex-path`). It is anchored and compiled exactly like `REGEX` and ranks as `REGEX` does, but it is tested against the request-target up to its first `?`. `REGEX`, `PREFIX` and `EQUALS` are unchanged. With `rewrite_path`, a `REGEX_PATH` frontend's `$PATH[n]` captures, `$PATH[0]` included, come from the path alone. The original `?query` is appended to the rewritten path unless that path carries a `?` of its own, and a redirect `Location` follows it. The frontend gets its own `ConfigState` key letter (`Q`), so a `REGEX` and a `REGEX_PATH` frontend on the same pattern don't collide. Library API: `PathRule::from_cli_options` takes a fourth argument, and `PathRule` (sozu-lib) and `PathRuleType` (sozu-command-lib) gain a `RegexPath` variant. A worker that predates the kind refuses such a frontend with `RouterError::InvalidPathRule`. Fixes #1909 Signed-off-by: Florentin Dubois <florentin.dubois@clever.cloud>

Commit:c4b00f9
Author:Florentin DUBOIS
Committer:GitHub

feat(health-check): protect probe headers in memory (#1903) Support protected, memory-only HTTP health-check headers with typed capability negotiation and all-worker transactions. Preserve authority across worker replacement and hot upgrade, redact saved/output metadata and reject foreign secret state until explicit provisioning. Document file-based inputs and validate cold start, hot upgrade, worker replacement and old-main refusal with causal regression tests. Run process observers under an isolated checkout-owner namespace and fix strict Rustdoc links. Closes #1901 Signed-off-by: Florentin Dubois <florentin.dubois@clever.cloud>

Commit:01c7b73
Author:Florentin Dubois

feat(health-check): keep probe headers protected in memory Add validated file/stdin provisioning with acknowledged worker transactions, keyed public fingerprints, zeroizing storage and memory-only upgrade v3. Preserve cold provenance and align reload state with the protected journal. Document the operator contract and cover probe encoding and lifecycle paths with unit, simulation and process regressions. Closes #1901 Signed-off-by: Florentin Dubois <florentin.dubois@clever.cloud>

Commit:7852202
Author:Florentin DUBOIS
Committer:GitHub

feat(health-check): probe a port other than the backend data port Add optional health-check port overrides for HTTP and TCP clusters while preserving the backend data endpoint for ordinary traffic. Carry the effective probe address through IP and UNIX transports, retain backend-incarnation guards, and refuse probe-port plus unix_path configurations before state or worker mutation. Keep UNIX/IP transitions convergent through diff, static reload and SaveState/LoadState without resetting idempotent backend health. Document that every worker must run 2.3.2 before enabling the additive field, since older workers can ignore it and probe the data port. Validated with RED/GREEN transition and transport regressions, focused command/library/worker tests, four probe-port E2E scenarios, DNS-plus-UNIX E2E, locked workspace build, clippy, formatting and documentation citations. Signed-off-by: Florentin Dubois <florentin.dubois@clever.cloud>

Commit:bbce286
Author:Florentin Dubois

feat(health): reconcile probe ports with unix backends Signed-off-by: Florentin Dubois <florentin.dubois@clever.cloud>

Commit:ffb5368
Author:Florentin DUBOIS
Committer:GitHub

feat(backends): dial backends over UNIX domain sockets Support stream backends over UNIX domain sockets, including Linux abstract sockets (`@name`), while retaining existing IP backend behavior. Configuration carries the socket path through state and workers; DNS expansion preserves that path for every resolved address and rejects identity collisions before emitting requests. UDP rejects UNIX stream targets before mutation. Canonical UNIX-backend requests use protobuf verb 72 so older decoders cannot accidentally dial the identity IP address. Validation covers filesystem and abstract sockets, unchanged IP behavior, state/command handling, old-decoder fail-closed behavior, UDP refusal, and a real DNS-plus-UNIX backend exchange in addition to weighted-load-balancing, header-routing, silent-close, and close-session integration. The exact integrated head is independently reviewed and must complete the entire GitHub Actions matrix before merge. Closes #152 Signed-off-by: Florentin Dubois <florentin.dubois@clever.cloud>

Commit:7b68d86
Author:Florentin Dubois

fix(backends): integrate UNIX sockets with header routing Preserve the fail-closed wire verbs and canonical request handling from header routing while carrying UNIX socket transports through DNS-expanded configuration. Refs #152 Signed-off-by: Florentin Dubois <florentin.dubois@clever.cloud>

Commit:497e8e8
Author:Florentin Dubois

Merge remote-tracking branch 'origin/main' into feat/health-check-port

Commit:88a3bfe
Author:Florentin DUBOIS
Committer:GitHub

feat(router): header-based routing predicates on HTTP frontends Add declarative exact, regular-expression, and presence predicates to HTTP and HTTPS frontends with AND semantics. Frontend identity normalizes predicate order, header-name case, and duplicate predicates; TCP configuration rejects this HTTP-only feature. Additive field 17 and guarded request verbs 66-69 preserve fail-closed behavior with older decoders. Validation covers command handling, HTTP/1 and HTTP/2 routing, predicate identity, frontend removal, TCP refusal, legacy-decoder refusal, and integration with silent-close frontends. The exact integrated head is independently reviewed and must complete the entire GitHub Actions matrix before merge. Closes #1253 Signed-off-by: Florentin Dubois <florentin.dubois@clever.cloud>

Commit:3342592
Author:Florentin Dubois

fix(backends): integrate DNS-expanded UNIX sockets Merge the current backend ownership, close-session, and DNS configuration changes into UNIX socket backends. Preserve unix_path through literal and expanded backend configuration, validate identities after resolution, and cover a real DNS-named backend that dials only its UNIX socket. Refs #152 Signed-off-by: Florentin Dubois <florentin.dubois@clever.cloud>

Commit:14e0cbe
Author:Florentin Dubois

chore(merge): integrate Sōzu main into header routing Signed-off-by: Florentin Dubois <florentin.dubois@clever.cloud>

Commit:ae35ed8
Author:Florentin DUBOIS
Committer:GitHub

feat(command): close sessions by cluster or backend Add a CLI action and all-worker request to close active sessions belonging to a cluster or backend, with per-worker results. Session closure is transient: it is never persisted in configuration state or replayed to replacement workers. Validation covers HTTP/1, HTTP/2, TCP and WebSocket closure, multi-worker result aggregation and failures, session identity, and persistence/replay invariants. The exact integrated head is independently reviewed and must complete the entire GitHub Actions matrix before merge. Closes #123 Signed-off-by: Florentin Dubois <florentin.dubois@clever.cloud>

Commit:4800762
Author:Florentin Dubois

fix(backends): reconcile UNIX sockets with cluster ownership Signed-off-by: Florentin Dubois <florentin.dubois@clever.cloud>

Commit:8953343
Author:Florentin DUBOIS
Committer:GitHub

fix(state): tie backends to their cluster Reject AddBackend for an unknown cluster in both configuration state and workers before creating or mutating a backend bucket. Preserve the existing RemoveCluster cascade and frontend independence, and update examples and end-to-end fixtures to declare clusters before their backends. Validation covers command/state and worker behavior, example regressions, mutable-command simulation, and the affected end-to-end cases with their strict HTTP and metric assertions intact. The exact integrated head is independently reviewed and must complete the entire GitHub Actions matrix before merge. Closes #996 Signed-off-by: Florentin Dubois <florentin.dubois@clever.cloud>

Commit:c218cb6
Author:Florentin Dubois

chore(merge): integrate Sōzu main into header routing Signed-off-by: Florentin Dubois <florentin.dubois@clever.cloud>

Commit:5577a06
Author:Florentin Dubois

chore(backends): integrate Unix sockets with main Signed-off-by: Florentin Dubois <florentin.dubois@clever.cloud>

Commit:8b5d44f
Author:Florentin Dubois

fix(state): reconcile backend ownership with main Closes #996 Signed-off-by: Florentin Dubois <florentin.dubois@clever.cloud>

Commit:5b391b5
Author:Florentin Dubois

Merge remote-tracking branch 'origin/main' into feat/health-check-port

Commit:3c240b0
Author:Florentin Dubois

chore(command): integrate close sessions with main Signed-off-by: Florentin Dubois <florentin.dubois@clever.cloud>

Commit:496e4d6
Author:Florentin DUBOIS
Committer:GitHub

feat(http): silent close frontend action (nginx 444) (#1893) * wip: feat(http): silent close frontend action (nginx 444) Signed-off-by: Florentin Dubois <florentin.dubois@clever.cloud> * fix(command): preserve silent-close request diagnostics Signed-off-by: Florentin Dubois <florentin.dubois@clever.cloud> * docs(changelog): add topic anchors for concurrent entries Signed-off-by: Florentin Dubois <florentin.dubois@clever.cloud> --------- Signed-off-by: Florentin Dubois <florentin.dubois@clever.cloud>

Commit:411aa1e
Author:Florentin Dubois

feat(health-check): probe a port other than the backend data port HealthCheckConfig gains an optional probe port (tag 9). When set, HTTP and TCP probes dial the backend's IP with that port, and the HTTP probe names the probed address in Host (h2c :authority). Unset keeps today's behaviour; a peer that predates the field ignores it. UDP clusters already have UdpHealthConfig.tcp_port. Validation bounds the port to 1..=65535. The TOML file config, the CLI (cluster health-check set --port), the health-check list output and the docs carry it. Signed-off-by: Florentin Dubois <florentin.dubois@clever.cloud>

Commit:77d13c9
Author:Florentin Dubois

docs(command): clarify unix backend compatibility Signed-off-by: Florentin Dubois <florentin.dubois@clever.cloud>

Commit:653de1e
Author:Florentin Dubois

test(command): cover close sessions lifecycle Signed-off-by: Florentin Dubois <florentin.dubois@clever.cloud>

Commit:2e87831
Author:Florentin Dubois

feat(router): route frontends by headers Closes #1253 Signed-off-by: Florentin Dubois <florentin.dubois@clever.cloud>

Commit:99dbc75
Author:Florentin Dubois

fix(command): fail closed for unix backends Signed-off-by: Florentin Dubois <florentin.dubois@clever.cloud>

Commit:b934f10
Author:Florentin Dubois
Committer:Florentin Dubois

feat(command): close the sessions of a cluster or backend on demand Add `Request.close_sessions` (field 65, `CloseSessions { cluster_id, backend_id?, address? }`) and `ResponseContent.closed_sessions` (field 19, `ClosedSessions { closed, draining }`), plus `sozu cluster close-sessions --id <cluster> [--backend-id <id>] [--address <ip:port>]`. The main process fans the request out to every worker and answers with each worker's counts in `WorkerResponses`; it never touches `ConfigState`, so the request is neither saved nor replayed by `load_state` or an upgrade. A worker selects every session holding a backend connection to the cluster, narrowed to one backend by id and/or address (`ProxySession::targets`: mux backend connections, WebSocket pipes, TCP sessions), and closes it the way a soft stop does with `ProxySession::shutting_down`: idle HTTP/1.1, TCP and WebSocket sessions close at once; a session with HTTP work in flight drains and is re-asked every 100 ms tick by `Server::drive_closing_sessions`, which holds a `Weak` so a reused slab key is never closed by mistake. HTTP/2 sessions receive a GOAWAY and are bounded by their graceful-shutdown deadline. `Mux::shutting_down_inner` now marks a linked HTTP/1.1 stream `closing`, so the pending response announces `Connection: close` and the frontend closes once it is sent. This also applies to a soft stop. Refs #123 Signed-off-by: Florentin Dubois <florentin.dubois@clever.cloud>

Commit:f054ec1
Author:Florentin Dubois

wip: feat(http): silent close frontend action (nginx 444) Signed-off-by: Florentin Dubois <florentin.dubois@clever.cloud>

Commit:c43cbf4
Author:Florentin Dubois

wip: fix(state): tie backends to their cluster Signed-off-by: Florentin Dubois <florentin.dubois@clever.cloud>

Commit:50076b2
Author:Florentin Dubois

wip: feat(command): close sessions by cluster or backend Signed-off-by: Florentin Dubois <florentin.dubois@clever.cloud>

Commit:c2dec96
Author:Florentin Dubois

wip: feat(router): header-based routing predicates on HTTP frontends Signed-off-by: Florentin Dubois <florentin.dubois@clever.cloud>

Commit:75b4ad8
Author:Florentin Dubois
Committer:Florentin Dubois

feat(backends): dial backends over UNIX domain sockets A backend can now declare `unix_path`: an absolute filesystem path, or `@name` for a Linux abstract-namespace socket. HTTP, HTTPS and TCP clusters dial that socket instead of `address`, which stays the backend's identity (removal, retry exclusion, hash and shard placement, events) and must be unique in its cluster. UDP clusters refuse it. The field is additive everywhere: `AddBackend.unix_path` (proto field 7), the `response::Backend` state record so replays and saved states keep it, `BackendConfig.unix_path` in the configuration file, and `sozu backend add --unix-path`. `parse_backend_unix_path` is the one validator used at config load, by the main process state and by every worker. `connect_backend` dials the UNIX stream with mio and hands its descriptor to the pipeline as a `mio::net::TcpStream`, so the H1, H2, TCP and WebSocket paths take it without a generic stream type. Every syscall they make on a backend socket answers the same way on AF_UNIX, except std's `peer_addr`, which refuses a non-INET peer: the two connect-completion checks (`connect_outcome`, `tcp_connect_outcome`) now use `socket2`'s family-agnostic `getpeername`, and TCP_NODELAY is skipped for unix backends. Health checks probe through the same dial. Tests: unit tests for the parser, the config and state round trip, the dial and both connect-outcome checks (red with std `peer_addr`, green with the fix), and e2e tests for HTTP over a path and an abstract socket and for TCP passthrough over a path socket. Closes #152 Signed-off-by: Florentin Dubois <florentin.dubois@clever.cloud>

Commit:eeffdc2
Author:Mbechezi Mlanawo
Committer:GitHub

feat(https): add mutual TLS client certificate authentication on HTTPS listeners (#1300) Adds client_auth (none/optional/required), client_ca_certificates and client_ca_crls on HTTPS listeners, refreshed by sozu reload. Closes #1299.

Commit:0f5430c
Author:Florentin DUBOIS
Committer:GitHub

test(e2e): cover real gRPC lifecycles (#1850) Exercise unary and bidirectional gRPC through HTTPS/H2 and transparent TCP. Validate real deadline and cancellation cleanup plus fresh frontend reconnection in a dedicated workflow. Signed-off-by: Florentin Dubois <florentin.dubois@clever.cloud>

Commit:d2cea3f
Author:Florentin Dubois

test(e2e): cover real gRPC lifecycles Exercise unary and bidirectional gRPC through HTTPS/H2 and transparent TCP. Validate real deadline and cancellation cleanup plus fresh frontend reconnection in a dedicated workflow. Signed-off-by: Florentin Dubois <florentin.dubois@clever.cloud>

Commit:7e2ebc0
Author:Florentin DUBOIS
Committer:GitHub

fix(command): transfer pending commands during main upgrade (#1846) The main-process hot upgrade rebuilt an empty command Hub while worker work could still be in flight. The original command client then received EOF, and a later worker response reached the replacement as an unknown task even though its data-plane request could still finish. This change transfers the complete control-plane ownership graph to the replacement main: connected clients, client and worker channel buffers, active and queued tasks, worker-response routes, subscribers, worker pending queues, stopped workers still referenced by a task or route, task deadlines, audit clocks, and future-ID counters. The replacement restores task state directly without replaying operator requests or re-scattering work. The handoff uses an explicit V2 `PREPARED` / `COMMIT` / `ACTIVATED` protocol: - the replacement validates and registers inherited descriptors while paused, without command or worker data I/O; - pre-commit rejection closes and reaps the candidate, restores `CLOEXEC`, and lets the unchanged old Hub continue; - the first attempt to send `COMMIT` irreversibly fences the old Hub, so an ambiguous post-commit failure cannot execute work twice; - after a complete commit, the replacement activates the Hub, schedules restored sessions once, publishes PID and systemd readiness, preserves the initiating client's terminal response under backpressure, and acknowledges activation before the old main exits. Compatibility fails closed before adoption. A running main from before this protocol cannot export the missing Hub state, so the first deployment requires a controlled service restart after pending commands are drained and state is saved. Subsequent V2-to-V2 hot upgrades transfer pending commands. A V2 sender rejects a legacy candidate before freezing live ownership; a legacy sender remains subject to its existing generation, audit, and child-reaping limitations when a V2 candidate refuses its old envelope. `SIGTERM` observed before commit aborts and reaps the candidate, then the old event loop consumes the preserved stop intention. Operators should signal the service control group. A signal aimed only at the old PID after the final pre-commit observation remains outside this contract, and a failure after commit is fail-stop rather than crash recovery. The branch is based on `9561b24e61c23511326ffeb547b0b7fe51ee7055`, including the deadline and cancelled-route fixes from #1838 and the PROXY protocol corrections merged in #1844. Channel snapshot validation accepts all five defined `Ready` bits in both serialized readiness and interest, preserves them exactly, and rejects the first unknown bit `0x20`. This covers the Linux readiness word `WRITABLE | HUP | WRITE_CLOSED` (`0x001a`) without special-casing that observed value. Descriptor-cleanup tests observe the original command-channel and SCM socket peers instead of probing released numeric descriptors or requiring immediate EOF. EOF is the only success outcome within a fixed one-second bound: data, other errors, and expiration fail. This tolerates transient descriptor copies across a concurrent fork-to-exec window while retaining a durable duplicate of either endpoint still fails the test. The exact transient holder in the original CI jobs was not instrumented, so that scheduler-level cause remains unverified. Validation completed locally with Rust 1.93.1 and at most four build jobs: - `cargo test -p sozu-command-lib --locked`: 244 unit tests, 9 integration tests, and 2 doctests passed; 3 existing doctests were ignored. - `cargo test -p sozu --locked --test upgrade_fd_inheritance_e2e -- --ignored`: failed 0/1 on the earlier snapshot mask with `invalid channel snapshot readiness bits: 0x001a`, then passed 1/1 after accepting `WRITE_CLOSED`. - the snapshot regression exercises all 32 subsets of the five defined bits in both readiness and interest; it rejects `0x20` in each field. Removing `WRITE_CLOSED` makes the test fail on `0x10`. - `cargo test -p sozu --locked --no-default-features --features crypto-openssl,opentelemetry,splice,simd --verbose`: 152/152 library tests passed and every integration and doctest target completed successfully. - `cargo test -p sozu --locked --no-default-features --features fips,opentelemetry,splice,simd --verbose`: 152/152 library tests passed and every integration and doctest target completed successfully. - retaining durable duplicates of both the command channel and SCM endpoint makes each corrected cleanup oracle fail with both EOF deadlines; restoring the candidate makes each targeted test pass 1/1. - the accepted-handoff, rejected-candidate, SIGTERM-during-PREPARE, and descriptor-inheritance process tests pass 1/1 on the rebased head. - the compatibility matrix passes 1/1 in both directions between the rebased candidate and the frozen pre-protocol `d7b5f14d` binary. This verifies those two transition paths rather than certifying every historical version pair; the legacy generation, audit, and child-reaping limitations described above remain observable. - `cargo clippy --all-targets --locked --no-default-features --features crypto-openssl,opentelemetry,splice,simd -- -D warnings` and the equivalent FIPS matrix passed. - `cargo +nightly fmt --all --check` passed. - `.github/scripts/check_doc_citations.py --base 9561b24e61c23511326ffeb547b0b7fe51ee7055` passed all 37 citations, 1,078 test names, and 15 pinned excerpts. - `env -u RUST_LOG ... cargo +1.93.1 test -p sozu --locked --features opentelemetry,splice,simd --test sigterm_soft_stop_e2e -- --ignored`: 1/1 passed. The local 27/27 missing-access-log result was caused by ambient `RUST_LOG=warn`, which GitHub CI does not set; the test requires its configured `info` access records. GitHub CI checks are pending on the rebased commit `a8bdd54b8833060576674a8c2aff0374f6e8083b`. Fixes #1832 Signed-off-by: Florentin Dubois <florentin.dubois@clever.cloud>

Commit:7a508dd
Author:Florentin Dubois
Committer:Florentin Dubois

fix(command): transfer pending commands across main upgrade Signed-off-by: Florentin Dubois <florentin.dubois@clever.cloud>

Commit:3fdf940
Author:Florentin DUBOIS
Committer:GitHub

fix(metrics): expire metric-detail leases from the worker event loop (#1834) Fixes #1831 ## Cause CAUSE: `lib/src/server.rs` `Server::notify` (the only production `lease_tick` call, reached only from command-channel messages) -> no event-loop path ran the janitor -> SYMPTOM: an abandoned `SetMetricDetail` lease outlived TTL + 5 s until the next worker command, regardless of data-plane traffic. ## Fix - The janitor body moves into `tick_metric_detail_leases(now)` (`lib/src/server.rs`), called once per `Server::run` iteration (before `send_queue`, so the `lease_tick_expired` event leaves on the same turn) and still from `notify` (keeps the single clock read shared with `lease_apply`). `lease_tick_due` still gates the table walk to once per 5 s. The loop wakes at least once per 1 s `poll_timeout`, so expiry is bounded by TTL + 5 s + 1 s with no traffic and no command. - Docs: `lib/src/metrics/mod.rs` comments, `command.proto` lifecycle comment, `doc/configure.md` (the old "at most one `ttl` window" was not true), `doc/sozu-top.md`, CHANGELOG (Unreleased / Fixed). ## Regression test `test_abandoned_lease_expires_without_a_command` (`e2e/src/tests/metrics_lifecycle_tests.rs`): real worker, apply a 1 s `DETAIL_BACKEND` lease, then write nothing more on the command channel while HTTP requests keep flowing; passively read the channel until a `lease_tick_expired` `METRIC_DETAIL_CHANGED` event (backend -> cluster) arrives within TTL + 8 s. - Red on `bd19a78e` (unfixed): `test ... test_abandoned_lease_expires_without_a_command ... FAILED` (`stability check FAILED: iteration 1 of 3`) - Green with fix: `test result: ok. 4 passed; 0 failed` (all `metrics_lifecycle` e2e, `--features opentelemetry,splice,simd`) ## Validation - `cargo +nightly fmt --check` -> exit 0 - `cargo +1.93.1 clippy --all-features --all-targets --locked -- -D warnings` -> exit 0 - `RUSTDOCFLAGS="-D warnings" cargo +1.93.1 doc --no-deps --locked -p sozu-lib -p sozu-command-lib` -> exit 0 - `cargo +1.93.1 test -p sozu-command-lib --all-features --locked` -> exit 0 - `cargo +1.93.1 test -p sozu-lib --lib --all-features --locked` -> 1591 passed, 1 failed: `protocol::pipe::tests::splice_readable_eof_drains_kernel_pipe_bytes_before_closing` hit its own host precondition guard (`splice pipe capacity is 8192 bytes ... exceed fs.pipe-user-pages-soft (16384) ... this is a host limit, not a Sozu regression`) on a shared, heavily loaded host; three isolated reruns: pass / same precondition / pass. `pipe.rs` is untouched here. - `python3 .github/scripts/check_doc_citations.py --root . --base origin/main` -> exit 0 - Full e2e suite not run locally (left to CI). #1828 (same-ID cluster incarnation) is deliberately not in this PR: it needs a design decision, see the issue. Signed-off-by: Florentin Dubois <florentin.dubois@clever.cloud>

Commit:579af63
Author:Florentin Dubois

fix(metrics): expire metric-detail leases from the worker event loop The lease janitor ran only at the top of `Server::notify`, which only a worker command reaches. A lease whose owner disappeared (a crashed `sozu top`) therefore kept the worker's metric cardinality elevated past its TTL until the next control request, regardless of data-plane traffic. Move the janitor into `tick_metric_detail_leases` and call it once per event-loop iteration, before the response queue is flushed, as well as from `notify`. The loop wakes at least once per one-second poll timeout, so an abandoned lease is retired and its `lease_tick_expired` event is pushed within TTL + 5 s + one poll timeout with no further command. The new real-worker regression applies a one-second lease, keeps requests flowing, writes nothing more on the command channel and waits for the expiry event; it fails on the previous code. Fixes #1831 Signed-off-by: Florentin Dubois <florentin.dubois@clever.cloud>

Commit:d2d95fd
Author:Florentin DUBOIS
Committer:GitHub

docs: fix stale flood-window, idle-timeout and rejected-per-IP statements (#1816) Docs and comments only, no behaviour change. Each statement was checked against the code on `main` (6c3b8e89). | Statement | Code | Fix | |---|---|---| | `h2_flood_detector.rs` comments size the connection window at 1 MiB | `ENLARGED_CONNECTION_WINDOW = 16 * 1024 * 1024` (`h2.rs`), stream window `DEFAULT_INITIAL_WINDOW_SIZE = 65535` | Stream-0 example: Sōzu acks every 8 MiB of 16 MiB. Glitch budget: a cancelled upload is bounded by the 64 KiB stream window, about four glitches in 16 KiB frames (same arithmetic as the refused-stream comment in `h2.rs`), not 64. Chained-proxy test doc now describes the 1 MiB peer it models, and notes that Sōzu's 16 MiB window is sparser. The test body is unchanged. | | `command.proto` (×2) and `command/src/config.rs`: `h2_stream_idle_timeout_seconds` "Default: 30" | `get_h2_stream_idle_timeout` (`http.rs`, `https.rs`): `unwrap_or(back_timeout.max(30))`, explicit `s.max(1)` | Unset, it inherits `back_timeout` with a floor of 30 (so 30 with the default `back_timeout`); an explicit value wins, and `0` means 1. `doc/configure.md` already said this. | | `connections.rejected_per_cluster_ip` counts TCP rejections (`doc/configure.md`, `doc/rate-limit-design.md` §3.5/§4), and e2e asserts it (`rate-limit-design.md` §5, `cluster_ip_limit_tests.rs` header) | Sole increment: `answers.rs` `set_default_answer`, status `429`. The TCP gate (`tcp.rs` `connect_to_backend` → `TooManyConnectionsPerIp` → `handle_connection_result` → `Close`) increments nothing, and no e2e test reads the counter | Docs now say it counts HTTP/HTTPS 429 answers only (per-IP or per-subnet gate), and that TCP rejections are not counted. The false "confirms the metric" claims are dropped. | Not changed: - The TCP per-(cluster, IP) rejection path has no counter. Adding one would be a behaviour change and needs its own decision. - The remaining `CONN_RETRIES` mentions are historical ("replaces the former constant") and accurate. - `BackendMap::max_failures` (`lib/src/backends.rs`) is set to 3 but nothing in the tree reads it. It is a dead public field and is left untouched here. Validation: `cargo +nightly fmt --check`, `cargo +1.93.1 clippy --all-features --all-targets --locked -- -D warnings`, `RUSTDOCFLAGS="-D warnings" cargo +1.93.1 doc --no-deps --locked -p sozu-lib -p sozu-command-lib`, `cargo +1.93.1 test -p sozu-command-lib --all-features --locked` and `check_doc_citations.py --base origin/main` all pass. Signed-off-by: Florentin Dubois <florentin.dubois@clever.cloud>

Commit:65ab6af
Author:Florentin Dubois
Committer:Florentin Dubois

docs: fix stale flood-window, idle-timeout and rejected-per-IP statements - h2_flood_detector.rs: the connection window is 16 MiB (acked every 8 MiB), not 1 MiB; a cancelled upload's in-flight DATA is bounded by the 64 KiB stream window, about four glitches in 16 KiB frames. - command.proto, command/src/config.rs: h2_stream_idle_timeout_seconds inherits back_timeout floored at 30 when unset; 0 means 1. - doc/configure.md, doc/rate-limit-design.md, cluster_ip_limit_tests.rs: connections.rejected_per_cluster_ip is incremented only by the HTTP/HTTPS 429 answer; a TCP rejection is not counted and no e2e test asserts the counter. No behaviour change. Signed-off-by: Florentin Dubois <florentin.dubois@clever.cloud>

Commit:bd787ac
Author:Florentin DUBOIS
Committer:GitHub

docs: align the example configuration and docs with recent changes (#1809) Audit of the docs, `bin/config.toml` and `os-build/config.toml` against everything merged on `main` from 2026-10-01 to 2026-10-03 (43 commits, #1724 through #1808). Most of the new keys were already documented by their own PRs (`max_connection_attempts`, `h2_stream_refusal_percent`, health check `mode`/`accepted_statuses`, the raised flood defaults, the 16 MiB connection window). This PR fixes what was still stale. | Area | Where | Was | Now | |---|---|---|---| | #1806 | `bin/config.toml`, `os-build/config.toml` answer comments | 503 only for "no backend available"; "504 means the backend timed out" | 503 also when every attempt failed; 504 = connected backend silent or frontend timeout while waiting; connect timeout fails over | | #1806 | `bin/config.toml` `max_connection_attempts` | connect timeout always fails over | a TCP session whose connect times out is closed | | #1806 | `doc/debugging_strategies.md` | 503 after a circuit breaker of 3 failed connections; ERROR log lines, one that no longer exists | `max_connection_attempts` (default 5), `backend.connect.retries_exhausted`, real TCP log levels and text | | #1806 | `doc/configure.md` `front_timeout` / `connect_timeout` rows | "inactivity for a request to connect" | time to establish a backend connection; HTTP fails over, TCP closes; H1 failover bound by `front_timeout` | | #1806 | `doc/configure.md` `client_timeout_during_response`, `backend_timeout`, `http.503.errors` | "connection-level slowness", "circuit breaker" | failover case included; established connection only; attempt budget | | #1806 | `LIFECYCLE.md` backend timer | 504 for any linked stream | established connection only | | #1806 | `cli.rs` help, `doc/configure_cli.md` | "answered 503" | "or a TCP session is closed" | | #1756 | `doc/configure.md` flood metrics | RST lifetime rows described as absolute ceilings; no `window_update_stream0_window` row | floor-and-ratio semantics; row added | | #1756/#1798 | `doc/architecture.md`, `doc/h2_mux_internals.md` | 6 / 13 `H2FloodConfig` thresholds | 14, with `h2_stream_refusal_percent` | | #1756 | `doc/configure_admin_ops.md` | "halve" example 50/25 (old defaults); H2 knobs HTTPS only | 1000/500; HTTP and HTTPS; `h2.flood.stream_refused` | | #1752 | `doc/configure.md` glitch tuning | "1 MiB window ... about 64" frames | 64 KiB stream window, about four | | #1805 | `doc/health_checks.md` | `remove` resets backends to healthy; DOWN logs an error; `load_balancing.rs::BackendList::healthy` | backends keep their last state (see note); warning; `BackendList::select_tiers` in `backends.rs` | | #1805 | `doc/metrics.md` | health-check counters "per cluster" | unlabelled | | #1805 | `doc/configure_cli.md` | no health-check section | short section, `--mode`, `--probe-timeout` vs global `--timeout` | | #1746 | `cli.rs` `--load-balancing-policy`, `bin/config.toml`, proto comment | HRW/MAGLEV key = UDP flow key | UDP affinity key (source IP, or IP and port) | | UDP | `doc/configure.md` | hot upgrade drains flows bounded by idle timeout; `udp.flows.evicted` reasons | soft-stop closes flows at once; listener removal/soft-stop/abort added | | #1770 | `doc/udp_simulation.md` | no per-source limits | in reconfig storms and the invariant list | | #1743 | `doc/lifetime_of_a_session.md` | keep-alive backend always reused | unless its backend was removed | | older | `command.proto` update messages | `h2_stream_shrink_ratio >= 1` | `>= 2`, as validated in `state.rs` | | #1806 | e2e shuffle-sharding doc comment | "three connection attempts" | `max_connection_attempts` (5 by default) | Reported here, not changed in this PR: - `sozu cluster health-check remove` does not reset backends a probe marked DOWN (`remove_health_check_state` skips the reset that `set_health_check_config(None)` does), so they stay out of rotation until the cluster is added again. The doc now describes this. It looks like a code bug. - `sozu top`'s H2 pane reads `h2.flood.violation.{rapid_reset,made_you_reset,ping,settings,priority,continuation}`, which the detector never emits (it emits `rst_stream_pre_response_lifetime`, `ping_window`, ...). Only `glitch_window` matches, and `h2.flood.stream_refused` is not shown. - `h2_flood_detector.rs` comments still refer to a 1 MiB connection window, and the `h2_stream_idle_timeout_seconds` proto comment says "Default: 30" where the value inherits `back_timeout`. - `connections.rejected_per_cluster_ip` is documented as also counting TCP rejections; only the HTTP 429 path increments it. Validation: - `sozu config check` passes on `bin/config.toml`, on a copy with every commented key under review uncommented (global and cluster `max_connection_attempts`, the H2 block, the health check block), and on `os-build/config.toml` with local state/socket paths. Negative controls (`max_connection_attempts = 0`; `expected_status = 200` next to `accepted_statuses`) are rejected. - `cargo +1.93.1 test -p sozu-command-lib --all-features --locked`: 235 passed. - `cargo +1.93.1 test -p sozu --bin sozu --locked`: 124 passed (including the `--load-balancing-policy` help tests). - `cargo +nightly fmt --all --check` and `python3 .github/scripts/check_doc_citations.py --root . --base origin/main` are clean. Signed-off-by: Florentin Dubois <florentin.dubois@clever.cloud>

Commit:832d77c
Author:Florentin Dubois

docs: align the example configuration and docs with recent changes Audit the documentation, bin/config.toml and os-build/config.toml against the changes merged on main from 2026-10-01 to 2026-10-03 and fix every statement that no longer matches the code: - connection attempts (#1806): 503 and 504 answer comments in both example configurations, the TCP close on connect timeout, the front_timeout and connect_timeout rows, the access-log timeout tokens, http.503.errors, the three-attempt circuit breaker and its stale log lines in doc/debugging_strategies.md, the TCP close in the CLI help; - H2 flood protection (#1756, #1798): floor-and-ratio semantics of the RST_STREAM lifetime violation metrics, the missing window_update_stream0_window metric row, fourteen H2FloodConfig thresholds, the admin-ops worked example against the current defaults, h2.flood.stream_refused, h2_stream_refusal_percent among the listener-update fields; - health checks (#1805): health-check remove keeps the last health state, DOWN logs a warning, the fail-open citation, unlabelled health_check counters, a configure_cli.md section, the TCP-mode timeout comment; - UDP (#1746, #1770): HRW/MAGLEV key on the affinity key rather than the flow key, soft-stop closes flows at once, udp.flows.evicted reasons, per-source limits in the simulation doc; - pooled connections to a removed backend are not reused (#1743); - the h2_stream_shrink_ratio floor of 2 in the update-message comments. Documentation only, no behaviour change. Signed-off-by: Florentin Dubois <florentin.dubois@clever.cloud>

Commit:e959838
Author:Florentin DUBOIS
Committer:GitHub

fix(mux): fail over to another backend on connect timeout (#1806) Fixes #1800 ## Summary A backend that accepts no connection (blackholed: SYN dropped, accept queue full) hit `connect_timeout` and the request was answered `504`. The timeout path marked the connection `Disconnecting`, so the dead-backend branch of `Mux::ready_inner`, the only place a connect failure is counted, never saw it. The backend's retry policy was never armed and the load balancer kept picking it. - **Connect timeout counts as a connect failure.** `Mux::timeout_inner` now flags a still-`Connecting` connection's readiness `ERROR`, the way the kernel flags a refused dial. `Server::timeout` then runs the `ready` pass that only it can trigger (new `ProxySession::needs_ready_pass`, default `false`). That pass reuses the existing dead-backend branch: it counts the failure (`failures`, `retry_policy`, `backend.connections.error`, `backend.down`), closes the connection, and re-links the request through `EndStreamAction::Reconnect`. The request sent no byte, so it is safe to resend. A response timeout on an established connection still answers `504`. - **A WRITABLE event no longer marks a connecting socket connected.** `SO_ERROR` (`take_error`) is read first, then `getpeername(2)`. An error is a connect failure. `ENOTCONN` means the WRITABLE was raised by Sōzu itself, not by the kernel, so the connection waits for the kernel's completion edge. This fixes a separate defect you can observe on its own: on `main`, a dial from an `http2 = true` cluster to a blackholed backend was marked connected and then hung until `back_timeout`. - **A synchronous `connect(2)` failure is retried** instead of being answered `503` at once. - **Each retry skips the backends that already failed for this request** (`Stream::tried_backends`), for every load-balancing policy. The exclusion narrows the candidate set (`BackendList::select_with_key_excluding`). If every selectable backend was already tried, selection falls back to the plain behaviour, fail-open included, so the request retries one of them instead of being refused. A sticky cookie that names a tried backend is not followed. - **The retry budget is configurable.** `max_connection_attempts` counts every attempt, the first included. It replaces the hard-coded `CONN_RETRIES = 3`, and **the default is now 5**. It can be set: - globally: TOML and `ServerConfig` field 29; - per cluster: `Cluster` field 23, TOML, `sozu cluster add --max-connection-attempts`, and `sozu cluster connection-attempts set|unset` (query, then re-send with the field changed, like `cluster h2`). A cluster table column shows the value; - at runtime on every worker: `sozu connection-attempts set` (`Request.set_max_connection_attempts`, field 62). Like `connection-limit set`, this is not saved. Allowed range is `1..=255`. It is validated at config load, on `AddCluster` (main and worker), and on the runtime set. A cluster value wins over the global one, for both HTTP and TCP clusters. - **Failover is bounded by `front_timeout`.** The frontend's nominal timeout is set only on a request's first link, not on every re-link. The deadline armed then therefore bounds the whole failover, so a request can no longer run for `max_connection_attempts × connect_timeout`. When the deadline expires during failover, the request gets the frontend timeout's existing answer: `504`, with access-log message `client_timeout_during_response`. `doc/configure.md` recommends keeping `connect_timeout` well below `front_timeout / max_connection_attempts`, and explains how back-off grows. - **Socket error logged.** The socket error read by `take_error()` is logged at `debug!`, the same level as the synchronous-refusal path. **Status change for alerting:** when every attempted backend is unreachable (connect timeout or refused), the answer is now `503`. A connect timeout used to answer `504`. Requests that arrive during a back-off window can also fail fast with `503`. Operators alerting on `504` rates should watch `503`, `backend.connections.error` and `backend.connect.retries_exhausted` instead. With the default budget at 5, an idempotent request on a stale keep-alive connection can be replayed to up to 4 backends (`n - 1`), against 2 before. **Coordination:** `Request` field 62 is also used by the open PR #1300. That PR already needs renumbering, so this one keeps 62. **Library API break:** `sozu_lib::server::CONN_RETRIES` is removed, and new fields are added to `Cluster`, `ServerConfig`, `RequestType`, `FileConfig`, `FileClusterConfig`, `HttpClusterConfig`, `TcpClusterConfig`, `Stream` and `SessionManager`. Upgrade the main process, the workers and the CLI together before setting a per-cluster budget: an older CLI that patches clusters by read-modify-write drops the field. Docs: `doc/configure.md` (new "Backend connection failover" section, global key, metrics, shard and replay budget text), `doc/configure_cli.md`, `doc/health_checks.md`, `doc/lifetime_of_a_session.md`, `doc/rate-limit-design.md`, `lib/src/protocol/mux/LIFECYCLE.md` §7.3, `bin/config.toml`, and `CHANGELOG.md`. ## Existing tests touched Two existing tests now call `mark_connected(h1)` in their fixtures: `a_backend_timeout_leaves_no_linked_stream_behind_on_the_close_path` and `a_completed_backend_response_timeout_unlinks_through_end_stream_alone`. Both describe a backend that has already sent response bytes, but their fixture left it `Connecting`. With this change, a `Connecting` connection is treated as a connect timeout. No assertion changed, and neither test encodes the old `504`; they cover the response-timeout arms. An alternative would be to narrow the timeout branch to `Connecting && no linked stream has back.consumed`. I kept the fixture fix instead, because that extra condition cannot be reached in production. ## Tests New: `e2e/src/tests/backend_failover_tests.rs`, plus `connect_outcome_tests`, `exclusion_tests`, `a_connect_timeout_flags_the_dial_dead_instead_of_answering_504`, the `max_connection_attempts` config tests, and the state validation test. What each test proves: - The connect-timeout e2e tests use 3 healthy and 3 blackholed backends with `connect_timeout = 1`. The blackhole is a backlog-0 listener with a full accept queue; `test_blackholed_listener_times_out` checks that it really times out. Every request must answer 200, and the cluster-level `backend.connections.error` must be greater than 0. That counter shows the timeouts are now counted against the backend, which arms its retry policy. It does not measure a drop in how often dead backends are picked. - The refused-backend e2e tests use a refused/healthy alternation that drives the first request onto three refused backends in a row. The budget raise is what fixes them. The tried set itself is covered by `exclusion_tests`. ## CI fix: leaked mock backend threads CI on `9bd8bfa6` and `379b4a4e` failed `tests::h2_tests::test_h2_rapid_reset_triggers_goaway` in every cell ("received 0 frames"). The cause was in this PR's test module, not in production code. **Cause:** `Backends` in `e2e/src/tests/backend_failover_tests.rs` dropped its `AsyncBackend` handles without stopping them. That backend's thread (`e2e/src/mock/async_backend.rs`, the spawn loop) only exits on a stop message, so every dropped handle left a thread spinning for the rest of the test process. That was about two dozen threads, all started before `h2_tests` in name order. On a small CI runner they starved the rapid-reset worker: CI logged its 999 resets over 12 s, in batches of about 17 every 110 ms. Main's CI run took 0.16 s for the same resets, with identical per-stream timings. **Fix:** `impl Drop for Backends` now stops those backends. The rapid-reset test is unchanged. **Evidence:** I ran `taskset -c 0 cargo test -p sozu-e2e --locked -- backend_failover tests::h2_tests::test_h2_rapid_reset_triggers_goaway --test-threads=4 --nocapture` before and after the fix. - Before: about 24 threads were busy during the rapid-reset test (`ps -L`), and the 999 resets took 1.57 s instead of 0.1 s. - After: no busy leftover threads, and the 999 resets took 0.32 s. - The full e2e suite dropped from 3098 s to 1963 s. ## CI fix: worker events read as answers At `dff0285e`, `test_connect_timeout_fails_over_h2` failed in the crypto-openssl cell even though every request got its 200. The test's metrics query read a worker event (`content_type: Some("event")`) in place of its answer. The failover tests make backends fail on purpose, and a backend whose retry policy gives up on it is reported `BACKEND_DOWN` on the same command channel. Reads in these tests now skip events (`read_answer`). That event's arrival depends on randomized back-off, so I did not reproduce the race locally. After the fix, both connect-timeout tests passed 20 runs out of 20 under `taskset -c 0`, and 20 out of 20 on 2 CPUs loaded with 6 busy processes. Validation at head `6c7dd668`, after merging origin/main `320e86d6` (#1805; conflicts in `command/src/config.rs` and `doc/health_checks.md` resolved keeping both features): Verified: `cargo +nightly fmt --all -- --check` -> exit 0 Verified: `cargo +1.93.1 clippy --all-features --all-targets --locked -- -D warnings` -> exit 0 Verified: `cargo test -p sozu-lib --lib --all-features` -> `test result: ok. 1586 passed; 0 failed` Verified: `cargo test -p sozu-command-lib -p sozu --all-features --locked` -> exit 0, all `test result: ok` (227 command-lib unit tests) Verified: `cargo test -p sozu-e2e --features opentelemetry,splice,simd -- --test-threads=4` -> `test result: ok. 621 passed; 0 failed` Verified: `RUSTDOCFLAGS="-D warnings" cargo +1.93.1 doc --no-deps --locked -p sozu-lib -p sozu-command-lib -p sozu` -> exit 0 (An earlier run at `f8d4c18e` had one timing failure under load: `test_h1_chunked_encoding_edge_cases`. That test passed 3 times out of 3 in isolation, and the full `h1_security_tests` module passed with `--test-threads=4`.) Verified: `python3 .github/scripts/check_doc_citations.py --root . --base origin/main` -> exit 0 Red without the fix: - New e2e tests on `main` (`dc25d2ff`): - `test_connect_timeout_fails_over_h1`/`_h2`: `[200, 504, 200, 504, …]` (12×200, 12×504) and `backend.connections.error: 0`. - `test_refused_backends_never_exhaust_attempts_h1`/`_h2`: first request `503`. - `test_h2_backend_connect_failure_fails_over_h1`/`_h2`: `[200, 0, 200, 200, 0, …]`, where `0` is no answer within 10 s. - `test_failover_is_bounded_by_front_timeout_h1`/`_h2` (6 blackholed backends, budget 20, `connect_timeout` 1 s, `front_timeout` 3 s): - in a scratch copy of this branch with the per-attempt re-arm restored: no answer within 10 s on both frontends; - with the fix: `[504]` after about 3.09 s on both. - `try_connect_timeout_fails_over` now asserts `backend.connections.error >= 3`, one failure per blackholed backend. Round robin hits each of the six backends in the first six requests, so this is deterministic. - Budget and unit tests, in scratch copies of this branch with one part of the fix reverted at a time: - default set back to 3: `test_default_connection_attempt_budget_is_five` fails (`503`), and so does the config default test; - cluster override ignored: `…overrides_global` fails (`503`), and so do the runtime cluster update and the default test's per-cluster `four`; - runtime setter disabled: `…changes_at_runtime` fails (`200` where `503` is expected); - exclusion, fallback or sticky filter removed: the matching `exclusion_tests` fail; - `ENOTCONN` treated as connected: `a_dial_nobody_answers_is_pending` and the H2-backend e2e tests fail; - `Connecting` early return removed: `a_connect_timeout_flags_the_dial_dead_instead_of_answering_504` fails; - validation removed: the config and state range tests fail. Signed-off-by: Florentin Dubois <florentin.dubois@clever.cloud>

Commit:6c7dd66
Author:Florentin Dubois

Merge remote-tracking branch 'origin/main' into fix/backend-connect-timeout-failover Signed-off-by: Florentin Dubois <florentin.dubois@clever.cloud>

Commit:320e86d
Author:Florentin DUBOIS
Committer:GitHub

feat(health-check): add a TCP connect probe mode and configurable accepted HTTP statuses (#1805) Fixes #1801 ## What - **Probe mode** — `HealthCheckConfig.mode` (proto field 7, `HealthCheckMode`, default `HTTP`): - `HTTP`: same behaviour as before. - `TCP`: the backend is healthy iff the TCP connection is established within `timeout` (the socket becomes writable with no `SO_ERROR`, `peer_addr` succeeds). Nothing is sent, and the stream is deregistered and dropped once connected. That sends a FIN on an empty receive queue, or an RST if a banner-first backend (SMTP, SSH) has already sent bytes the probe never reads. - `uri` is ignored. A non-zero `expected_status` or a non-empty `accepted_statuses` is refused in TCP mode at every validation layer. - It runs on the existing `HealthChecker` loop, so tokens, timeout, thresholds, metrics, events and fail-open are unchanged. - **Accepted statuses** — `HealthCheckConfig.accepted_statuses` (proto field 8, `repeated HttpStatusRange { start, end }`, inclusive): - TOML and CLI entries: `"404"`, `"200-399"`, `"2xx"`, `"any"` (= `100-599`). Bounds must lie in `100-599` (RFC 9110 §15). - When the list is set, it replaces `expected_status`. A non-zero `expected_status` together with a list is refused by `validate_health_check_config`, which the CLI, the TOML loader, `ConfigState` and the worker all call. - **Final status only** — HTTP probes skip interim `1xx` responses (except `101`), on HTTP/1.1 and on h2c. The h2c walk keeps one HPACK decoder across header blocks, so accepting `1xx` or `any` never lets a `100 Continue` stand in for a final 500. - The 4096-byte response cap covers interim responses too: a `103 Early Hints` with more than about 4 KB of headers fails the probe. - **Compatibility**: - `uri` stays `required` on the wire and may be empty in TCP mode. The TOML `uri` and `--uri` default to `/`. - Both new fields carry `#[serde(default)]`, so JSON saved states written before them still load. `state_compat_v1_1_1` stays green. - A message without field 7 decodes as `HTTP`. - **CLI** — `sozu cluster health-check set --mode http|tcp --accepted-statuses …`. - `list` output changes: a new `mode` column, `expected status` becomes `accepted statuses`, and a legacy `0` is shown as `2xx` instead of `any 2xx`. - In TOML, `mode` is `"HTTP"`/`"TCP"`, case-sensitive like the other protocol enum keys. It is distinct from the UDP `udp.health.mode = "TCP_PROBE"`. ## Upgrade warning (also in CHANGELOG and `doc/health_checks.md`) **Upgrade the main process and every worker, and use the new CLI, before enabling `tcp` mode or `accepted_statuses`.** Otherwise the default probe silently comes back: `GET <uri>` (`/` if unset) accepting only 2xx, which marks down every 3xx/404/500 backend. This happens with: - an older worker running between `sozu upgrade main` and `upgrade worker`, which ignores fields 7 and 8; - an older `sozu` CLI doing read-modify-write (`cluster h2 enable|disable` re-sends `AddCluster` without fields 7 and 8); - an older binary loading a new saved state. Downgrading while a cluster uses either field is not supported. **Library API (BREAKING):** - `HealthCheckConfig` gains the public fields `mode` and `accepted_statuses`. - `FileHealthCheckConfig` gains `mode` and `accepted_statuses`. - `FileHealthCheckConfig::to_proto` now returns `Result`. - `ConfigError` gains `InvalidHealthCheck`. ## Pre-existing bug fixed: `health-check set` always panicked The subcommand's `--timeout` (u32) had the same clap id as the global `-t/--timeout` command timeout (u64), so every invocation panicked before sending anything: ``` $ sozu cluster health-check set --id app --uri /x thread 'main' panicked at bin/src/cli.rs:25:18: Mismatch between definition and access of `timeout`. Could not downcast to u64, need to downcast to u32 ``` The probe timeout flag is now `--probe-timeout` (clap id `probe_timeout`). **`--timeout` on `health-check set` is now the global command timeout, in milliseconds, not the probe timeout.** `cluster_health_check_set_parses_mode_and_accepted_statuses` parses both timeouts together; it failed with this panic before the rename. ## Protocol / security impact - Two new optional `HealthCheckConfig` fields; no existing field number or type changes. - The TCP probe writes nothing to the backend. - URI validation (CR/LF/NUL/C0) still applies in every mode. The leading `/` is only required in HTTP mode, the only mode that puts the URI on the wire. ## Tests | Test | What it covers | | --- | --- | | `test_health_check_tcp_mode_marks_blackholed_and_refused_backends_down` | Blackhole: full accept queue, so the probe times out. Refused: closed port, so the probe sees the socket error. Both go DOWN. This guards the connect verdict. | | `test_health_check_tcp_mode_keeps_500_and_404_backends_up` | Backends that accept but answer 500/404 stay up. This guards the mode switch. | | `test_health_check_http_mode_accepted_statuses_keep_404_backend_up` | `200-499` keeps a backend that answers 404 on the probe path up. | | `test_health_check_legacy_config_marks_404_backend_down` | A legacy config still accepts only 2xx. | | Unit tests in `lib/src/health_check.rs` | Status matching for lists, ranges and `any`; the 2xx default being replaced; `tcp_connect_outcome`; `http1_interim_responses_are_skipped_for_the_final_status`; `h2c_interim_headers_are_skipped_for_the_final_status`. | | Unit tests in `command/src/config.rs` | Range parser and its `Display` round trip; validation (mode, TCP URI exemption, TCP status refusal, bounds, exclusivity); TOML to `AddCluster`. | | Unit test in `command/src/state.rs` | `set_health_check_tcp_mode_with_status_field_rejected`. | | Unit test in `bin/src/cli.rs` | CLI parsing of the new flags and of both timeouts. | The e2e tests are in `e2e/src/tests/health_check_mode_tests.rs`. They read the worker's `health_check.{success,failure,down}` counters and skip the event responses the checker queues on the command channel. Note: the `Blackhole` helper duplicates the one in the in-flight #1800 failover branch (`e2e/src/tests/backend_failover_tests.rs`). Whichever PR lands second should deduplicate it. **Seen red.** On `main` the new tests do not compile, because the fields are absent. I also ran mutation rounds on a `git archive` copy of this branch: 1. TCP mode forced back to the HTTP probe: `tcp_mode_keeps_500_and_404_backends_up` fails. 2. `accepted_statuses` ignored: the accepted e2e test and the four `accepted_statuses_*` unit tests fail. 3. Exclusivity check removed: the `command/src/config.rs` health-check tests fail. 4. `tcp_connect_outcome` ignores socket errors: the blackholed/refused e2e test and `tcp_connect_outcome_reports_established_and_refused` fail. 5. `expected_status = 0` accepts any status: the legacy e2e test, `test_is_status_healthy_any_2xx` and two parse tests fail. 6. Interim skip removed: both interim tests fail. A decoder rebuilt per h2c block also fails `h2c_interim_headers_are_skipped_for_the_final_status` (`left: Some(false) right: Some(true)`): its `x-hint` header is outside the HPACK static table, so the final block references the dynamic table. 7. TCP-mode status refusal removed: the state test and the two `command/src/config.rs` health-check tests fail. ## Commands run (all exit 0 on `d743a112`, after merging `main` at `3aec7035`) ``` cargo +nightly fmt --all --check cargo +1.93.1 clippy --all-features --all-targets --locked -- -D warnings cargo +1.93.1 test -p sozu-lib --lib --all-features --locked # 1578 passed python3 .github/scripts/check_doc_citations.py --root . --base origin/main ``` On `4a9290f6`, the previous head: `sozu-command-lib` + `sozu` tests all passed and the health_check e2e tests passed (10). The full e2e suite passed (592/592) on `9f7d62a4`. Neither was rerun on `d743a112`, which changes only a unit test and docs on this branch, plus the merge of `main`. ## Docs `doc/health_checks.md` (modes, parameters, accepted-status syntax, interim responses, TOML case and the UDP `TCP_PROBE` naming, upgrade warning, `--probe-timeout` vs global `--timeout`, list output), `doc/configure.md`, `doc/metrics.md`, `bin/config.toml`, `CHANGELOG.md`. Signed-off-by: Florentin Dubois <florentin.dubois@clever.cloud>

Commit:4a9290f
Author:Florentin Dubois

fix(health-check): judge the final status, refuse status fields in TCP mode, document upgrades HTTP probes skip interim 1xx responses (except 101) on HTTP/1.1 and h2c, with one HPACK decoder for the whole h2c walk, so accepting 1xx or any never lets a 100 Continue stand in for a final status. validate_health_check_config refuses a non-zero expected_status or a non-empty accepted_statuses with mode TCP, which judges no status. Document that an older worker, an older CLI patching a cluster by read-modify-write, or an older binary loading a saved state ignores fields 7 and 8 and runs GET <uri> accepting only 2xx; that --timeout on health-check set is the global command timeout; the library API break; the TOML case of mode against the UDP TCP_PROBE; and the RST a banner backend can see when the probe closes. Refs #1801 Signed-off-by: Florentin Dubois <florentin.dubois@clever.cloud>

Commit:aefec78
Author:Florentin Dubois
Committer:Florentin Dubois

Merge remote-tracking branch 'origin/main' into feat/health-check-tcp-mode Signed-off-by: Florentin Dubois <florentin.dubois@clever.cloud>

Commit:c005f03
Author:Florentin Dubois

Merge remote-tracking branch 'origin/main' into fix/backend-connect-timeout-failover Signed-off-by: Florentin Dubois <florentin.dubois@clever.cloud>

Commit:c81ded1
Author:Florentin DUBOIS
Committer:GitHub

feat(mux-h2): refuse new streams before the pre-response RST_STREAM cap (#1798) Fixes #1797 ## What The pre-response RST_STREAM cap (`h2_max_rst_stream_abusive_lifetime`) ends the connection with `GOAWAY(ENHANCE_YOUR_CALM)`, and every stream in flight on it goes with it. This adds a soft state below the cap. New client streams are refused with `RST_STREAM(REFUSED_STREAM)`, which RFC 9113 §8.7 makes safe to retry, and the streams already open keep being served. The cap itself is unchanged. ## Design `H2FloodDetector::refuses_new_streams` returns true only when all three of these hold: - the pre-response resets are past `h2_stream_refusal_percent` of the cap's floor (default 50); - pre-response resets already outnumber the backend-routed streams. This is the cap's own ratio, so a client that would never trip the cap is never refused; - the last pre-response reset is less than one flood window (1 s) old. The pre-response count never decays, and refused streams cannot raise the answered count, so this condition is what ends the refusal once the client stops cancelling. `ConnectionH2::handle_header_state` checks it after the drain check and before the MAX_CONCURRENT_STREAMS check, on server connections only. The refusal goes through `enqueue_rst` with `RstOrigin::Local`. It records no glitch and does not trigger the SETTINGS back-pressure, which would halve MAX_CONCURRENT_STREAMS for the rest of the connection over a state that decays. A refused stream is never registered, so its id never enters `rst_sent`. The refusal therefore feeds no flood counter, and in particular not the provoked-RST (MadeYouReset) cap. A refused stream never got a response. `ConnectionH2::flood_refused_streams` keeps the latest contiguous run of refused ids. When the client resets one of them, the reset counts as a pre-response reset, and `flood_refusal_ignored` ends soft refusal for that connection. A client that ignores REFUSED_STREAM then meets the cap exactly as it does today, with no extra RST_STREAM frames in front of the GOAWAY. Config: `h2_stream_refusal_percent` (`u32`; `0` disables; `100` or more never refuses before the cap). It is available in TOML, in `command.proto` (`HttpListenerConfig` 37, `HttpsListenerConfig` 50, `UpdateHttpListenerConfig` 43, `UpdateHttpsListenerConfig` 44) and as `--h2-stream-refusal-percent` on `sozu listener http|https update`. Every value has a meaning, so it has no floor and no clamp. New counter: `h2.flood.stream_refused`. `H2FloodConfig::new` / `from_optional` take a fourteenth argument (library API). ## Trade-offs - **The per-window RST_STREAM rate has no soft state.** I implemented it first and dropped it. That counter counts every reset, answered streams included, and it half-decays every window, so a client sustaining half the rate never reaches the limit. In practice it refused streams from `test_h2_cancels_during_a_backend_outage_keep_the_connection`, a client the limit never cuts off. - **PING, SETTINGS, empty DATA, CONTINUATION and stream-0 WINDOW_UPDATE floods are unchanged.** They open no streams. - **The provoked-RST cap has no soft threshold of its own.** Provoked resets still count toward the shared pre-response ratio. - **A cancel that races its own refusal ends soft refusal for that connection.** This happens when the client resets a stream before the REFUSED_STREAM reaches it. The connection then falls back to today's behaviour. - **While refusing, Sōzu still decodes each refused HEADERS block (HPACK) and queues one RST_STREAM.** This is bounded by the pending-RST queue bound per pass and by the one-window lifetime of the soft state. ## Tests Red runs were made against a `git archive origin/main` copy with only the test file copied in, or against a scratch copy carrying the named mutation. Green runs were made in the branch. - `test_h2_cancels_past_the_soft_threshold_refuse_new_streams` (e2e, new). The client keeps an upload open, cancels one past the soft threshold and under the floor, then opens a probe stream. Red on origin/main: `probe RST_STREAM code=None`. Green: `code=Some(7)`, the upload completes with 200, there is no GOAWAY, and a stream opened after 1.5 s is served. - `resets_of_streams_refused_under_flood_count_as_pre_response` (h2.rs unit). Red when refused-stream resets are not counted as pre-response. With the same mutation, `test_h2_rapid_reset_triggers_goaway` and `e2e_h2_flood_rapid_reset_abusive_lifetime` both go red (no GOAWAY). - Detector unit tests: entering and leaving the soft state, the ratio condition, the hard limits unchanged, `0` and `100` inert, and the RST rate ignored. - Rapid-reset attacker tests still end in `GOAWAY(ENHANCE_YOUR_CALM)`. ## Validation - `cargo +nightly fmt --all -- --check`: ok - `cargo +1.93.1 clippy --all-features --all-targets --locked -- -D warnings`: ok - `RUSTDOCFLAGS="-D warnings" cargo doc --all-features --no-deps`: ok - `cargo test -p sozu-lib --all-features`: 1536 passed - `cargo test -p sozu-command-lib -p sozu --all-features --locked`: 527 passed - `cargo test -p sozu-e2e --locked --features opentelemetry,splice,simd -- --test-threads=4 flood rapid_reset goaway h2spec`: 43 passed - `cargo +nightly fuzz build`: ok - `check_doc_citations.py` `--self-test`, `--root .`, `--root . --base origin/main`, `--audit --root .`: all ok Docs: `doc/configure.md` (new section, reference row, metric), `doc/h2_mux_internals.md`, `lib/src/protocol/mux/LIFECYCLE.md`, `bin/config.toml`, `CHANGELOG.md`. Signed-off-by: Florentin Dubois <florentin.dubois@clever.cloud>

Commit:9f7d62a
Author:Florentin Dubois
Committer:Florentin Dubois

feat(health-check): add a TCP connect probe mode and configurable accepted HTTP statuses `HealthCheckConfig` gains a `mode` (proto field 7): `HTTP`, the default and the previous behaviour, or `TCP`, where a backend passes as soon as the TCP connection is established within `timeout`. Nothing is sent, the connection is closed once established, and `uri` and the status fields are ignored. A platform can then enable health checks on every cluster to take blackholed backends out of rotation without marking down applications that answer 3xx, 401, 404 or 500 on a fixed path. HTTP mode gains `accepted_statuses` (proto field 8, inclusive `HttpStatusRange` entries; TOML/CLI entries "404", "200-399", "2xx" or "any", bounds within 100-599). It replaces `expected_status` when set; setting both a non-zero `expected_status` and a list is refused by `validate_health_check_config` at every boundary. `uri` stays a required proto field (may be empty in TCP mode); the TOML key and `--uri` default to "/". Messages and saved states without the new fields decode as HTTP mode with no list, so existing configurations keep their meaning. `sozu cluster health-check set` gains `--mode` and `--accepted-statuses`; `list` shows the mode and accepted statuses. The probe timeout flag of `health-check set` becomes `--probe-timeout`: its `--timeout` shared the clap id of the global `-t/--timeout` command timeout (u64 against u32), so every `health-check set` invocation panicked on the downcast before sending anything. Fixes #1801 Signed-off-by: Florentin Dubois <florentin.dubois@clever.cloud>

Commit:b63963d
Author:Florentin Dubois

fix(mux): fail over to another backend on connect timeout A backend that accepted no connection hit `connect_timeout` and the request was answered 504. The timeout path marked the connection `Disconnecting`, so the dead-backend branch of `Mux::ready_inner`, the only place a connect failure is counted, never saw it: the backend's retry policy was never armed and the load balancer kept choosing it. A connect timeout is now a backend connect failure like a refused dial. `Mux::timeout_inner` flags the connecting connection's readiness ERROR and `Server::timeout` runs the `ready` pass only it can trigger (`ProxySession::needs_ready_pass`); the dead-backend branch counts the failure, closes the connection and re-links the request, which sent no byte, through `EndStreamAction::Reconnect`. 504 stays the answer to a response timeout on an established connection. On the same path: - a WRITABLE event on a connecting socket reads `SO_ERROR` and `getpeername(2)` before the connection is marked established: an error is a connect failure, a dial still in flight (a WRITABLE sozu raised itself, as for an HTTP/2 backend's preface) waits for the kernel edge; - a synchronous `connect(2)` failure is retried instead of answered 503; - each retry skips the backends that already failed for the request, under every load-balancing policy, falling back to the plain selection when every selectable backend was tried; - the retry budget replaces the hard-coded `CONN_RETRIES = 3` with `max_connection_attempts`: global (default 5) and per cluster, in TOML, the protocol and the CLI, changeable at runtime with `sozu connection-attempts set` and `sozu cluster connection-attempts`. Fixes #1800 Signed-off-by: Florentin Dubois <florentin.dubois@clever.cloud>

Commit:4be6a69
Author:Florentin Dubois

feat(mux-h2): refuse new streams before the pre-response RST_STREAM cap The pre-response RST_STREAM cap (h2_max_rst_stream_abusive_lifetime) ends the connection with GOAWAY(ENHANCE_YOUR_CALM), and every stream in flight on it. Add a soft state below it: once a connection's pre-response resets are past h2_stream_refusal_percent (default 50, 0 disables) of the cap's floor, already outnumber its backend-routed streams, and the last one is less than a flood window old, each new client stream is refused with RST_STREAM(REFUSED_STREAM), retryable per RFC 9113 §8.7, while the open streams keep being served. The refusal feeds no flood counter: it is a local reset, records no glitch, and skips the SETTINGS back-pressure. A reset of a refused stream counts as a pre-response reset and ends the refusals on that connection, so a client that ignores them meets the cap exactly as before. The per-window RST_STREAM rate has no soft state: it counts every reset, and with its half-decay a client at half the rate never reaches the limit. New listener key h2_stream_refusal_percent (TOML, protobuf, sozu ctl) and counter h2.flood.stream_refused. Fixes #1797 Signed-off-by: Florentin Dubois <florentin.dubois@clever.cloud>

Commit:933eba0
Author:Florentin DUBOIS
Committer:GitHub

fix(mux-h2): raise H2 flood-protection defaults and scale RST caps with answered streams (#1756) Raise H2 flood-protection defaults that tripped on legitimate traffic; scale RST caps with the streams a backend answered; stop counting Sōzu-initiated resets. Fixes #1749 ## Changes - **Stream-0 WINDOW_UPDATE**: an update answering DATA Sōzu sent is credited (two per DATA frame, after Envoy's `max_inbound_window_update_frames_per_data_frame_sent`); stored credit is capped by the DATA sent recently and decays once DATA stops. `h2_max_window_update_stream0_per_window` bounds only unsolicited updates (RFC 9113 §6.9 leaves the cadence to the peer). - **RST caps relative to backend-routed streams**: the three `h2_max_rst_stream_*_lifetime` knobs become floors. Backend-routed streams are those a backend answered, those Sōzu answered 502/503/504 after selecting a backend, and those it answered 503 because every backend of the cluster was failing; refused streams and streams answered without a backend for another reason (no route, a cluster with no backend, a session or buffer limit) do not count. Pre-response resets received and peer-provoked resets emitted share one count and trip once they exceed the streams a backend answered, i.e. more than half of the backend-routed streams (Envoy's 50 % line). `h2_max_rst_stream_lifetime` compares only the resets received after a response started or on a closed stream with the answered streams; a stream reset after its response counts as answered before the check, so cancelling answered streams never trips it. No absolute per-connection reset ceiling is added. - **Sōzu-initiated resets** (idle-reaper `CANCEL`, `REFUSED_STREAM`, `STREAM_CLOSED`, converter errors on a backend failure) no longer feed the CVE-2025-8671 cap. Each peer-provoked reset also counts as a glitch. - **Refused streams**: counted as glitches once the client has acknowledged our SETTINGS (buffer-pool refusals excepted); their ids are no longer kept in the per-connection reset set. - **Backend connections**: a `RST_STREAM` a backend sends before its response is not counted as Rapid Reset. - **Pending RST queue**: bounded on what is pending (at least 4000, or four per advertised `h2_max_concurrent_streams`); GOAWAY only when an RST could not be queued. - **Defaults ×20** for every count threshold: per-window RST/PING/empty-DATA/stream-0 WINDOW_UPDATE 2000, SETTINGS 1000, glitch 2000, RST floors 200 000 / 1000 / 10 000, PING/SETTINGS lifetime 200 000, pending RST floor 4000, back-pressure 1000 refusals per 60 s. - **Memory bounds unchanged**: `h2_max_header_list_size`, `h2_max_header_fields` (128), `h2_max_continuation_frames`, `h2_max_header_table_size`, the PRIORITY map size and buffer sizes. - `doc/configure.md`, `doc/h2_mux_internals.md`, `LIFECYCLE.md`, `bin/config.toml`, proto/config/CLI help and `CHANGELOG.md` updated. Reference defaults compared: nghttp2 (reset burst 1000 / 33 s⁻¹, glitch 10 000 / 330 s⁻¹, outbound queue 1000), Go x/net/http2 (queued control frames 10 000), hyper h2 (local error resets 1024), Envoy (premature reset ≥ 50 % of ≥ 500 streams, WINDOW_UPDATE per DATA frame sent, outbound control frames 1000). ## Tests New, each seen failing without its fix and passing with it: - e2e `test_h2_pre_response_cancels_keep_the_connection`, `test_h2_per_frame_connection_window_updates_do_not_trip` - `local_resets_are_not_charged_to_the_peer`, `backend_resets_before_the_response_are_not_rapid_reset` - e2e `test_h2_cancels_during_a_backend_outage_keep_the_connection`; `test_flood_detector_pre_response_cap_trips_only_above_half`, `test_flood_detector_window_update_credit_covers_data_in_flight` - `refused_streams_keep_rst_sent_bounded_and_trip_the_glitch_budget`, `peer_provoked_resets_count_as_glitches`, `unanswered_streams_do_not_dilute_the_reset_caps`, `test_flood_detector_pre_response_caps_share_one_count` - `resets_after_the_response_never_trip_the_lifetime_cap`, `pre_response_resets_do_not_count_toward_the_lifetime_cap`, `a_5xx_answered_without_a_backend_is_not_routed`, `a_5xx_answered_during_a_backend_outage_is_routed`; guard `resets_of_streams_already_gone_trip_the_lifetime_cap` - detector and control-queue unit tests (ratio caps, WINDOW_UPDATE credit and its cap, pending bound) Existing flood tests keep their assertions; only their frame counts rise above the new thresholds. ## Validation - `cargo +nightly fmt --all -- --check` - `cargo +1.93.1 clippy --all-features --all-targets --locked -- -D warnings` - `RUSTDOCFLAGS="-D warnings" cargo doc --all-features --no-deps` - `cargo test --all-features --locked -p sozu-command-lib -p sozu-lib -p sozu` - targeted `cargo test -p sozu-e2e --locked --features opentelemetry,splice,simd` (flood, rapid reset, GOAWAY, h2spec) - `cargo +nightly fuzz build` - `check_doc_citations.py` in its four modes Signed-off-by: Florentin Dubois <florentin.dubois@clever.cloud>

Commit:931d7bf
Author:Florentin Dubois

Merge branch 'main' into fix/h2-flood-thresholds Resolve the overlap with #1753 in the closed-stream arms of ConnectionH2::handle_frame. Main's HEADERS arm for a stream this endpoint reset (discard the payload, continue) is kept as is. On the DATA arm, the RST_STREAM(STREAM_CLOSED) sent for a stream not reset locally takes this branch's third enqueue_rst argument, RstOrigin::Local: that DATA is already counted as a glitch just above, which bounds it, and charging it to the provoked-RST cap too would count it twice. The old comment's other reason, that such DATA is usually the tail of a stream sozu reset, no longer applies: main now ignores that case before this call. doc/configure.md keeps this branch's flood-threshold rows and main's h2_initial_connection_window default; doc/h2_mux_internals.md citations are re-anchored to the merged code. Signed-off-by: Florentin Dubois <florentin.dubois@clever.cloud>

Commit:99e6aee
Author:Florentin DUBOIS
Committer:GitHub

feat(udp): cap concurrent flows per source address and subnet (#1770) A UDP cluster can now cap concurrent flows per source address and per masked subnet with its own `max_connections_per_ip` and `max_connections_per_subnet` (with the worker's `subnet_ipv4_prefix` / `subnet_ipv6_prefix`). The limit is opt-in: the global defaults do not apply to UDP, so existing configurations admit the same UDP flows as before. - `UdpManager` counts live flows per `(cluster, source IP)` and `(cluster, masked subnet)`: incremented on admission, decremented in `close_flow`, checked before the `max_flows` cap. A datagram that would open a flow over either limit is dropped (`DropReason::Shed`), counted in `udp.flows.shed` and the new `udp.flows.shed.source_limit`. Datagrams of existing flows are not affected. - Counting is unconditional, so a cluster limit set at runtime (an `AddCluster` upsert) applies to the flows already open. - Counters are per UDP listener and separate from the TCP/HTTP connection counters. - Library API: `udp::ClusterConfig` gains four fields and `udp::MetricEvent` gains `FlowShedSourceLimit`. Tests: `test_udp_per_ip_flow_limit` (e2e), per-IP / per-subnet / runtime-limit / slot-release unit tests in `manager.rs`, `udp_flow_limits_come_from_the_cluster_not_the_global_defaults` in `server.rs`, the UDP simulation and fuzz harnesses with random per-source caps, and the flow-count invariant checked after every manager input. Docs: `doc/configure.md`, `doc/rate-limit-design.md` §3.5.1, `LIFECYCLE.md`, CHANGELOG. Fixes #1769 Signed-off-by: Florentin Dubois <florentin.dubois@clever.cloud>

Commit:9117603
Author:Florentin DUBOIS
Committer:GitHub

fix(mux-h2): emit fewer connection-level WINDOW_UPDATE frames when receiving DATA (#1752) Sōzu sends one stream-0 `WINDOW_UPDATE` per half of `h2_initial_connection_window` it receives (RFC 9113 §6.9). With the 1 MiB default, a 64 MiB transfer costs 125 of these frames. That happens whether Sōzu is the H2 server receiving a request body or the H2 client receiving a backend response. At LAN speed this rate can trip a peer's stream-0 `WINDOW_UPDATE` flood detection, including Sōzu's own. This raises the default to 16 MiB. For comparison, Chromium uses 15 MB, Envoy 24 MiB and Go's HTTP/2 client 1 GiB. The same transfer now costs 8 frames. The window is advertised but not enforced: DATA is read only when its stream buffer has room, so the buffer pool still bounds memory. Listeners that set the key explicitly keep their value. User-space memory stays bounded by the buffer pool. The kernel cost does grow: a connection Sōzu stops reading may now hold up to min(window, `h2_max_concurrent_streams` × 65535) octets, about 6.25 MiB by default instead of about 1 MiB, in its socket receive buffer. This is documented in `doc/configure.md`. On a backend connection, the one-shot connection-window grant fired on every SETTINGS frame while Sōzu's own send window was at most 65535, not only on the first. A backend re-sending SETTINGS could thus be pushed past 2^31-1 (RFC 9113 §6.9.1). The grant is now tracked by a flag and sent once per connection; `repeated_backend_settings_enlarge_the_connection_window_once` covers it. Stream-level credit is still returned per DATA frame. At the default 65535-octet stream window, returning it per half window halved single-stream throughput on loopback. A larger stream window is a separate decision. Tests: `e2e/src/tests/h2_window_update_tests.rs` moves 64 MiB through Sōzu in each position, with a sender that honours the windows Sōzu grants. Each test asserts that the transfer completes and that Sōzu sends at most 1 + 64 MiB / 8 MiB stream-0 `WINDOW_UPDATE` frames. With the 1 MiB default both tests fail (125 > 9); with 16 MiB both pass (8). Commands run: `cargo +nightly fmt --all -- --check`, `cargo +1.93.1 clippy --all-features --all-targets --locked -- -D warnings`, `RUSTDOCFLAGS="-D warnings" cargo doc --all-features --no-deps`, `cargo test -p sozu-lib --all-features -- protocol::mux`, targeted `cargo test -p sozu-e2e` (window / flow-control / large-response / upload), `cargo +nightly fuzz build`, `check_doc_citations.py` (self-test, root, base origin/main, audit). Docs: `doc/configure.md`, `doc/h2_mux_internals.md`, `bin/config.toml`, proto and config comments, CHANGELOG. Fixes #1744 Signed-off-by: Florentin Dubois <florentin.dubois@clever.cloud>

Commit:5867edb
Author:Florentin Dubois

Merge branch 'main' into fix/h2-window-update-coalescing Signed-off-by: Florentin Dubois <florentin.dubois@clever.cloud>

Commit:f891b52
Author:Florentin Dubois
Committer:Florentin Dubois

fix(mux-h2): stop the received-RST cap tripping on answered-stream cancels A frontend stream reset after its response started now counts as answered before the RST caps are checked, and the received-RST lifetime cap compares only the resets outside the pre-response count (after a response, or on a stream already gone) with the answered streams. A client that cancels every stream after its response, or mixes pre- and post-response cancels, no longer trips it past its floor. A 502/503/504 stream counts toward the caps' denominator only when a backend was selected for it, or when its cluster has backends and every one of them is failing (the outage case, flagged on the request context). A 503 for a cluster with no backend, a session or buffer limit hit before selection, or a routing error no longer dilutes the ratio. Docs, CLI help and the configuration example now state the exact ratios. Signed-off-by: Florentin Dubois <florentin.dubois@clever.cloud>

Commit:63b0f60
Author:Florentin Dubois
Committer:Florentin Dubois

fix(mux-h2): trip pre-response RST caps above half of backend-routed streams - Trip the pre-response caps once pre-response resets exceed the answered streams, i.e. more than half of all backend-routed streams were reset before their response. - Count a stream answered 502/503/504 by Sōzu, or with a selected backend, as backend-routed. - Size the stream-0 WINDOW_UPDATE credit cap on the DATA frames sent recently, decaying with the flood window. Refs #1749 Signed-off-by: Florentin Dubois <florentin.dubois@clever.cloud>

Commit:9575925
Author:Florentin Dubois
Committer:Florentin Dubois

fix(mux-h2): bound refused-stream state and share the pre-response RST ratio - Do not record never-registered stream ids (refused streams, DATA on a closed stream) in the RST dedupe set; only stream-table eviction removes an id from it. - Count every refused stream as a glitch once the peer has acknowledged our SETTINGS (buffer-pool refusals excepted). - Compare pre-response resets received and peer-provoked resets emitted as one count against half of the answered streams. - Count each peer-provoked reset as a glitch. - Count toward the RST caps' denominator only the frontend streams a backend answered. - Cap the stored stream-0 WINDOW_UPDATE credit. Refs #1749 Signed-off-by: Florentin Dubois <florentin.dubois@clever.cloud>

Commit:17acaf0
Author:Florentin Dubois
Committer:Florentin Dubois

fix(mux-h2): raise H2 flood-protection defaults and make RST caps decay Raise H2 flood-protection defaults that tripped on legitimate traffic; make RST caps decay; stop counting Sōzu-initiated resets. - Stream-0 WINDOW_UPDATEs answering DATA sent are credited (two per DATA frame); h2_max_window_update_stream0_per_window bounds only unsolicited ones. - The received, pre-response and peer-provoked emitted RST caps trip only once their count also exceeds the streams opened (received) or half of them (the other two); the configured values become floors. - Resets Sōzu decides on its own (idle CANCEL, REFUSED_STREAM from its concurrency limit, back-pressure or buffer pool, STREAM_CLOSED, converter errors) no longer count toward the CVE-2025-8671 cap. - On a backend connection, backend resets before the response are not counted as Rapid Reset. - The pending RST_STREAM queue bound limits what is pending, not the connection's lifetime; GOAWAY only when an RST could not be queued. - Count defaults x20: per-window RST/PING/empty-DATA/WINDOW_UPDATE 2000, SETTINGS 1000, glitch 2000, RST lifetime floors 200000/1000/10000, PING/SETTINGS lifetime 200000, pending RST floor 4000, back-pressure 1000 refusals per 60 s. Memory bounds are unchanged. - A stream beyond the advertised max_concurrent_streams counts as a glitch (RFC 9113 §5.1.2). Fixes #1749 Signed-off-by: Florentin Dubois <florentin.dubois@clever.cloud>

Commit:81218ca
Author:Florentin Dubois
Committer:Florentin Dubois

feat(udp): cap concurrent flows per source address and subnet Each client source IP and port is its own UDP flow, with its own connected upstream socket and max_flows slot, and nothing bounded the flows one source address held. UdpManager now counts live flows per (cluster, source IP) and per (cluster, masked source subnet), incremented on admission and decremented in close_flow, and sheds a datagram that would open a flow over the cluster's own max_connections_per_ip / max_connections_per_subnet (with the worker's subnet_ipv4_prefix / subnet_ipv6_prefix) through DropReason::Shed, counted in udp.flows.shed and the new udp.flows.shed.source_limit. Datagrams of existing flows are never refused. The limit is opt-in per cluster: the global max_connections_per_ip / max_connections_per_subnet defaults are not inherited by UDP, so an existing configuration admits the same UDP flows as before. Signed-off-by: Florentin Dubois <florentin.dubois@clever.cloud>

Commit:fcdec35
Author:Florentin DUBOIS
Committer:GitHub

fix(udp): key flows on the client source address, not on the affinity key (#1746) UDP flows were keyed on the affinity key: under the default `affinity_key = SOURCE_IP`, `FlowKey::from_src` zeroed the source port, so every socket of one client IP shared one flow and one connected upstream socket, and backend replies went to the port of the socket that opened the flow. The flow table is now keyed on the client source IP and port whatever the affinity key, as the #1273 design states (virtual 4-tuple flows, one connected upstream socket per flow). `affinity_key` only feeds the backend-selection hash, so the sockets of one IP still land on one backend under HRW / Maglev. - `FlowKey::from_src` loses its `with_port` argument; `UdpManager::affinity_with_port` and the shell's port normalisation are removed. - Each source port of a client IP now holds its own flow, upstream socket and `max_flows` slot. - The idle-expiry part of the report does not reproduce on `main`: it matches the lost timer wakeup fixed in 596b3ec0, which is not in a release yet. This adds the end-to-end idle-eviction test that fix lacked. - Docs (`doc/configure.md`, `LIFECYCLE.md`, proto comment) and CHANGELOG updated. Tests: `same_ip_clients_get_distinct_flows_and_their_own_replies` (unit), `test_udp_same_ip_clients_are_distinct_flows` and `test_udp_idle_flow_is_torn_down` (e2e). Fixes #1732 Signed-off-by: Florentin Dubois <florentin.dubois@clever.cloud>

Commit:8614c7e
Author:Florentin Dubois
Committer:Florentin Dubois

fix(udp): key flows on the client source address, not on the affinity key Under the default affinity_key = SOURCE_IP, FlowKey::from_src zeroed the source port, so every socket of one client IP shared one flow and one connected upstream socket, and every backend reply went to the port of the socket that opened the flow. The flow table is now keyed on the client source IP and port whatever the affinity key, as the #1273 design states; the affinity key only feeds the backend-selection hash, so the sockets of one IP still land on one backend under HRW / MAGLEV. FlowKey::from_src loses its with_port argument, UdpManager::affinity_with_port and the shell's client_key normalisation are removed. Also add the end-to-end idle-eviction test the lost-wakeup fix (596b3ec0) lacked: a flow silent past its idle timeout is torn down and the next datagram opens a new one. Fixes #1732 Signed-off-by: Florentin Dubois <florentin.dubois@clever.cloud>

Commit:d244853
Author:Florentin Dubois
Committer:Florentin Dubois

fix(mux-h2): emit fewer connection-level WINDOW_UPDATE frames when receiving DATA Sōzu returns connection credit in one stream-0 WINDOW_UPDATE per half of h2_initial_connection_window received (RFC 9113 §6.9). With the 1 MiB default that is one frame per 512 KiB: a 64 MiB transfer cost 125 of them, enough at LAN speed to trip a peer's stream-0 WINDOW_UPDATE flood detection, Sōzu's own included. Raise the default to 16 MiB (Chromium 15 MB, Envoy 24 MiB): the same transfer now costs 8, as H2 server and as H2 client. The window stays advertised, not enforced; DATA is read only when its stream buffer has room, so the buffer pool still bounds memory. Stream-level credit is still returned per DATA frame: at the 65535-octet stream window, returning it per half window halved single-stream throughput on loopback. Fixes #1744 Signed-off-by: Florentin Dubois <florentin.dubois@clever.cloud>

Commit:b33416c
Author:Florentin Dubois

fix(mux-h2): trip pre-response RST caps above half of backend-routed streams - Trip the pre-response caps once pre-response resets exceed the answered streams, i.e. more than half of all backend-routed streams were reset before their response. - Count a stream answered 502/503/504 by Sōzu, or with a selected backend, as backend-routed. - Size the stream-0 WINDOW_UPDATE credit cap on the DATA frames sent recently, decaying with the flood window. Refs #1749 Signed-off-by: Florentin Dubois <florentin.dubois@clever.cloud>

Commit:eabfa45
Author:Florentin Dubois
Committer:Florentin Dubois

fix(udp): key flows on the client source address, not on the affinity key Under the default affinity_key = SOURCE_IP, FlowKey::from_src zeroed the source port, so every socket of one client IP shared one flow and one connected upstream socket, and every backend reply went to the port of the socket that opened the flow. The flow table is now keyed on the client source IP and port whatever the affinity key, as the #1273 design states; the affinity key only feeds the backend-selection hash, so the sockets of one IP still land on one backend under HRW / MAGLEV. FlowKey::from_src loses its with_port argument, UdpManager::affinity_with_port and the shell's client_key normalisation are removed. Also add the end-to-end idle-eviction test the lost-wakeup fix (596b3ec0) lacked: a flow silent past its idle timeout is torn down and the next datagram opens a new one. Fixes #1732 Signed-off-by: Florentin Dubois <florentin.dubois@clever.cloud>

Commit:1c48744
Author:Florentin Dubois

fix(mux-h2): bound refused-stream state and share the pre-response RST ratio - Do not record never-registered stream ids (refused streams, DATA on a closed stream) in the RST dedupe set; only stream-table eviction removes an id from it. - Count every refused stream as a glitch once the peer has acknowledged our SETTINGS (buffer-pool refusals excepted). - Compare pre-response resets received and peer-provoked resets emitted as one count against half of the answered streams. - Count each peer-provoked reset as a glitch. - Count toward the RST caps' denominator only the frontend streams a backend answered. - Cap the stored stream-0 WINDOW_UPDATE credit. Refs #1749 Signed-off-by: Florentin Dubois <florentin.dubois@clever.cloud>

Commit:e140306
Author:Florentin Dubois

fix(mux-h2): raise H2 flood-protection defaults and make RST caps decay Raise H2 flood-protection defaults that tripped on legitimate traffic; make RST caps decay; stop counting Sōzu-initiated resets. - Stream-0 WINDOW_UPDATEs answering DATA sent are credited (two per DATA frame); h2_max_window_update_stream0_per_window bounds only unsolicited ones. - The received, pre-response and peer-provoked emitted RST caps trip only once their count also exceeds the streams opened (received) or half of them (the other two); the configured values become floors. - Resets Sōzu decides on its own (idle CANCEL, REFUSED_STREAM from its concurrency limit, back-pressure or buffer pool, STREAM_CLOSED, converter errors) no longer count toward the CVE-2025-8671 cap. - On a backend connection, backend resets before the response are not counted as Rapid Reset. - The pending RST_STREAM queue bound limits what is pending, not the connection's lifetime; GOAWAY only when an RST could not be queued. - Count defaults x20: per-window RST/PING/empty-DATA/WINDOW_UPDATE 2000, SETTINGS 1000, glitch 2000, RST lifetime floors 200000/1000/10000, PING/SETTINGS lifetime 200000, pending RST floor 4000, back-pressure 1000 refusals per 60 s. Memory bounds are unchanged. - A stream beyond the advertised max_concurrent_streams counts as a glitch (RFC 9113 §5.1.2). Fixes #1749 Signed-off-by: Florentin Dubois <florentin.dubois@clever.cloud>

Commit:99fe7cb
Author:Florentin Dubois

fix(mux-h2): emit fewer connection-level WINDOW_UPDATE frames when receiving DATA Sōzu returns connection credit in one stream-0 WINDOW_UPDATE per half of h2_initial_connection_window received (RFC 9113 §6.9). With the 1 MiB default that is one frame per 512 KiB: a 64 MiB transfer cost 125 of them, enough at LAN speed to trip a peer's stream-0 WINDOW_UPDATE flood detection, Sōzu's own included. Raise the default to 16 MiB (Chromium 15 MB, Envoy 24 MiB): the same transfer now costs 8, as H2 server and as H2 client. The window stays advertised, not enforced; DATA is read only when its stream buffer has room, so the buffer pool still bounds memory. Stream-level credit is still returned per DATA frame: at the 65535-octet stream window, returning it per half window halved single-stream throughput on loopback. Fixes #1744 Signed-off-by: Florentin Dubois <florentin.dubois@clever.cloud>

Commit:c2cad33
Author:Florentin DUBOIS
Committer:GitHub

feat(state): remove a cluster's frontends and backends with it (#1729) Closes #1723 ## What changes Removing a cluster (`RemoveCluster`, `sozu cluster remove`) now removes every frontend and backend that names it, for HTTP, HTTPS, TCP and UDP, in the main process state and in every worker. A frontend without a `cluster_id` (a deny or answer route) stays. - **State.** `ConfigState::remove_cluster` also removes the cluster's `backends`, `tcp_fronts` and `udp_fronts` buckets, plus the matching `http_fronts`/`https_fronts` entries. New debug assertions check two things: nothing naming the cluster remains, and no other cluster's objects are touched. - **Worker.** Before the worker applies `RemoveCluster` to its own state, it calls `ConfigState::cluster_removal_requests` to get the cluster's objects as removal orders. It then drops each one from its proxies through the same removal the object's own order takes: `HttpProxy::remove_http_frontend`, `HttpsProxy::remove_https_frontend`, `TcpProxy::remove_tcp_front`, `UdpProxy::remove_udp_front` and `Server::remove_backend`. Routes, tags, the TCP and UDP listener→cluster mappings and per-backend metrics are therefore cleaned the same way. Finally `BackendMap::remove_cluster` drops the cluster's backend list, health-check config and http2 hint. The worker still sends exactly one answer to the `RemoveCluster`. - **Live sessions.** An established session keeps its backend connection and drains on it. A new request gets the listener's 404, and a new TCP connection is closed. - **Mixed versions.** A worker of this release may receive, from an older main process, the removal of an object its `RemoveCluster` already removed. It answers ok instead of failing on a missing pre/post route. - **`diff()`.** `ConfigState::diff` now computes its frontend and backend diffs against `self` with the cascade already applied. For a removed cluster it emits `RemoveCluster` alone, with no per-object `Remove*` after it. It also adds back, after the `RemoveCluster`, any object the target state still holds for that cluster. The existing `diff` test now expects the new orders. ## Protocol and upgrade impact This is a behaviour change, recorded in CHANGELOG (Unreleased, Changed): - Upgrade the main process and every worker before relying on the cascade. An older worker removes only the cluster definition and keeps routing the cluster's frontends. - A `Remove*Frontend`/`RemoveBackend` sent after the `RemoveCluster` of the same cluster, whose object went with the cluster, is answered ok without any change by the main process and by the workers (`ConfigState::removed_with_its_cluster`), so `sozu cluster remove … && sozu frontend http remove …` keeps working. - An HTTP/HTTPS frontend removal matches only a frontend of the cluster it names. The route key has no cluster in it, so before this change a late `RemoveHttpFrontend(A, K)` could evict the route that cluster B took over under key K. A worker never applies to its proxies a frontend removal that its state refused (NotFound or NoChange). TCP and UDP proxies strip a listener's cluster by address alone, so a stale TCP/UDP removal naming a re-added cluster could otherwise unmap another cluster's listener. - The ok answer for an object gone with its cluster also covers a typo in the cluster id. The main process answer says the cluster does not exist and names the cluster, or the route with no cluster, that now holds the key. - An older main process keeps the removed cluster's objects in its own state, while workers of this release drop them. - A request to a removed cluster's hostname can still be served by another cluster's wildcard or catch-all route that covers the same hostname. - A caller that re-adds frontends or backends after removing a cluster must re-add the cluster first. ## Tests added (each one seen red before the fix) - `command/src/state.rs`: `remove_cluster_removes_its_frontends_and_backends_for_every_protocol`, `diff_removing_a_cluster_relies_on_the_cascade`, `diff_re_adds_a_frontend_the_target_keeps_for_a_removed_cluster`, `a_removed_cluster_leaves_nothing_for_load_state_to_replay`; `diff` updated. - State and worker: `a_stale_frontend_removal_never_removes_another_clusters_route`, which replays the sequence RemoveCluster(A), AddCluster(B), AddHttpFrontend(B, K), then a stale RemoveHttpFrontend(A, K). - Worker: `a_stale_tcp_or_udp_frontend_removal_never_unmaps_another_clusters_listener`, which replays RemoveCluster(A), B takes X, A is re-added on Y, then a stale Remove(A, X), for both TCP and UDP. - `lib/src/server.rs`: `remove_cluster_removes_its_routes_and_backends_from_the_worker`, covering routes, the deny route, the other cluster, the `BackendMap`, a single answer, and ok answers to follow-up old-style removals. - `e2e/src/tests/remove_cluster_tests.rs`: HTTP (404 on new and kept-alive connections) and TCP (new connection closed, established session drains). ## Commands run (all exit 0 unless noted) - `cargo +nightly fmt --all --check` - `cargo +1.93.1 clippy --all-features --all-targets --locked -- -D warnings` - `RUSTDOCFLAGS="-D warnings" cargo doc --no-deps --all-features --locked` - `cargo test -p sozu-command-lib` / `-p sozu-lib` / `-p sozu` - `cargo test -p sozu-e2e -- remove_cluster_tests metrics_lifecycle` - Full `cargo test -p sozu-e2e`: exit 101. 538 passed and 3 failed under full-suite load: `test_h1_global_limit_emits_429`, `test_h2_graceful_shutdown_completes_large_transfer`, `test_proxy_protocol_keepalive_hup`. None of the three uses `RemoveCluster`, and each passed 3 isolated runs out of 3. - `cd fuzz && cargo +nightly fuzz build` - `RUSTFLAGS='--cfg tokio_unstable' cargo test -p sozu-sim` - `check_doc_citations.py` in all four modes: `--self-test`, `--root .`, `--root . --base origin/main`, `--audit --root .` ## Docs `doc/configure.md` (Clusters), `doc/configure_cli.md` (new "Remove a cluster" section), `doc/observability.md`, the `sozu cluster remove` help, and the `remove_cluster` comment in `command/src/command.proto`. Signed-off-by: Florentin Dubois <florentin.dubois@clever.cloud>

Commit:a810211
Author:Florentin DUBOIS
Committer:GitHub

feat(lb): shuffle-shard clients over the HRW ranking of their affinity key (#1712) Refs #524 (PR 2 of 2: shuffle sharding over the HRW ranking; PR 1 was #1706). ## What A cluster can serve each client from a **shard** of its backends instead of all of them. A client that overloads or poisons its backends then takes down only the clients whose shard lies inside its own. References: - Colm MacCárthaigh, [*Workload isolation using shuffle sharding*](https://aws.amazon.com/builders-library/workload-isolation-using-shuffle-sharding/), Amazon Builders' Library, 2019. It gives `C(N, k)` shards against `N / k` partitions: 28 against 4 for eight backends in shards of two. - The shard is the top of the rendezvous ranking: Thaler & Ravishankar, *Using name-based mappings to increase hit rates*, IEEE/ACM ToN 1998, [doi:10.1109/90.663936](https://doi.org/10.1109/90.663936). ## Decisions implemented (maintainer comment on #524) - **Key:** the affinity key from #1706. That is the source IP (post PROXY protocol), or the `affinity_header` / `affinity_cookie` value, falling back to the IP when absent. - **Shard:** `k = max(2, ceil(shard_percent × N / 100))`, capped at `N`. It is the top `k` of the weighted HRW ranking (`hrw_score`, now public) over the cluster's **configured primary** backends. - Backends that are down stay members, so a shard can be exhausted. - It is deterministic across workers and restarts. - A change of `N` moves a client only at the tail of its ranking. - **Activation:** only when `N >= shard_min_backends`, which defaults to 8. Off until `shard_percent` is set. - **Exhausted shard:** - `FALLBACK` (default) spills over to the rest of the cluster, backup tier and fail-open included, and counts `backend.shard.spillover`. - `STRICT` selects nothing (HTTP 503, TCP close) and counts `backend.shard.exhausted`. - In the fail-open regime a strict shard fails open inside itself only. - **Priority:** a sticky cookie naming a live backend wins, even outside the shard. Connection reuse keeps priority. - **Upgrade note:** explicit in `CHANGELOG.md` and `doc/configure.md`. - Nothing changes until `shard_percent` is set. - Once set, the cluster derives a key under every policy. - Every client behind one NAT, or behind an L4 balancer without PROXY, shares one shard. ## Design - `BackendList::select_with_key` (`lib/src/backends.rs`) narrows the primary candidates to the shard before the cluster's `load_balancing` policy runs. Any policy therefore picks inside the shard (`LEAST_LOADED` balances within it, `HRW` pins to its top member with in-shard failover), and retries stay in it. - `next_available_backend_with_key` keeps its signature. `BackendMap::select_backend` (HTTP reserve and TCP connect) and the UDP keyed path read the `ShardOutcome` and count it under the cluster id. - **Zero allocation:** two scratch buffers (`shard_scores`, `shard`) are reserved in `add_backend` beside `candidates`. `select_nth_unstable_by` partitions the ranking in O(N), and `retain` keeps the candidates in list order for `Candidates::new`. - **Key derivation widened:** `cluster_reads_affinity_key` (`router.rs`, shared by the TCP proxy) is true for `HRW`/`MAGLEV` **or** `shard_percent`. Without this, a sharded `ROUND_ROBIN` cluster would silently not shard. - **Config:** - `Cluster` proto fields `shard_percent = 20`, `shard_min_backends = 21`, `shard_mode = 22` (new enum `ShardMode { FALLBACK, STRICT }`). - TOML keys; `--shard-percent` / `--shard-min-backends` / `--shard-strict` on `sozu cluster add`; a `shuffle_sharding` column in the cluster table; the fields in `Debug`. - `validate_shuffle_sharding` runs at config load and on every `AddCluster`: percent in `1..=100`, min ≥ 2, and the knobs are refused without `shard_percent`. - Hot-updatable through `BackendMap::set_shuffle_sharding_for_cluster` from `server.rs::add_cluster`. - **Retry budget interaction (documented):** `CONN_RETRIES = 3`. - With `k = 2`, a request whose two members refuse spills over, or is refused, on its third attempt. - With `k ≥ 3`, that request exhausts its attempts inside the shard (503), and the next one, finding the members in back-off, spills over or is refused at once. - **UDP:** sharding lives in the backend list, so a sharded UDP cluster shards its flows by their flow key. Protocol/security impact: selection narrowing only; no parser, framing or TLS change. `STRICT` can answer 503 while other backends are healthy, by design. ## Red → green Each fix was reverted, the test run, the fix restored, and the test run again. | Mutation | Fails | Evidence | |---|---|---| | `compute_shard` never shards | e2e in-shard, e2e strict, `a_sharded_selection_stays_in_the_shard`, `an_exhausted_shard_refuses_in_strict_and_spills_in_fallback` | `shard ["pong0", "pong2"], reached ["pong0", "pong1", "pong2", "pong3", …]`, `strict answered Some((200, "pong2"))` | | `STRICT` spills over | e2e strict, unit strict | `shard [2, 3] down, strict answered Some((200, "pong0"))`, `key 0: strict never leaves the shard` | | `FALLBACK` refuses | e2e fallback | `shard [0, 3] down, fallback answered Some((503, …))` | | no key for a sharded non-HRW cluster | `only_hrw_maglev_and_sharded_clusters_derive_a_key` | `a sharded cluster derives a key whatever its policy` | | shard collected into a fresh `Vec` | `a_sharded_selection_allocates_nothing` | `256 sharded selections made 256 allocations` | Green: every test above passes after restoring. ## Tests added - **Unit (`lib/src/backends.rs`):** - `a_shard_is_the_top_of_the_hrw_ranking` - `a_shard_changes_only_at_its_tail_when_backends_change`: add with `k` fixed, add with `k` growing, remove a non-member, remove a member - `a_sharded_selection_stays_in_the_shard` (ROUND_ROBIN, LEAST_LOADED, RANDOM, HRW) - `an_exhausted_shard_refuses_in_strict_and_spills_in_fallback` - `a_strict_shard_fails_open_inside_itself` - `a_sticky_backend_outside_the_shard_wins` - `a_sharded_selection_allocates_nothing` - **Unit (`load_balancing.rs`):** `a_shard_size_follows_the_decided_formula`. - **Unit (`router.rs`):** `only_hrw_maglev_and_sharded_clusters_derive_a_key`. - **Command crate:** `shuffle_sharding_reaches_the_add_cluster_order_and_is_validated`, `add_cluster_with_invalid_shuffle_sharding_rejected`. - **e2e (`e2e/src/tests/shuffle_sharding_tests.rs`):** four backends, `shard_percent = 50`, `shard_min_backends = 4`, with the shard predicted via `hrw_score`. The shard's members are left without a listener, so connections to them are refused. - in-shard: every request lands in the predicted shard - FALLBACK: 200 from a non-member - STRICT: 503 ## Validation (head 2cda4f01 on main 2e55942d) - `cargo +nightly fmt --all -- --check`: 0. - `cargo clippy --all-features --all-targets --locked -- -D warnings`: 0. - `cargo +1.93.1 clippy --all-features --all-targets --locked -- -D warnings`: 0. - `RUSTDOCFLAGS="-D warnings" cargo doc --no-deps --all-features --locked`: 0. - `cargo test -p sozu-lib`: 1420 passed. `-p sozu-command-lib`: 217. `-p sozu`: 123. `RUSTFLAGS="--cfg tokio_unstable" cargo test -p sozu-sim`: ok. - `cargo test -p sozu-e2e -- shuffle_sharding_tests affinity_key_tests cluster_ip_limit`: 15/15. The full e2e suite (before the rebase onto #1708): 532 passed, 0 failed. - `cd fuzz && cargo +nightly fuzz build`: 0. - `check_doc_citations.py`: `--self-test`, `--root .`, `--root . --base origin/main`, `--audit --root .` all 0. ## Docs - `doc/configure.md`: - a new "Shuffle sharding" section: literature, formula, stability, keys, TOML example, composition with policy, retries, sticky, reuse and backups, UDP, and the enable-time note - metrics rows - a "Client affinity" cross-reference - `doc/metrics.md`, `doc/configure_cli.md`, `doc/architecture.md`, `lib/src/protocol/mux/LIFECYCLE.md`, `doc/testing.md`, `bin/config.toml`, `CHANGELOG.md` (Added, with the before-enabling note). - The line citations of `cluster_h2_command` in `doc/configure_admin_ops.md` became prose, because they drift with every edit of `request_builder.rs`. ## Review fixes (FIX-FIRST on a029ac63) - **M1:** `STRICT` now decides between refusing and failing open on the **primary** tier only. The `cluster_can_open` that also counted backups is replaced by `primaries_can_open`, so a healthy backup no longer turns a strict shard's fail-open into a 503. Test: `a_healthy_backup_does_not_turn_a_strict_fail_open_into_a_refusal`. Red with backups counted: `panicked … a strict shard fails open inside itself`. `doc/configure.md` "Backups are outside every shard" now states the STRICT/backup interaction explicitly. - **M2:** the upgrade note in `CHANGELOG.md` and in the `doc/configure.md` "Before enabling it" note now says: - Upgrade the main process and every worker before setting `shard_*`: an older worker ignores fields 20–22 and serves unsharded. - An older CLI's `QueryClusterById` → `AddCluster` read-modify-write (`cluster h2`) drops the fields and silently disables sharding. - **L2:** a `FALLBACK` fail-open pick that lands back in the shard is reported `InShard` (`settle_spill_outcome`) and not counted as `backend.shard.spillover`. Test: `a_fallback_fail_open_pick_inside_the_shard_is_not_a_spill_over`. Red without the settle: `127.0.0.1:20004 is a shard member, left: SpilledOver right: InShard`. - **L3:** tests added. - `a_strict_shard_failing_health_checks_refuses_while_others_are_healthy`. Red with the `primaries_can_open` guard forced false: `key 0: strict must refuse, not fail open`. - Non-uniform weights in `a_shard_is_the_top_of_the_hrw_ranking`, which also asserts that the weights move some shard. - e2e `test_re_adding_a_cluster_without_shard_percent_turns_sharding_off`, re-added under `HRW` so the key is still derived. Red when `Server::add_cluster` skips `set_shuffle_sharding_for_cluster` without `shard_percent`: `sharded Some((503, …)), then unsharded Some((503, …))`. - The backup-tier interaction is covered by M1's test. - **L4:** `compute_shard` counts the primaries and checks `shard_size` / `min_backends` before hashing, so a cluster below the threshold pays no hash. The `has_tier_candidate` walk is gone, including for unsharded clusters. - **L5:** - The weight wording now reads "ranks higher for more keys, not proportional once `k > 1`". - The UDP `DefaultHasher` cross-toolchain caveat is on "across restarts". - The `backend.shard.*` rows are aligned between `configure.md` and `metrics.md`. - The `--affinity-header` / `--affinity-cookie` help mentions that sharding reads the key under any policy. - **L1 (done):** the worker mirrors `validate_shuffle_sharding` on `AddCluster` next to the health-check mirror in `server.rs`. Signed-off-by: Florentin Dubois <florentin.dubois@clever.cloud>

Commit:1ca4b05
Author:Florentin DUBOIS
Committer:GitHub

feat(lb): pass a client affinity key to Maglev and HRW on HTTP and TCP (#1706) Refs #524 (PR 1 of 2: the affinity key; shuffle sharding follows in PR 2). ## Problem Only the UDP datapath handed `HRW` / `MAGLEV` a key. HTTP, HTTPS and TCP selected with `None`, so a cluster configured with either policy silently fell back to round-robin: a client's requests visited every backend in turn, contrary to the documentation. ## Design - **Key derivation.** `affinity_key_from_ip` and `affinity_key_from_value` (`lib/src/load_balancing.rs`) use the `DEFAULT_HASH_SEED`-seeded FNV-1a + splitmix64 hash the policies already use, with distinct domain tags for addresses and values. - Deterministic across workers and restarts; the values are pinned by a unit test. - `::ffff:a.b.c.d` keys as `a.b.c.d`. - **HTTP/HTTPS (H1 and H2).** `Router::plan_connect` derives the key right after routing (`affinity_key` in `router.rs`), for `HRW`/`MAGLEV` clusters only; every other policy derives none and selects exactly as before. - Source: the cluster's `affinity_header` value (case-insensitive name, first occurrence) or `affinity_cookie` value (exact name), else the source IP after the PROXY protocol. - Header blocks and the cookie jar are read in place and hashed without copying. Values are hashed raw: a quoted cookie and its unquoted spelling are two keys. - An `affinity_cookie` named like the listener's sticky cookie reads the value captured before that cookie is elided. - Stored in `HttpContext::affinity_key`, so a replay reuses it like the routed cluster. - **TCP.** `TcpSession::connect_to_backend` keys on `effective_session_address`, which is the PROXY-v2 source when the cluster expects one. - **Where the key enters selection** (on top of #1709's shapes): - HTTP: `BackendSelector::select(cluster, affinity, key)` → `BackendMap::reserve_backend(cluster, key, now)` / `reserve_sticky_backend(cluster, cookie, key, now)` → `select_backend` → `next_available_backend_with_key`. - TCP: `BackendMap::backend_from_cluster_id(cluster, key, now)`. - **Precedence.** - A sticky cookie naming a live backend still wins over the key. - **Connection reuse keeps priority** (owner decision): the key applies only when a new backend connection is dialled. A request reusing a backend connection its session holds (H1 keep-alive, H2 multiplexed) follows that connection. Behind an upstream proxy or CDN multiplexing tenants over warm keep-alive/H2 connections, tenants follow the connection's backend. Documented and pinned by e2e tests. - **Config plumbing.** - Proto `Cluster.affinity_header = 18`, `affinity_cookie = 19`. - TOML keys, `--affinity-header` / `--affinity-cookie` on `sozu cluster add`, an `affinity_key` column in the cluster table, lengths-only `Debug`. - `validate_affinity_key` runs at config load and on every `AddCluster`: at most one of the two, each an RFC 9110 token. `Host` and `Cookie` are refused case-insensitively, since Sōzu never keeps them as headers. - The main state refuses a cluster that names a key and serves a TCP frontend, in both orders (`AddCluster` after `AddTcpFrontend`, and the reverse). The TOML loader refuses the keys on a `tcp` cluster. - Hot-updatable. Protocol/security impact: a header/cookie key is client-chosen, so a client can pick its backend. That is the same property as a sticky cookie, and it is documented. No parser, framing or TLS change. ## Check before upgrading HTTP, HTTPS and TCP clusters already configured with `HRW` or `MAGLEV` ran round-robin and now pin each client by its source IP. Clusters whose clients arrive from few addresses — behind a NAT, or an L4 balancer without the PROXY protocol — concentrate on few backends. Switch policy, enable the PROXY protocol, or key on a header or cookie. The note is in both `CHANGELOG.md` and `doc/configure.md`. ## Allocations - `deriving_an_affinity_key_allocates_nothing`: 256 derivations make 0 allocations. - `stamping_a_dialled_backend_allocates_nothing` is unchanged at 0. ## Red / green Each case was run with the fix reverted, then restored. - **e2e** (`e2e/src/tests/affinity_key_tests.rs`). Each test predicts every key's backend with the library's `Rendezvous`: provable outcomes, not statistical ones. - Key dropped in `RegistrySelector::select` (after the #1709 rebase): HTTP, HTTP header, HTTPS, H2 header/cookie and H2-reuse all fail. For example: `source 127.0.0.2: predicted pong0 …, reached ["pong0","pong1","pong2","pong3"]`. - Key dropped in `TcpSession::connect_to_backend`: `source 10.0.0.1: predicted backend-0 for every session, reached [backend-0 … backend-3]`. - Header/cookie keying disabled: `h2 X-Tenant: tenant-0: predicted pong0 …, reached ["pong3" ×4]`. - Reuse pinning, with reuse disabled in `Router::decide_after_gate` (`reuse_token.filter(|_| false)`): - `one keep-alive connection: predicted Some("pong1") for both … reached [Some("pong1"), Some("pong0")]` - `one H2 connection: predicted Some("pong2") for both … reached [Some("pong2"), Some("pong3")]` - Green: 7/7 (HTTP IP, HTTP header, HTTPS IP+cookie, H2 header+cookie, TCP PROXY-v2, H1 keep-alive reuse, H2 reuse), 3 consecutive runs. - **Unit.** - `key 0 must stay on one backend under HRW` → green. - `256 derivations made 128 heap allocations, expected none` → green. - Sticky-named cookie: `left: Some(1686…) right: Some(1733…)` with the capture arm removed → green. - Also added: pinned hash values, derivation rules, `Host`/`Cookie` refusal, the TCP-frontend/key refusal in both orders. ## Validation (head c70f65ab, rebased on main 4420d132 = #1709) - `cargo +nightly fmt --all -- --check`: 0. - `cargo clippy --all-features --all-targets --locked -- -D warnings`: 0. - `cargo +1.93.1 clippy --all-features --all-targets --locked -- -D warnings`: 0. - `RUSTDOCFLAGS="-D warnings" cargo doc --no-deps --all-features --locked`: 0. - `cargo test -p sozu-lib`: 1407 passed. `-p sozu-command-lib`: 212. `-p sozu`: passed. - `cargo test -p sozu-e2e -- affinity_key_tests cluster_ip_limit`: 11/11. - `cd fuzz && cargo +nightly fuzz build`: 0. - `check_doc_citations.py` `--self-test`, `--root .`, `--root . --base origin/main`, `--audit --root .`: all 0. ## Docs `doc/configure.md` ("Client affinity", reuse behaviour, upgrade check, raw values, refused names, forwarding-policy interactions), `doc/configure_cli.md`, `doc/architecture.md`, `doc/lifetime_of_a_session.md`, `lib/src/protocol/mux/LIFECYCLE.md`, `bin/config.toml`, `doc/testing.md`, `CHANGELOG.md` (Fixed with the upgrade check, Added). Signed-off-by: Florentin Dubois <florentin.dubois@clever.cloud>

Commit:5471bfb
Author:Florentin DUBOIS
Committer:GitHub

feat(listener): make forwarded headers configurable per listener (#1685) ## Summary Sōzu added both forwarding header families (`X-Forwarded-*` and RFC 7239 `Forwarded`) to every forwarded request, unconditionally. This PR adds a per-listener enum for HTTP and HTTPS listeners: ```toml forwarded_headers = "both" # both (default) | x_forwarded | rfc7239 | none ``` | Mode | Sōzu adds | Client-supplied headers | | --- | --- | --- | | `both` (default) | `X-Forwarded-For`, `-Proto`, `-Port`, `Forwarded` | same as before, byte for byte: XFF and `Forwarded` chains extended, client `-Proto`/`-Port` kept | | `x_forwarded` | `X-Forwarded-For`, `-Proto`, `-Port` | same handling as `both` for that family; `Forwarded` passes through | | `rfc7239` | `Forwarded` only | `Forwarded` extended (RFC 7239 §4); `X-Forwarded-For/-Proto/-Port/-Host` **removed** | | `none` | nothing | everything passes through untouched (documented trust caveat) | ## Design - **Wire and config**: `enum ForwardedHeaders { BOTH = 0; X_FORWARDED; RFC7239; NONE }` (`command/src/command.proto:824`). `BOTH = 0` means an absent field, or a state file written before this change, keeps today's behaviour. The field is `optional ForwardedHeaders forwarded_headers` on `HttpListenerConfig` (36), `HttpsListenerConfig` (49), `UpdateHttpListenerConfig` (42) and `UpdateHttpsListenerConfig` (43). The TOML-side `ForwardedHeadersMode` (`command/src/config.rs:1773`) is `snake_case` and uses the `MetricDetailLevel` ↔ `MetricDetail` pattern, because `build.rs` puts `SCREAMING_SNAKE_CASE` on every prost enum. - **Hot update**: it follows the `send_x_real_ip` path through `L7ListenerHandler::get_forwarded_headers` (`lib/src/lib.rs:661`, `lib/src/http.rs:975`, `lib/src/https.rs:1423`) and `update_config`, and is captured per stream in `Context::create_stream` (`lib/src/protocol/mux/mod.rs:1206`), like `strict_sni_binding`. An H2 stream picks up a patch at its next HEADERS, an H1 connection when it opens (`mux/LIFECYCLE.md` §2.5). Unlike `send_x_real_ip`/`elide_x_real_ip`, which `ConfigState::update_http(s)_listener` does not apply (see Residual risks), the main-process state records the patched mode. The state (`command/src/state.rs:874,999`) and the worker (`lib/src/http.rs:1460`, `lib/src/https.rs:1744`) both run `validate_forwarded_headers` (`command/src/state.rs:3226`) and refuse an unknown wire value. - **Editor** (`lib/src/protocol/kawa_h1/editor.rs`, shared by H1 and H2 frontends): `emits_x_forwarded` / `emits_rfc7239` / `strips_x_forwarded` (L240–L254) only gate *emission*: the chain extensions (L1324), the synthesis (L1343, L1395) and the `rfc7239` elision branch (L1147). `ForwardingHop` stays mode-agnostic and renders every value once per connection. That keeps `send_x_real_ip` working in every mode, keeps the `renders()` cache and `reset()` invariants unchanged, and keeps the allocation-count tests for the default path unchanged. `X-Real-IP` is injected at the same position as before, so `both` stays byte-exact (`header_editing_output_is_byte_exact_across_keep_alive_requests` is unchanged and green). A debug postcondition (L1478) checks that `rfc7239` forwards no managed `X-Forwarded-*` header. `xff_chain` (access log) is still recorded in every mode. The `http.trusting.x_{proto,port}` counters count only in `both` and `x_forwarded`. - **`X-Forwarded-Host` on `rewrite_host`** (`lib/src/protocol/mux/router.rs:1539`): injected, replacing a client one, only in `both` and `x_forwarded`. In `rfc7239` and `none` nothing is injected, and under `none` a client `X-Forwarded-Host` passes through as sent. Sōzu's `Forwarded` element keeps its current parameters (`proto`, `for`, `by`); adding RFC 7239 §5.3 `host=` would change the wire format for every rewrite, so it is left as a possible follow-up. As a result the pre-rewrite authority is not conveyed in `rfc7239`, which the docs state. - **`X-Real-IP` independence**: `elide_x_real_ip` / `send_x_real_ip` are untouched by the mode. `X-Real-IP` belongs to neither family, and `send_x_real_ip = true` still injects it under `none` and `rfc7239`. Unit and e2e tests assert this. - **H2 trailers**: no change. `pkawa::handle_trailer` already drops `X-Forwarded-For`, `Forwarded` and `X-Real-IP` from trailer blocks in every mode. - **CLI**: `sozu listener {http,https} {add,update} --forwarded-headers <both|x_forwarded|rfc7239|none>` (clap `ValueEnum` with the TOML spellings). `listener list` shows the mode, and the patch audit diff renders it by name. ## What "strict" means for `rfc7239` In `rfc7239` mode, client-supplied `X-Forwarded-For`, `X-Forwarded-Proto`, `X-Forwarded-Port` and `X-Forwarded-Host` (the four Sōzu itself manages) are **elided**. They are neither passed through nor converted. The reasons: 1. **RFC 7239 §8.1** (Header Validity and Integrity): the header "cannot be relied upon to be correct, as it may be modified, whether mistakenly or for malicious reasons, by every node on the way to the server, including the client making the request". Its caveat that "the chain of IP addresses listed before the request came to the proxy cannot be trusted" applies equally to `X-Forwarded-*`. 2. **Pass-through would be weaker than today.** In this mode Sōzu no longer appends its hop to `X-Forwarded-For`, so a backend still reading the rightmost `X-Forwarded-For` would trust a client-forged value with nothing Sōzu wrote after it. Elision leaves the backend one forwarding family, whose last element is Sōzu's. 3. **No §7.4 conversion.** RFC 7239 §7.4 (Transition) only *encourages* converting `X-Forwarded-*` into `Forwarded` "if it can be done in a sensible way". It warns that when several `X-Forwarded-*` fields coexist "it may not be possible to do a conversion, since it is not possible to know in which order the already existing fields were added". Converted elements would also be exactly as untrusted as the originals. 4. **RFC 7239 §4** lets a proxy remove forwarding fields ("A proxy MAY remove all "Forwarded" header fields from a request"). Removing the non-standard family in a mode that promises the standard one is within that intent. The client `Forwarded` chain itself is preserved and extended, following §4 ("append it to the last existing "Forwarded" header field after a comma separator") and §7.2 ("A directly forwarded request should preserve and possibly extend it"). 5. **RFC 7239 §7.1** (HTTP Lists): a list "may be split over multiple header fields", so several `Forwarded` lines form one list. Appending Sōzu's element to the *last* client `Forwarded` line therefore extends the whole list, which is why the editor keeps only the last `Forwarded` header it sees. The same list semantics are why `rfc7239` elides *every* client `X-Forwarded-For` line, not only the last (the editor unit test sends two). Other `X-Forwarded-*` names (for example `X-Forwarded-Prefix`) are not attestations Sōzu manages and pass through in every mode. `none` is the RFC 7239 §8.2 (Information Leak) option of "preventing the internal nodes from updating the HTTP header field", for operators who do not want Sōzu to disclose the peer or its own address. In `x_forwarded` mode a client `Forwarded` passes through untouched. The owner's decision was that this mode keeps exactly the current handling of the family it emits. The docs carry the same trust caveat for it as for `none`. ## Review follow-ups (folded into this commit) 1. **Malformed client `Forwarded`: MEDIUM, fixed.** In `both` and `rfc7239`, a client line such as `Forwarded: for="6.6.6.6` (unclosed quoted-string) was extended into `for="6.6.6.6, proto=http;for="10.0.0.1:54321";by=127.0.0.1`. That swallowed Sōzu's element, and I reproduced it in both modes, so the bug predates this PR in `both`. The client value is now checked against the RFC 7239 §4 grammar by `is_valid_forwarded` (`lib/src/protocol/kawa_h1/editor.rs`): a list of `;`-separated `token=value` pairs, where a value is a token or an RFC 9110 §5.6.4 quoted-string. Empty list elements and OWS around `,`/`;` are tolerated, as RFC 9110 §5.6.1 asks of recipients. A malformed line is elided, which RFC 7239 §4 permits. Sōzu's element then goes to an earlier well-formed line or into a synthesised `Forwarded`, so it is always parseable and last. - **Also applied in `both`:** it is the other mode that extends the chain and had the same corruption. Well-formed input is byte-identical; every existing byte-exact editor test is unchanged and green. - **Not applied in `x_forwarded` and `none`:** they do not touch `Forwarded`, so the line passes through as sent. - **Tests:** `a_malformed_client_forwarded_is_elided_where_the_chain_is_extended` (the probe input, plus a well-formed line followed by a malformed one) and `is_valid_forwarded_follows_the_rfc7239_grammar` (unit), and `test_forwarded_headers_rfc7239_malformed_client_forwarded_h1` (e2e). 2. **H2 trailers: LOW, fixed.** `pkawa::handle_trailer` now also drops `x-forwarded-proto`, `x-forwarded-port` and `x-forwarded-host`. The drop is unconditional, matching the existing semantics there: every header the initial-HEADERS pass rewrites is dropped from trailers regardless of listener flags, and in every mode each of these three is either synthesised (`both`/`x_forwarded`) or removed (`rfc7239`), so a trailer copy is a spoof vector in all of them. Test: `handle_trailer_drops_each_spoof_header_individually`, extended. 3. **Add-path validation: LOW, fixed.** `validate_forwarded_headers` now runs on `AddHttpListener`/`AddHttpsListener` in `ConfigState::add_http(s)_listener` (before anything is stored) and in the worker's `HttpListener::new` / `HttpsListener::try_new`. Tests: `add_listener_rejects_an_unknown_forwarded_headers_value` (state), and `a_listener_with_an_unknown_forwarded_headers_value_is_refused` (http.rs and https.rs). Follow-up RED, then GREEN: - Editor, against a stub validator returning `true` (the pre-fix behaviour): `a_malformed_client_forwarded_is_elided_where_the_chain_is_extended ... FAILED` with left `["for=\"6.6.6.6, proto=http;for=\"10.0.0.1:54321\";by=127.0.0.1"]`, right `["proto=http;for=\"10.0.0.1:54321\";by=127.0.0.1"]`; the grammar test also FAILED. With the fix: both `ok`. - e2e, with the validity check disabled: `test_forwarded_headers_rfc7239_malformed_client_forwarded_h1 ... FAILED`. Restored: `ok`. - Trailer, before extending the drop list: `trailer with only x-forwarded-proto should leave no surviving block ... FAILED`. After: `ok` (all 4 `handle_trailer` tests). - State add, before the fix: `called Result::unwrap_err() on an Ok value ... FAILED`. After: `ok`. - Worker add, before the fix: both `a_listener_with_an_unknown_forwarded_headers_value_is_refused` FAILED (`an unknown mode must be refused`). After: `ok`. ## Tests (seen RED, then GREEN) RED runs were produced by removing the behaviour under test (for the editor, before the gating existed; otherwise by temporarily reverting the named line), never by a compile error. Editor, before gating (`both` passes from the start because it pins today's behaviour): ``` test ...editor::tests::forwarded_headers_survives_reset ... ok test ...editor::tests::forwarded_headers_x_forwarded_adds_only_the_x_forwarded_family ... FAILED test ...editor::tests::forwarded_headers_none_adds_nothing_and_passes_client_headers_through ... FAILED test ...editor::tests::forwarded_headers_rfc7239_adds_only_forwarded_and_strips_client_x_forwarded ... FAILED test ...editor::tests::forwarded_headers_both_adds_both_families_and_extends_client_chains ... ok test result: FAILED. 2 passed; 3 failed ``` After gating: `test result: ok. 5 passed; 0 failed`. Router, with `synthesises_xfh = rewriting_host` alone: `a_host_rewrite_discloses_x_forwarded_host_only_in_x_forwarded_modes ... FAILED` (left: `[b"rewrite.example.com"]`, right: `[]`). Restored: `ok`. Config, with `to_http`/`to_tls` not setting the field: `forwarded_headers_parses_every_mode_defaults_to_both_and_rejects_others ... FAILED` (left `(None, None)`, right `(Some(0), Some(0))`). Restored: `ok`. The test also covers every spelling, the absent key (`both`) and serde's `unknown variant` error for `RFC7239`, `x-forwarded`, `forwarded` and `""`. State, without the patch arm: `update_listener_forwarded_headers_applies_and_rejects_unknown_values ... FAILED` (left `Both`, right `Rfc7239`). Restored: `ok`. CLI, without `#[value(name = "x_forwarded")]`: `listener_https_update_parses_forwarded_headers ... FAILED`. Restored: `ok`. e2e, without `http_context.forwarded_headers = listener.get_forwarded_headers()` in `Context::create_stream`: ``` test tests::tests::test_forwarded_headers_none_h1 ... FAILED test tests::tests::test_forwarded_headers_rfc7239_h1 ... FAILED test tests::listener_update_tests::test_forwarded_headers_hot_change ... FAILED test tests::protocol_pair_matrix::forwarded_headers_rfc7239::test_h1_h1 ... ok test tests::protocol_pair_matrix::forwarded_headers_rfc7239::test_h2_h2 ... FAILED test tests::protocol_pair_matrix::forwarded_headers_rfc7239::test_h1_h2 ... FAILED test tests::protocol_pair_matrix::forwarded_headers_rfc7239::test_h2_h1 ... ok test result: FAILED. 2 passed; 5 failed ``` The `*_h1` backend cells pass here because, as for the existing X-Real-IP matrix feature, `AsyncBackend` exposes no request bytes; those cells only check forwarding. The H2 frontend is proved by `h2_h2`. Without the worker's `update_config` arm, `test_forwarded_headers_hot_change` fails with `patch_ok=true before_ok=true after_ok=false`. Restored: `test result: ok. 7 passed; 0 failed`. ## Validation Run on head `1076a246`, rebased onto `origin/main` `d9ce2a02` (after #1686): - `cargo build --all-features`: exit 0 - `cargo clippy --all-features --all-targets -- -D warnings`: exit 0 - `cargo +nightly fmt --all -- --check`: exit 0 - `RUSTDOCFLAGS="-D warnings" cargo doc --no-deps --all-features --locked`: exit 0 - `cargo test -p sozu-lib`: exit 0 (`1368 passed; 0 failed`) - `cargo test -p sozu`: exit 0 (`122 passed`) - `cargo test -p sozu-command-lib`: exit 0 (`202 passed`) - e2e groups, each exit 0: `forward` (23), `x_real_ip` (9), `listener_update` (15), `listener` (28), `protocol_pair_matrix` (28), `trailer` (5) - `python3 .github/scripts/check_doc_citations.py` `--self-test`, `--root .`, `--base origin/main` and `--audit --root .`: all exit 0 - the full e2e suite (`508 passed; 0 failed`) was run on the previous head `a339a456`, before #1686 Rebase onto #1686 (listener `interface`): - `command.proto`: #1686 took field numbers 35 (`HttpListenerConfig`) and 48 (`HttpsListenerConfig`) for `interface`, the numbers this PR had used. `interface` keeps them, and `forwarded_headers` moves to 36 and 49. The Update messages keep 42 and 43. No release has shipped the old numbers. - `state.rs`: `add_http(s)_listener` keeps #1686's `validated_listener_key` and runs `validate_forwarded_headers` before it. - `cli.rs` / `request_builder.rs`: both new CLI tests are kept, and the `add` builders chain `.with_interface(interface)` after the optional `with_forwarded_headers`. - `CHANGELOG.md`: both new entries are kept. - `doc/configure_admin_ops.md`: kept this PR's symbol citation in place of the renumbered `https.rs:515`. - e2e: the new H1 helper gains `interface: None` in its `ActivateListener`, as #1686 did for the sibling helpers. ## Docs `doc/configure.md` (new "Forwarding headers" section with the security note and trust caveat, an updatable-fields row, and the `send_proxy` paragraph), `doc/configure_admin_ops.md` (patch boundary list), `doc/lifetime_of_a_session.md`, `lib/src/protocol/kawa_h1/LIFECYCLE.md` §2 and §4, `lib/src/protocol/mux/LIFECYCLE.md` §2.5 capture table, `bin/config.toml` (both listener examples), `e2e/COVERAGE.md`, and `CHANGELOG.md` (Unreleased). ## Residual risks - In `rfc7239` and `none`, a `rewrite_host` conveys no pre-rewrite authority. The RFC 7239 §5.3 `host=` parameter is the natural follow-up. - The trust caveat for `none` (and for `Forwarded` under `x_forwarded`) is documented, not enforced. - Existing gap, not changed here: H1 chunked trailers have no spoof-header filter at all (`X-Real-IP`, `X-Forwarded-For` and `Forwarded` included); only H2 trailers are filtered. - Existing gap, not changed here: `ConfigState::update_http(s)_listener` does not apply `elide_x_real_ip` / `send_x_real_ip` patches, so after a hot update the main-process state (`listener list`, `SaveState`, upgrade replay) still shows the old values for those two knobs. `forwarded_headers` is applied there. Closes #322 Signed-off-by: Florentin Dubois <florentin.dubois@clever.cloud>

Commit:d9ce2a0
Author:Florentin DUBOIS
Committer:GitHub

feat(listener): bind listeners to a network interface (#1686) ## What A listener can be bound to a network interface by name, for an interface whose address is not known ahead or not stable (a WireGuard `wg0`, the case in #719): ```toml [[listeners]] protocol = "https" address = "0.0.0.0:443" interface = "wg0" ``` - New optional `interface` on `HttpListenerConfig`, `HttpsListenerConfig`, `TcpListenerConfig`, `UdpListenerConfig` (`command/src/command.proto:530`, `:681`, `:715`, `:739`), on `ActivateListener` / `DeactivateListener` / `RemoveListener` (`:829`, `:839`, `:848`), on the TOML `ListenerBuilder`, and `--interface` on `sozu listener <protocol> add|remove|activate|deactivate`. - Applied with socket2 `Socket::bind_device(Some(name))` **before** `bind(2)` by `bind_to_device` (`lib/src/socket.rs:1634`), called from `server_bind` (`:1667`, HTTP/HTTPS/TCP) and `udp_bind` (`:1749`). Setting it before `bind` is what lets two listeners share an address under `SO_REUSEADDR`/`SO_REUSEPORT`. - **UDP included**: `SO_BINDTODEVICE` is a `SOL_SOCKET` option, so `udp_bind` takes it exactly like `server_bind`. Replies to clients leave through that same listener socket. The per-flow upstream sockets (`udp_connect`) talk to backends, not to clients, so they are left alone: binding them would tie backend traffic to the frontend interface. - **Not in `Update*ListenerConfig`**: changing the interface needs a rebind (remove + add). ## Design: the interface is part of the listener identity `ListenerKey { address, interface: Option<String> }` (`command/src/listener_key.rs:120`) replaces the bare `SocketAddr` wherever a listener is stored or looked up: | Surface | Change | |---|---| | `ConfigState::{http,https,tcp,udp}_listeners` | `BTreeMap<ListenerKey, _>` (`command/src/state.rs:206`); add, remove, activate, deactivate, diff and `generate_*_requests` carry the interface | | `ActivateListener` / `DeactivateListener` / `RemoveListener` | optional `interface` selector; `listener_key()` on each message and config (`command/src/listener_key.rs`) | | SCM hand-off (`Listeners`, `ListenersCount` manifest) | `Vec<(ListenerKey, RawFd)>` (`command/src/scm_socket.rs:501`), manifest entries parsed as keys (`:622`) | | Worker proxies | `inherited_socket_fate`, `activate_listener`, `listener_token`, `give_back_listener(s)`, `remove_listener` match on the key (`HttpListener::is` and siblings in `lib/src/{http,https,tcp,udp}.rs`); `Server` builds the key from the request (`lib/src/server.rs:2706`, `:3142`, `:3398`) | | Main rollback of a failed listener add | `compute_rollback` carries the interface (`bin/src/command/requests.rs:2445`) | | `ListenersList` / `sozu listener list` / `sozu top` / audit targets | keyed and shown as the key text; `interface` column/row in the tables | **Backward compatibility.** The key's text form is the bare `SocketAddr` text for a listener without interface and `<address>%<interface>` otherwise (`FromStr`/`Serialize` at `command/src/listener_key.rs:163`, `:184`). `%` is refused in interface names (`validate_interface_name`, `:63`) and a `SocketAddr` only prints `%` inside IPv6 brackets, so parsing is unambiguous. Consequences: - `ConfigState` JSON crossing a **main-process upgrade** (`UpgradeData`) keeps identical **map keys** for every existing listener (the listener values now carry `"interface": null`, which an older Sōzu ignores as an unknown field); an older state without `interface` fields deserializes to the same state. - **Saved state files** (`WorkerRequest` JSON) without `interface` load unchanged (`Option` defaults to `None`). - The **SCM manifest** from an older Sōzu carries bare addresses, which parse as keys without interface. - A listener without `interface` behaves exactly as before. **Frontends and certificates stay keyed by address** and apply to **every** listener of their protocol on that address (`listeners_at` in `lib/src/http.rs:1134`, `lib/src/https.rs:2338`, `lib/src/tcp.rs:2905`; `listener_tokens_at` in `lib/src/udp.rs:763`). This keeps `RequestHttpFrontend`, `RequestTcpFrontend`, `RequestUdpFrontend`, `AddCertificate` & co. untouched. - **A listener added later converges with its siblings.** Each proxy's `add_listener` copies the routes of a listener already on the address when it creates the new one: `Router` + tags for HTTP, `Router` + tags + a deep copy of the `CertificateResolver` into the new listener's own resolver (the one its rustls config already points at) + an HSTS-inheritance refresh for HTTPS, cluster/SNI routes + tags for TCP, cluster + tags + flow-manager `SetCluster` for UDP (`adopt_sibling_route`). This covers `sozu listener <protocol> add --interface` and a configuration reload, which both send only the new listener's add/activate (`reload_adding_an_interface_listener_sends_only_that_listener`). I chose the worker-side copy over a main-side replay of the address's frontends/certificates: frontend and certificate requests name an address, so a replay would reach the existing siblings too, which already hold those routes (duplicate adds would fail the scatter and trigger the main's rollback). Replaying would need an interface selector on every frontend/certificate message, which is the per-interface-routing change this PR keeps out of scope. `Router`, `TrieNode` and `CertificateResolver` gain `#[derive(Clone)]` for this. - **Removals reach every listener.** Frontend removal (HTTP, HTTPS) and certificate removal/replacement now apply to every listener on the address even when one fails, and return the first failure in token order (`listeners_at` sorts by token), so no sibling keeps a route or certificate the others dropped. TCP/UDP removals were already infallible per listener. - In the configuration file, listeners sharing an address must use the same protocol (`ConfigError::ListenerProtocolMismatchOnAddress`), because a file frontend finds its listener's protocol from its address. At runtime, frontend requests are typed per protocol, so I do not enforce this in `ConfigState`: an HTTP listener on `wg0` and a TCP listener on the bare address are unambiguous and useful. **Per-interface routing (different frontends per interface on one address) is out of scope**: it would need an interface on every frontend/certificate message, state key and CLI verb. **Update patches** still name their listener by address only. When exactly one listener sits on that address (with or without interface) the patch applies to it; when several do, the main process refuses it with `StateError::AmbiguousListenerAddress` (`command/src/state.rs:139`, `listener_key_for_update` at `:154`) instead of patching an arbitrary one. If you prefer an optional key-only `interface` selector on the four `Update*ListenerConfig` messages, that is a small follow-up; I kept them untouched per the "not part of the Update messages" decision. ## Mixed versions The main process tracks no worker version (`WorkerInfo` carries none), so it cannot refuse an interface listener while an older worker runs. An older worker ignores the `interface` field and would bind on every interface, and matches Activate/Deactivate/Remove by address only. `doc/configure.md` and the CHANGELOG say: upgrade the main process and every worker before using `interface`; downgrading while a listener names an interface is unsupported (an older Sōzu cannot parse `<address>%<interface>` keys in a saved state or an upgrade hand-off). ## Errors and platform - Invalid names (empty, > 15 bytes, `.`/`..`, `/`, `:`, `%`, whitespace, NUL) are refused at config load (`ListenerBuilder::validated_interface`, `command/src/config.rs:944`) and at state dispatch (`validated_listener_key`, `command/src/state.rs:144`). - **Non-Linux refuses the key** with `InterfaceError::Unsupported` ("… SO_BINDTODEVICE, which only Linux provides") at config load and dispatch (`validate_listener_interface`, `command/src/listener_key.rs:90`); `bind_to_device` also returns `BindToDeviceUnsupported` off Linux. - A failed `SO_BINDTODEVICE` returns `ServerBindError::BindToDevice`: `could not bind the listener <addr> to the network interface '<iface>' (SO_BINDTODEVICE): <os error>. The interface must exist when the listener is activated, and on Linux kernels before 5.7 binding a socket to an interface requires the CAP_NET_RAW capability`. - **systemd**: `os-build/systemd/sozu.service` and `sozu@.service` set no `User=`, `CapabilityBoundingSet=` nor `AmbientCapabilities=`, so Sōzu runs as root with its full capability set and `SO_BINDTODEVICE` works on every kernel: no functional change. Both units gain a comment saying a drop-in that restricts capabilities on a kernel before 5.7 must keep `CAP_NET_RAW`; `doc/configure.md` says the same. - The binding survives upgrades: it belongs to the socket, and the socket crosses SCM under its key. ## Tests (each seen red, then green) | Test | RED (before the fix, or with the fix mutated out) | GREEN | |---|---|---| | `state::tests::listeners_on_one_address_with_different_interfaces_coexist` (`command/src/state.rs:6978`) | `an HTTP listener on another interface must not collide: StateError(HttpListener with id_bytes=12 already exists; …)` | ok | | `state::tests::listener_diff_distinguishes_interfaces` (`:7123`) | `left: [None] right: [Some("eth0")]` | ok | | `state::tests::update_listener_is_refused_when_the_address_is_ambiguous` (`:7162`) | `no entry found for key` | ok | | `config::tests::listener_interface_round_trips_from_the_config_file` (`command/src/config.rs:5359`), `invalid_listener_interface_rejected_at_config_load` (`:5470`) | `unknown field \`interface\`, expected one of \`address\`, …` | ok | | `socket::tests::bind_sets_so_bindtodevice_to_the_listener_interface` (`lib/src/socket.rs:2172`): binds TCP and UDP to `lo`, reads back `Socket::device()`; skips with a printed reason only on `EPERM` (it did not skip here) | `the TCP listener must be bound to lo — left: None` | ok | | `socket::tests::bind_to_a_missing_interface_names_the_interface_and_the_capability` (`:2222`) | `binding to a missing interface must fail: ()` | ok | | `listener_key::tests::listener_interface_is_refused_without_bind_to_device` (platform check mutated out) + cfg-gated `listener_interface_is_refused_off_linux` (compiles only off Linux, not run here) | `called Result::unwrap_err() on an Ok value: ()` | ok | | `state_compat_v1_1_1::legacy_config_state_json_without_interface_deserializes` (bare-address key parsing mutated out), `legacy_listener_records_without_interface_load_as_bare_address_listeners` | `a pre-#719 ConfigState must deserialize: Error("invalid listener address '127.0.0.1:8080'")` | ok | | `tcp::sni_routing_tests::listeners_sharing_an_address_on_different_interfaces_coexist` (`lib/src/tcp.rs:4417`): two listeners, `lo` + none, in one worker; key lookups, frontend fan-out, `SO_BINDTODEVICE` on the handed-back sockets (fan-out mutated to first listener only) | `the frontend must reach the listener at Token(1) — left: None right: Some("cluster-a")` | ok | | `scm_socket::tests::send_and_receive_listeners_keyed_by_interface` (`command/src/scm_socket.rs:755`) | new behaviour (keys on the wire) | ok | | e2e `test_upgrade_interface_bound_listener` (`e2e/src/tests/tests.rs:1374`): `lo`-bound HTTP listener serves, worker upgrade, 8 fresh connections served (manifest mutated to drop the interface) | `connection 3 after the upgrade was not served` | ok ×3 | | `rollback_inverse_of_a_listener_add_carries_its_interface` (`bin/src/command/requests.rs:5288`) — found in review: the rollback dropped the interface | `left: None right: Some("wg0")` | ok | | `cli::tests::listener_commands_parse_the_interface_flag` (`bin/src/cli.rs:1985`) | new flag | ok | | `http::listener_sibling_tests::a_listener_added_next_to_a_sibling_takes_over_its_frontends` (review finding 1) | `the listener added after the frontend must route it too` | ok | | `http::listener_sibling_tests::removing_a_frontend_reaches_every_listener_and_reports_a_failure` (review finding 1, removal) | `the failure on one listener must not spare its sibling's route` (HashMap order, not every run) | ok | | `https::listener_sibling_tests::a_listener_added_next_to_a_sibling_takes_over_its_certificates_and_frontends` | `the listener added after the certificate must serve it too` | ok | | `tcp::sni_routing_tests::a_listener_added_next_to_a_sibling_takes_over_its_routes` (no-SNI cluster and SNI routes) | `the listener added after the frontend must route to its cluster — left: None` | ok | | `udp::listener_sibling_tests::a_listener_added_next_to_a_sibling_takes_over_its_cluster` | `left: None right: Some("cluster-a")` | ok | | e2e `test_listener_added_on_an_interface_serves_existing_frontends`: bare `0.0.0.0:P` routes a frontend, then a `lo`-bound `0.0.0.0:P` is added + activated at runtime (what CLI add and reload send); 4 requests through `127.0.0.1` (route takeover mutated out) | `request 0 through lo was not routed: Some("HTTP/1.1 404 Not Found …")`, the reviewer's outage | ok ×3 | | `tcp::sni_routing_tests::traffic_through_an_interface_reaches_the_listener_bound_to_it` (review finding 3): bare and `lo`-bound `0.0.0.0:P`, 8 connections through `127.0.0.1`, all accepted by the `lo` listener's token, none by the bare one (`SO_BINDTODEVICE` mutated out) | `the lo listener must accept every connection, got 4 of 8: WouldBlock` (3/3 runs red) | ok | | `state::tests::reload_adding_an_interface_listener_sends_only_that_listener` | characterization of the reload diff (passes before and after; it is why the fix lives in the worker) | ok | ## Validation (head after the review fixes) - `Verified: cargo build --all-features -> exit 0` - `Verified: cargo clippy --all-features --all-targets -- -D warnings -> exit 0` - `Verified: cargo +nightly fmt --all -- --check -> exit 0` - `Verified: cargo test -p sozu-lib -> 1367 passed, 0 failed` (all test binaries) - `Verified: cargo test -p sozu -> 242 passed, 0 failed` (both test binaries) - `Verified: cargo test -p sozu-command-lib -> 210 passed, 0 failed` - `Verified: cargo test -p sozu-e2e -- listener upgrade tcp_tests udp_tests tls_tests https hsts_tests command_channel_security -> 78 passed, 0 failed` (the full e2e suite, 495 passed, ran on the previous head) - `Verified: python3 .github/scripts/check_doc_citations.py --self-test | --root . | --base origin/main | --audit --root . -> exit 0 each` File:line references above point at the initial head; the symbols named next to them are the stable reference. ## Residual risks - Per-interface routing is not supported (see design); certificate listings show one entry per listener, so an address shared by two listeners appears twice. - HTTPS HSTS: frontends copied from a sibling that inherited the sibling's listener-default HSTS are refreshed to the new listener's default; a frontend added while the sibling had no default is copied as explicit, so a new listener whose default differs from its sibling's does not apply its default to those copies. - Cluster-level answer templates (`AddCluster.answers`) are applied to the listeners that exist when the cluster is added; this PR does not change that pre-existing behaviour for later listeners. - With the fan-out, `TcpProxy::fronts` (cluster → listener token) records the last listener on the address only. It is write-only bookkeeping today (no reader in the proxy), so nothing changes; a future reader would need a set of tokens. - Downgrading while a listener has an `interface` is unsupported (documented): an older Sōzu fails to parse that listener's key in the state / SCM manifest; listeners without interface are unaffected. - Mixed versions: see above; not enforced by the main process. - The non-Linux refusal is proven through the platform-parameterized validator on Linux; the `cfg(not(target_os = "linux"))` test and the non-Linux branch of `bind_to_device` were not compiled here (no non-Linux target installed). Closes #719 Signed-off-by: Florentin Dubois <florentin.dubois@clever.cloud>

Commit:274268a
Author:Florentin DUBOIS
Committer:GitHub

feat(server): add a per-(cluster, source-subnet) connection limit (#1538) ## The bypass The per-`(cluster, source-IP)` limiter has no concept of a subnet. Someone on an IPv6 `/64` — the standard residential allocation — bypasses it **entirely** by taking a fresh address for every connection: the per-address counter never reaches 2. Reported by @jpds in #1270 from reading the release notes and the limiter code. ## The shape: two independent limiters Per @FlorentinDUBOIS in the issue: > I think we need both as the current rate limit is for handling rate limit by > ip as an api gateway, and the subnet aware will be there for ddos purposes. So this adds a **second, independent counter** rather than changing the first. `cluster_ip_at_limit`, `track_cluster_ip`, `effective_max_connections_per_ip` and both per-IP maps are untouched — same fields, same semantics, same bodies. What did change around them: the two call sites now go through a combined gate, and `untrack_all_cluster_ip` also drains the subnet slots (it is the single release point all three protocols already call on close, and a second entry point would be one more thing each could forget). Both gates are consulted on every admission and **both must admit**, which is what makes "10 per IP **and** 100 per /64" expressible: ```toml max_connections_per_ip = 10 # existing, untouched max_connections_per_subnet = 100 # new subnet_ipv4_prefix = 24 subnet_ipv6_prefix = 56 ``` The conjunction lives in exactly one place, `SessionManager::cluster_connection_at_limit`, so the H1+H2 mux and raw TCP cannot drift on what "admitted" means. ## The defaults leave current behaviour intact `max_connections_per_subnet` defaults to `0` (disabled) and the prefixes default to `/32` and `/128`, which mask nothing. An upgrade is inert until an operator opts in. `/24` and `/56` are **documented recommendations**, not defaults — matching what the reporter did in the equivalent Caddy module (default left at single IPs, those prefixes highly recommended in the docs). They are written up in `doc/configure.md` and `doc/rate-limit-design.md` §7.1.1. Stronger than inert: while the cap is `0` the subnet bookkeeping is **not written at all**, so a deployment that never enables the feature allocates nothing for it. `an_unset_subnet_cap_leaves_the_admission_path_untouched` asserts both maps stay empty. This is a **deliberate divergence from the per-IP twin**, which tracks unconditionally — flagging it explicitly as the point where you may want to decide otherwise. The cost is that enabling the cap at runtime does not retroactively count sessions already in flight; the counter converges as they close. ## New configuration surface Global TOML keys (`command/src/config.rs`): | Key | Type | Default | |---|---|---| | `max_connections_per_subnet` | integer, 0 = unlimited | `0` | | `subnet_ipv4_prefix` | integer 0-32 | `32` | | `subnet_ipv6_prefix` | integer 0-128 | `128` | Plus a per-cluster `max_connections_per_subnet` override with the same `None` inherits / `Some(0)` explicit-unlimited / `Some(n)` overrides semantics as its per-IP twin. Out-of-range prefixes are **rejected** at config-load time (`ConfigError::InvalidSubnetPrefix`), never silently clamped. ### Additions to `command.proto`, named explicitly | Message / field | Tag | |---|---| | `Request.set_max_connections_per_subnet` (uint64) | 60 | | `Request.query_max_connections_per_subnet` | 61 | | `message QueryMaxConnectionsPerSubnet {}` | — | | `message MaxConnectionsPerSubnetLimit { limit, ipv4_prefix, ipv6_prefix }` | — | | `ResponseContent.max_connections_per_subnet_limit` | 18 | | `Cluster.max_connections_per_subnet` (optional uint64) | 17 | | `ServerConfig.max_connections_per_subnet` | 26 | | `ServerConfig.subnet_ipv4_prefix` | 27 | | `ServerConfig.subnet_ipv6_prefix` | 28 | `command/src/proto/command.rs` is generated and was not edited. ### CLI New `sozu subnet-connection-limit {set|remove|show}`, mirroring `sozu connection-limit`. `show` reports the two prefixes alongside the limit: they are boot-time only, so that reply is the only runtime surface that can tell an operator what the live cap is a cap *on*. Like its twin the setter is non-sticky — workers reset to the TOML value on restart. The prefixes have no runtime setter on purpose: the mask is part of the counter key, so patching it live would re-key every counter in flight. ## Masking `Ipv4Addr` / `Ipv6Addr` have no standard masking method and no dependency was added. Both shift-overflowing bounds are branched explicitly — `u128::MAX << 128` panics under `debug_assertions` and wraps silently in release, so a full-width prefix must never reach the shift; a zero prefix is branched from the other end. **IPv4-mapped IPv6 sources are canonicalised first.** A dual-stack `[::]` listener reports an IPv4 client as `::ffff:a.b.c.d`; masking that with an IPv6 prefix would collapse the entire IPv4 internet into one `::/56` bucket, so it is converted to IPv4 and charged to `subnet_ipv4_prefix`. The per-IP counter is deliberately left un-canonicalised — part of what keeps its behaviour untouched. ## To SEE THIS RED Delete the `|| self.cluster_subnet_at_limit(...)` disjunct from `SessionManager::cluster_connection_at_limit` in `lib/src/server.rs`, leaving every other piece — config keys, proto fields, masking, tracking — in place: ``` cargo test -p sozu-lib --locked --no-default-features \ --features crypto-ring,opentelemetry,splice,simd subnet ``` ``` test ...::a_subnet_cap_refuses_a_fresh_address_from_an_already_counted_subnet ... FAILED panicked at lib/src/protocol/mux/router.rs: a fresh address from an already-counted /24 must be refused — an Admitted verdict here is the #1270 bypass: a new address per connection defeats the cap ``` The test drives the real `plan_connect` → `consult_ip_gate` → `plan_connect_resume` sequence for four **distinct** addresses in one `/24`, under a per-IP cap of 10 that no single address comes near — so only the subnet counter can refuse the fourth. A fifth address in a *different* `/24` is asserted admitted, so the refusal is the subnet key and not a blanket cap on new addresses. Restoring the disjunct returns it to green; the cycle was run twice. ## Tests added - `a_subnet_cap_refuses_a_fresh_address_from_an_already_counted_subnet` — the regression above. - `an_unset_subnet_cap_leaves_the_admission_path_untouched` — the default changes nothing, and allocates nothing. - `a_cluster_override_dropping_to_zero_after_a_track_is_a_no_op` — a cluster override can drop to `Some(0)` after a session was tracked under a positive limit (`AddCluster` is an upsert and clears no counters; only the *global* `SetMaxConnectionsPerSubnet(0)` drains). An earlier revision asserted that could not happen and panicked on it. - `masking_handles_both_shift_overflowing_prefix_bounds` — prefixes 0, full width, and over-wide. - `masking_groups_addresses_at_the_recommended_prefixes` — /24 and /56, including the issue's own `/64`-holder scenario. - `an_ipv4_mapped_source_is_charged_to_the_ipv4_prefix` — including the negative case where two mapped `/24`s must not collide. - `subnet_connection_limit_keys_round_trip_from_toml` and `an_out_of_range_subnet_prefix_is_rejected_not_clamped` — the config plumbing and its validation. ## Known limitation A subnet refusal currently surfaces as `BackendConnectionError::TooManyConnectionsPerIp` and increments the shared `connections.rejected_per_cluster_ip` counter, so an operator cannot tell the two caps apart in metrics. Kept shared to hold this changeset to the feature; a distinct error variant and counter are a reasonable follow-up. ## Validation All run locally at the tip of this branch, rebased onto `main` at `42ed1111` (after #1537 and #1540 landed); CI is the verdict. The rebase over #1537 conflicted only on line citations, in `doc/testing.md` and `lib/src/protocol/mux/LIFECYCLE.md` (13 hunks). Each was resolved by locating the text the citation named at the merge base inside the merged tree, never by arithmetic; the resulting diff on those two files is citations only — every removed line has an identical added counterpart once the digits are stripped, so none of #1537's prose was altered. | Command | Result | |---|---| | `cargo build --locked --no-default-features --features crypto-ring,opentelemetry,splice,simd` | pass | | `cargo +nightly fmt --all -- --check` | pass | | `cargo clippy --all-targets --locked --no-default-features --features crypto-ring,opentelemetry,splice,simd -- -D warnings` | pass | | `cargo test -p sozu-lib …` | 1147 passed, 0 failed | | `cargo test -p sozu …` | pass | | `cargo test -p sozu-command-lib --locked --no-default-features` | 171 passed, 0 failed | | `cargo test -p sozu-e2e … -- --skip tests::fuzz_tests::` | 449 passed, 0 failed | | `RUSTFLAGS="--cfg tokio_unstable" cargo test -p sozu-sim --locked` | pass | | `RUSTFLAGS="--cfg tokio_unstable" cargo clippy -p sozu-sim --all-targets --locked -- -D warnings` | pass | | `check_doc_citations.py --self-test / --root . / --base / --audit --root .` | all pass | | `RUSTDOCFLAGS="-D warnings" cargo doc --no-deps --all-features --locked` | pass | > **Note on the two config tests — resolved since this PR was opened.** > `subnet_connection_limit_keys_round_trip_from_toml` and > `an_out_of_range_subnet_prefix_is_rejected_not_clamped` live in > `command/src/config.rs`. When this PR was first pushed, CI ran `cargo test` for > `sozu-lib`, `sozu`, `sozu-e2e` and `sozu-sim` but **never** for > `sozu-command-lib`, and there was no workspace-wide `cargo test` anywhere — so > these two would have landed without ever running. That gap was reported as > [#1539](https://github.com/sozu-proxy/sozu/issues/1539), and > [#1540](https://github.com/sozu-proxy/sozu/pull/1540) landed the step that shut it; > it is now in `main`. > This branch is rebased on top of it, so CI does exercise both tests. Docs updated in the same changeset: `doc/configure.md`, `doc/rate-limit-design.md` (§3.1, §3.2, §3.3, §3.7, §3.8, §7.1.1), `bin/config.toml`, `CHANGELOG.md`. Line citations displaced by the insertions were renumbered and each renumbered bound verified against the text it named at the base. > **On the renumbering, for `check_doc_citations.py`'s own record.** The first > automated pass produced **three wrong ranges** on lines whose text is ambiguous > (a bare `}`), two of them inverted. The checker caught two. The third — > `lib/src/server.rs:1060-1068`, which should have been `1056-1064` — **passed > silently, because a citation this changeset re-anchored is exempt from the > base comparison and is never checked at all** (the run reports this honestly: > "39 re-anchored spans were exempt from the comparison"). It was only found by > auditing all 29 renumbered bounds content-against-content, resolving each one > through the text it named at the base. Another agent hit the same mechanism on > a different changeset today, so this is an independent confirmation that the > re-anchoring exemption is a real hole, not a theoretical one: the rule cannot > guard precisely the citations a changeset touches, which are the ones most > likely to be wrong. Closes #1270 Signed-off-by: Florentin Dubois <florentin.dubois@clever.cloud>

Commit:eac0409
Author:Florentin DUBOIS
Committer:GitHub

fix(parser): match the HTTP method token case-sensitively (#1474) `Method::new` compared with `compare_no_case`, so a request line reading `get /path HTTP/1.1` produced `Method::Get`. RFC 9110 §9.1 makes the method token case-sensitive and the origin receives the original `get` bytes, so sozu and the backend could disagree about what method a request carries. Long harmless, made load-bearing by the stale-upstream replay (#1442): `is_idempotent` gates whether a request already written to a pooled upstream may be replayed on a fresh one, and `Method::Custom` is deliberately not idempotent. A lowercase `get` was therefore judged replayable while the origin was free to treat the token as an extension method with side effects of its own. The narrow scoping is not available: `context.method` is already normalised by the time the replay decision runs, and the raw token is gone. The match is now exact against the eight canonical spellings; anything else is `Method::Custom`, carrying the token verbatim. No case-insensitive fallback and no alias table: either would reintroduce the same divergence in a form that is harder to see. Metric cardinality was measured before changing, not after. No metric key or label is derived from the method: the `http.status.*` keys come only from `context.status: u16` through a closed eighteen-code list plus buckets, and the only labels the emission sites pass are `cluster_id` and `backend_id`, both operator-declared. A peer cannot mint label values by varying case, before or after this change. Routing cuts both ways, because `MethodRule::new` classifies a frontend's declared `method` through the same `Method::new`. A frontend declared without a `method` is unaffected. A frontend declared `method = "GET"` no longer matches a client sending `get`; symmetrically one declared `method = "get"` stops answering canonical `GET` requests, so a method declared in non-canonical case must be corrected to the case the client sends. Response framing changes in the direction worth having. `Method::Head` is what terminates the response parse after the headers, and `head` no longer reaches that arm. Before, an RFC-conforming origin answering `405` with a body left that body unread on a pooled keep-alive upstream, desyncing the next request on it — the common pairing. After, the cost lands only on the rare non-conforming pairing, a case-folding origin answering `head` bodiless with a `Content-Length`, where sozu serves 504 at `configured_backend_timeout`: a bounded stall, not a hang. Two smaller consequences: the access log echoes the token verbatim, as it already did for every custom method, and `Method`'s `Debug` now renders a lowercase verb as `Custom(bytes=N)` at the three sites that print `context.method` with `{method:?}`. `doc/configure.md` documented the old lenient behaviour explicitly, in both the routing-rule and replay-eligibility sections, and is updated alongside `mux/LIFECYCLE.md`, the `RequestHttpFrontend.method` proto field and the `--method` CLI help. Closes #1451. Signed-off-by: Florentin Dubois <florentin.dubois@clever.cloud>

Commit:16e26b7
Author:Florentin Dubois
Committer:Florentin Dubois

fix(parser): match the HTTP method token case-sensitively `Method::new` compared with `compare_no_case`, so a request line reading `get /path HTTP/1.1` produced `Method::Get`. RFC 9110 §9.1 makes the method token case-sensitive and the origin receives the original `get` bytes, so sozu and the backend could disagree about what method a request carries. Long harmless, made load-bearing by the stale-upstream replay (#1442): `is_idempotent` gates whether a request already written to a pooled upstream may be replayed on a fresh one, and `Method::Custom` is deliberately not idempotent. A lowercase `get` was therefore judged replayable while the origin was free to treat the token as an extension method with side effects of its own. The narrow scoping is not available: `context.method` is already normalised by the time the replay decision runs, and the raw token is gone. The match is now exact against the eight canonical spellings; anything else is `Method::Custom`, carrying the token verbatim. No case-insensitive fallback and no alias table: either would reintroduce the same divergence in a form that is harder to see. Metric cardinality was measured before changing, not after. No metric key or label is derived from the method: the `http.status.*` keys come only from `context.status: u16` through a closed eighteen-code list plus buckets, and the only labels the emission sites pass are `cluster_id` and `backend_id`, both operator-declared. A peer cannot mint label values by varying case, before or after this change. Routing cuts both ways, because `MethodRule::new` classifies a frontend's declared `method` through the same `Method::new`. A frontend declared without a `method` is unaffected. A frontend declared `method = "GET"` no longer matches a client sending `get`; symmetrically one declared `method = "get"` stops answering canonical `GET` requests, so a method declared in non-canonical case must be corrected to the case the client sends. Response framing changes in the direction worth having. `Method::Head` is what terminates the response parse after the headers, and `head` no longer reaches that arm. Before, an RFC-conforming origin answering `405` with a body left that body unread on a pooled keep-alive upstream, desyncing the next request on it — the common pairing. After, the cost lands only on the rare non-conforming pairing, a case-folding origin answering `head` bodiless with a `Content-Length`, where sozu serves 504 at `configured_backend_timeout`: a bounded stall, not a hang. Two smaller consequences: the access log echoes the token verbatim, as it already did for every custom method, and `Method`'s `Debug` now renders a lowercase verb as `Custom(bytes=N)` at the three sites that print `context.method` with `{method:?}`. `doc/configure.md` documented the old lenient behaviour explicitly, in both the routing-rule and replay-eligibility sections, and is updated alongside `mux/LIFECYCLE.md`, the `RequestHttpFrontend.method` proto field and the `--method` CLI help. Closes #1451. Signed-off-by: Florentin Dubois <florentin.dubois@clever.cloud>

Commit:32b8e48
Author:Florentin Dubois
Committer:Florentin Dubois

fix(command): resolve `certificate list --domain` through the SNI trie `sozu certificate list --domain foo.example.com` returned nothing when the certificate actually serving that host was registered as `*.example.com`, while TLS handshakes for it kept succeeding. The inventory contradicted the running proxy exactly when an operator was debugging a certificate problem and trusting the query, and the wildcard case is the common one. It fails closed rather than open, so this is a diagnostic defect, not a security one. The main process answered the filter from `ConfigState::get_certificates`, which compares the SAN list with `Vec::contains` — exact string equality — whereas the certificate that serves a handshake is chosen by `CertificateResolver::domain_lookup`, a `TrieNode` lookup resolving `*.` wildcard and, since sozu#1378, regex labels. The two disagree for every certificate not bound by an exact name. `--domain` is now resolved through that same trie, rebuilt from the control plane's own record, in `certificates_serving_domain` (`bin/src/command/requests.rs`). It is fixed at the caller because `TrieNode` lives in `sozu-lib`, which already depends on `sozu-command-lib`: the reverse edge is a cargo cycle, moving the trie into `command/` would grow a `regex` production dependency that crate does not have, and reimplementing hostname matching there would duplicate it and drift from the resolver — which is this bug. `bin/` depends on both crates and holds the only `--domain` consumer of `get_certificates`. Three properties of the rebuilt index, each pinned by a test. One trie per listener address, unioned, because `ConfigState::certificates` is keyed by listener and `TrieNode::insert` answers `InsertResult::Existing` and keeps the incumbent, so a single global trie would silently drop a second listener's certificate for the same name. Both sides are ASCII-lowercased, which is the sole derivation difference between the two name sets — `add_certificate` resolves the SAN/CN set exactly as `CertifiedKeyWrapper::try_from` does, but only the resolver folds case. And a name longer than `MAX_HOSTNAME_LENGTH`, which the resolver refuses outright, is skipped rather than indexed. This changes the shape of the answer: a trie lookup resolves to one entry, so `--domain` returns at most one certificate per HTTPS listener where the exact-equality filter could return several. The main process does not retain `AddCertificate::expired_at`, so it cannot reproduce the resolver's longest-lived tie-break and reports a reproducible choice instead; `--workers` reports what each worker's resolver actually selected. `doc/configure_cli.md` gains the section stating which question `--domain` answers — the documentation said nothing at all before — and the CLI help text and the `QueryCertificatesFilters.domain` proto comment now state it too. One divergence is deliberately left in place and commented where it lives: `lib/src/server.rs` intercepts a `QueryCertificatesFromWorkers` request whose filter carries a fingerprint and hands it to `get_certificates`, whose domain arm is tested first, so `--fingerprint <hex> --domain <host> --workers` still takes the exact-SAN arm. The two filters were never a conjunction on any path. Closes #1383 Repairs the citations this changeset moved. The added imports, `certificates_serving_domain` and the test module shift every `bin/src/command/requests.rs` line citation below them; twenty citation endpoints across twelve groups in five files drifted, of which the CI resolver reported one, because it fails only on a blank landing and every other one had drifted onto a different non-blank line. Each is re-anchored on the symbol the surrounding prose already names — `Server::handle_client_request`, `Config::command_allowed_uids`, `audit_record_to_json`, `validate_frontend_request`, `should_rollback_fanout`, `load_state`, and the `RequestType::Update*Listener` variants — rather than on a fresh line number. A symbol survives every edit above it, and these citations had already rotted once without anyone noticing: `requests.rs:1071`, cited three times in `listener_update_tests.rs` as the merge-only `LoadState` path, named a `PeerCredUnavailable` doc comment before this branch existed. No range is kept where the enclosing item was already cited in the same sentence, and the import list in `doc/configure_admin_ops.md` is replaced by the three `RequestType` variants it was standing in for. Signed-off-by: Florentin Dubois <florentin.dubois@clever.cloud>

Commit:9c8feed
Author:kannar

feat(ctl): filter `frontend list` by cluster id `sozu frontend list` could only be narrowed by protocol and by domain, so answering "which frontends belong to this cluster, and what access-log tags do they carry?" meant dumping every frontend and filtering client-side. `FrontendFilters` gains an optional `cluster_id` (proto tag `5`), wired to `sozu frontend list -i/--cluster-id <id>`. `ConfigState::list_frontends` applies it to HTTP, HTTPS, TCP and UDP frontends, ANDed with the existing protocol and `--domain` filters. TCP and UDP frontends are keyed by cluster id in the state, so the filter is applied to the map key; HTTP/HTTPS frontends carry an optional cluster id, and a frontend that denies traffic (no cluster id) therefore never matches. Covered by `list_frontends_filters_by_cluster_id`, which asserts both the positive and the negative space: the wanted frontend of every protocol is listed, no other cluster's frontend leaks through, an unknown id matches nothing, a `Deny` frontend is never selected, and `cluster_id` composes with (rather than replaces) the protocol and domain filters. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: kannar <kannarfr@gmail.com>

Commit:2ed35c8
Author:Florentin Dubois
Committer:Florentin DUBOIS

feat(tcp): SNI+ALPN ClientHello preread passthrough routing (#1279) Route TCP connections by the SNI (and offered ALPN) read from the TLS ClientHello without terminating TLS: buffer and parse the cleartext ClientHello on an SNI-enabled TCP listener, select a backend cluster by exact/wildcard SNI + client-preference-order ALPN, and forward the original encrypted bytes byte-for-byte, preserving end-to-end TLS and client-cert mTLS (e.g. many Kubernetes API servers multiplexed behind one :443). Architecture mirrors the UDP sans-io work: a pure ClientHello preread core (no I/O, injected time/config, invariant sweeps, no panic on adversarial input) in lib/src/protocol/tcp_preread/, driven by an impure mio shell; a deterministic simulation in sozu-sim, a cargo-fuzz target, and focused e2e. Config: TCP frontends gain optional `sni`/`alpn`; TCP listeners gain `sni_preread_timeout` (default 5s) and `sni_preread_max_bytes` (default 16 KiB, clamped to buffer_size, floored at the 5-byte TLS record header). SNI-enabled listeners reject non-TLS/no-SNI/malformed first bytes; non-SNI TCP listeners are byte-identical to before. Validation is enforced both at config load and at the worker request boundary (dynamic AddTcpFrontend / state replay). Also fixes a general Pipe close-before-flush data-truncation shared by H1/H2/WebSocket/TCP: on frontend EOF with bytes still queued for the backend, Pipe::readable returned Close unconditionally and dropped them, silently truncating any upload racing a frontend close. It now transitions to the half-closed WriteOpen state, arms the backend writable, and defers teardown to check_connections so the queue drains first. Tests: tcp_preread unit + parser fuzz (fuzz_tcp_clienthello) + deterministic simulation (tcp_preread_sim) + SNI/ALPN/mTLS/reject-matrix/PROXY/splice/ counter-conservation e2e. All current TCP e2e unchanged and green; full e2e suite (all protocols) green. CI wired for the new fuzz and simulation targets. Docs: doc/configure.md, doc/testing.md, CHANGELOG.md, and a new lib/src/protocol/tcp_preread/LIFECYCLE.md. Signed-off-by: Florentin Dubois <florentin.dubois@clever.cloud>

Commit:9114ca2
Author:Florentin Dubois

feat(tcp): SNI+ALPN ClientHello preread passthrough routing (#1279) Route TCP connections by the SNI (and offered ALPN) read from the TLS ClientHello without terminating TLS: buffer and parse the cleartext ClientHello on an SNI-enabled TCP listener, select a backend cluster by exact/wildcard SNI + client-preference-order ALPN, and forward the original encrypted bytes byte-for-byte, preserving end-to-end TLS and client-cert mTLS (e.g. many Kubernetes API servers multiplexed behind one :443). Architecture mirrors the UDP sans-io work: a pure ClientHello preread core (no I/O, injected time/config, invariant sweeps, no panic on adversarial input) in lib/src/protocol/tcp_preread/, driven by an impure mio shell; a deterministic simulation in sozu-sim, a cargo-fuzz target, and focused e2e. Config: TCP frontends gain optional `sni`/`alpn`; TCP listeners gain `sni_preread_timeout` (default 5s) and `sni_preread_max_bytes` (default 16 KiB, clamped to buffer_size, floored at the 5-byte TLS record header). SNI-enabled listeners reject non-TLS/no-SNI/malformed first bytes; non-SNI TCP listeners are byte-identical to before. Validation is enforced both at config load and at the worker request boundary (dynamic AddTcpFrontend / state replay). Also fixes a general Pipe close-before-flush data-truncation shared by H1/H2/WebSocket/TCP: on frontend EOF with bytes still queued for the backend, Pipe::readable returned Close unconditionally and dropped them, silently truncating any upload racing a frontend close. It now transitions to the half-closed WriteOpen state, arms the backend writable, and defers teardown to check_connections so the queue drains first. Tests: tcp_preread unit + parser fuzz (fuzz_tcp_clienthello) + deterministic simulation (tcp_preread_sim) + SNI/ALPN/mTLS/reject-matrix/PROXY/splice/ counter-conservation e2e. All current TCP e2e unchanged and green; full e2e suite (all protocols) green. CI wired for the new fuzz and simulation targets. Docs: doc/configure.md, doc/testing.md, CHANGELOG.md, and a new lib/src/protocol/tcp_preread/LIFECYCLE.md. Signed-off-by: Florentin Dubois <florentin.dubois@clever.cloud>

Commit:a923c00
Author:Florentin Dubois
Committer:Florentin DUBOIS

feat(udp): add first-class UDP load balancing (#1273) Add a `udp` listener type alongside tcp/http/https so Sōzu can load-balance datagram services (DNS, syslog, NTP, generic UDP) without a second load balancer in front of it, reusing the control plane, metrics/logging and e2e harness. Architecture: a two-level sans-io core (lib/src/protocol/udp/: UdpManager admission/flow-table/generation-token timer-wheel over per-flow UdpFlow, pure and property-tested) driven by an I/O shell (lib/src/udp.rs) on the existing single-threaded mio loop. Static per-listener worker ownership; scale across cores by running multiple listeners. Flows reset on hot-upgrade (listener fd handed off via SCM, flow state not migrated). - Control plane: UdpListenerConfig + RequestUdpFrontend proto, TOML config parsing, ConfigState with diff/generate_requests/activate replay symmetry, SCM listener-fd hand-off, ctl CLI + master-process routing, and sozu top/listener-list/frontend-list/cluster display parity. - Datapath: virtual 4-tuple flows, three-knob teardown (responses / idle timeout / requests), symmetric NAT return via a per-flow connected upstream socket, bounded per-flow/per-listener write-queue egress backpressure with writable re-arm, max_flows fd-pressure shedding. No panic on network input (EMFILE/ENFILE/oversize/invalid -> drop + metric). - Load balancing: round-robin + HRW/rendezvous (default) + Maglev (opt-in), selectable per cluster via a key: Option<u64> threaded through the LB trait; the Maglev table is rebuilt only on backend-set change, never per datagram. - PROXY protocol v2 to the backend (first-datagram default, per-cluster every-datagram opt-in; DGRAM transport byte 0x12/0x22). - Active health checks bound to the endpoint: companion TCP probe (primary) + app-level UDP probe (secondary), rise/fall hysteresis, fail-open. - Metrics (udp.*) with no gauge underflow on any teardown path, and per-flow access logs under the UDP / UDP-FLOW log tag. - Tests: sans-io quickcheck/state-machine core tests, 14 e2e scenarios (RR/HRW/Maglev affinity, NAT return, teardown, truncation, PPv2, health/fail-open, hot reconfig, upgrade fd hand-off) with a mock UDP backend/client, and a fuzz_udp_flow cargo-fuzz target. - Docs: doc/configure.md UDP listener/cluster/health section + metrics, CHANGELOG, README, architecture. Plaintext only. Single-listener multi-core / eBPF SK_REUSEPORT / userland dispatcher / DSR / io_uring / DTLS / QUIC / UDP-over-HTTP / flow-state migration across upgrade are explicit non-goals. Signed-off-by: Florentin Dubois <florentin.dubois@clever.cloud>

Commit:6d79213
Author:Florentin Dubois
Committer:Florentin DUBOIS

fix(mux): mitigate the HTTP/2 bomb (HPACK header-field cap + window-stall reaping) Closes both halves of the HTTP/2 memory-amplification DoS class (calif.io disclosure; same family as Apache CVE-2026-49975). HPACK header bomb ----------------- Thousands of 1-byte HPACK indexed references — or a single `cookie` header split into many crumbs (RFC 9113 §8.2.3) — each materialize a `Pair` of per-entry bookkeeping, amplifying wire bytes into allocation. `SETTINGS_MAX_HEADER_LIST_SIZE` was accounted as name+value bytes only, omitting the RFC 9113 §6.5.2 mandated 32-octet per-field overhead, and nothing capped the field count. - `pkawa.rs`: account `+32` octets/field per §6.5.2 and enforce a per-block materialized-field-count cap in `decode_headers_with_budget` and `handle_trailer`; cookie crumbs count individually. Over-limit blocks are rejected with `RST_STREAM` / `GOAWAY(ENHANCE_YOUR_CALM)`. - New per-listener `h2_max_header_fields` knob (default 128), plumbed through `H2FloodConfig`, `command.proto`, `config.rs`, the http/https flood-config getters, live-update patch-apply + state replay + validation, the sozuctl CLI, and listener `Display`. Window-stall ------------ A peer that holds its receive window shut while a response is buffered can pin the stream and its `MAX_CONCURRENT_STREAMS` slot. The bidirectional liveness timer (`stream_last_activity_at`) is refreshed by any inbound frame, so a 1-byte inbound DATA drip kept a stalled stream warm indefinitely. - New per-stream flow-control-stall deadline (`stream_fc_stalled_since`): armed only when a stream holds buffered response data it cannot send because its effective send window `min(stream.window, connection.window)` is exhausted, cleared on real outbound progress (`consumed > 0`), and never refreshed by inbound DATA/HEADERS — so the inbound-drip vector cannot keep it warm. Reaped by `cancel_timed_out_streams` (governed by `h2_stream_idle_timeout_seconds`) via a deduped union with the idle guard (`collect_timed_out_streams`). - The reaper runs from both `readable()` and `MuxState::timeout`, so even a fully-silent peer's stalled stream is reaped (slot freed + queued `RST_STREAM(CANCEL)` flushed). Direction-correct: a slow upload has no buffered response, a socket-bound slow link keeps a positive window, and a legitimate slow download clears the deadline on every window grant — none arm it. No new config knob (reuses `h2_stream_idle_timeout_seconds`). Tests: `pkawa` unit tests (`+32` accounting, field cap, cookie-crumb cap, legit request) and `collect_timed_out_streams` unit tests; e2e `e2e_h2_parser_reject_header_field_bomb`, `test_h2_window_stall_response_reaped`, `test_h2_window_stall_inbound_drip_reaped`, `test_h2_window_stall_silent_reaped`, with `test_h2_active_upload_survives_idle_timeout` as a false-positive guard. Validated: `cargo build --all-features`, `clippy -D warnings`, nightly fmt, the full h2 e2e suite (222), lib + command unit tests, and the cargo-fuzz hpack / frame targets. Docs: `doc/configure.md`, `CHANGELOG.md`, mux `LIFECYCLE.md`. Signed-off-by: Florentin Dubois <florentin.dubois@clever.cloud>

Commit:f73efb7
Author:Florentin Dubois

feat(udp): add first-class UDP load balancing (#1273) Add a `udp` listener type alongside tcp/http/https so Sōzu can load-balance datagram services (DNS, syslog, NTP, generic UDP) without a second load balancer in front of it, reusing the control plane, metrics/logging and e2e harness. Architecture: a two-level sans-io core (lib/src/protocol/udp/: UdpManager admission/flow-table/generation-token timer-wheel over per-flow UdpFlow, pure and property-tested) driven by an I/O shell (lib/src/udp.rs) on the existing single-threaded mio loop. Static per-listener worker ownership; scale across cores by running multiple listeners. Flows reset on hot-upgrade (listener fd handed off via SCM, flow state not migrated). - Control plane: UdpListenerConfig + RequestUdpFrontend proto, TOML config parsing, ConfigState with diff/generate_requests/activate replay symmetry, SCM listener-fd hand-off, ctl CLI + master-process routing, and sozu top/listener-list/frontend-list/cluster display parity. - Datapath: virtual 4-tuple flows, three-knob teardown (responses / idle timeout / requests), symmetric NAT return via a per-flow connected upstream socket, bounded per-flow/per-listener write-queue egress backpressure with writable re-arm, max_flows fd-pressure shedding. No panic on network input (EMFILE/ENFILE/oversize/invalid -> drop + metric). - Load balancing: round-robin + HRW/rendezvous (default) + Maglev (opt-in), selectable per cluster via a key: Option<u64> threaded through the LB trait; the Maglev table is rebuilt only on backend-set change, never per datagram. - PROXY protocol v2 to the backend (first-datagram default, per-cluster every-datagram opt-in; DGRAM transport byte 0x12/0x22). - Active health checks bound to the endpoint: companion TCP probe (primary) + app-level UDP probe (secondary), rise/fall hysteresis, fail-open. - Metrics (udp.*) with no gauge underflow on any teardown path, and per-flow access logs under the UDP / UDP-FLOW log tag. - Tests: sans-io quickcheck/state-machine core tests, 14 e2e scenarios (RR/HRW/Maglev affinity, NAT return, teardown, truncation, PPv2, health/fail-open, hot reconfig, upgrade fd hand-off) with a mock UDP backend/client, and a fuzz_udp_flow cargo-fuzz target. - Docs: doc/configure.md UDP listener/cluster/health section + metrics, CHANGELOG, README, architecture. Plaintext only. Single-listener multi-core / eBPF SK_REUSEPORT / userland dispatcher / DSR / io_uring / DTLS / QUIC / UDP-over-HTTP / flow-state migration across upgrade are explicit non-goals. Signed-off-by: Florentin Dubois <florentin.dubois@clever.cloud>

Commit:f59de3e
Author:Florentin Dubois

fix(mux): mitigate the HTTP/2 bomb (HPACK header-field cap + window-stall reaping) Closes both halves of the HTTP/2 memory-amplification DoS class (calif.io disclosure; same family as Apache CVE-2026-49975). HPACK header bomb ----------------- Thousands of 1-byte HPACK indexed references — or a single `cookie` header split into many crumbs (RFC 9113 §8.2.3) — each materialize a `Pair` of per-entry bookkeeping, amplifying wire bytes into allocation. `SETTINGS_MAX_HEADER_LIST_SIZE` was accounted as name+value bytes only, omitting the RFC 9113 §6.5.2 mandated 32-octet per-field overhead, and nothing capped the field count. - `pkawa.rs`: account `+32` octets/field per §6.5.2 and enforce a per-block materialized-field-count cap in `decode_headers_with_budget` and `handle_trailer`; cookie crumbs count individually. Over-limit blocks are rejected with `RST_STREAM` / `GOAWAY(ENHANCE_YOUR_CALM)`. - New per-listener `h2_max_header_fields` knob (default 128), plumbed through `H2FloodConfig`, `command.proto`, `config.rs`, the http/https flood-config getters, live-update patch-apply + state replay + validation, the sozuctl CLI, and listener `Display`. Window-stall ------------ A peer that holds its receive window shut while a response is buffered can pin the stream and its `MAX_CONCURRENT_STREAMS` slot. The bidirectional liveness timer (`stream_last_activity_at`) is refreshed by any inbound frame, so a 1-byte inbound DATA drip kept a stalled stream warm indefinitely. - New per-stream flow-control-stall deadline (`stream_fc_stalled_since`): armed only when a stream holds buffered response data it cannot send because its effective send window `min(stream.window, connection.window)` is exhausted, cleared on real outbound progress (`consumed > 0`), and never refreshed by inbound DATA/HEADERS — so the inbound-drip vector cannot keep it warm. Reaped by `cancel_timed_out_streams` (governed by `h2_stream_idle_timeout_seconds`) via a deduped union with the idle guard (`collect_timed_out_streams`). - The reaper runs from both `readable()` and `MuxState::timeout`, so even a fully-silent peer's stalled stream is reaped (slot freed + queued `RST_STREAM(CANCEL)` flushed). Direction-correct: a slow upload has no buffered response, a socket-bound slow link keeps a positive window, and a legitimate slow download clears the deadline on every window grant — none arm it. No new config knob (reuses `h2_stream_idle_timeout_seconds`). Tests: `pkawa` unit tests (`+32` accounting, field cap, cookie-crumb cap, legit request) and `collect_timed_out_streams` unit tests; e2e `e2e_h2_parser_reject_header_field_bomb`, `test_h2_window_stall_response_reaped`, `test_h2_window_stall_inbound_drip_reaped`, `test_h2_window_stall_silent_reaped`, with `test_h2_active_upload_survives_idle_timeout` as a false-positive guard. Validated: `cargo build --all-features`, `clippy -D warnings`, nightly fmt, the full h2 e2e suite (222), lib + command unit tests, and the cargo-fuzz hpack / frame targets. Docs: `doc/configure.md`, `CHANGELOG.md`, mux `LIFECYCLE.md`. Signed-off-by: Florentin Dubois <florentin.dubois@clever.cloud>

Commit:1bd27bc
Author:miton18
Committer:Florentin DUBOIS

fix(otel): capture wall-clock start time for accurate span reconstruction Access-log consumers reconstructing OTel spans computed the request start timestamp as `precise_time - request_time`. This subtraction mixes two unsynchronised clock sources: - `precise_time` is a wall-clock reading (`CLOCK_REALTIME` via `OffsetDateTime::now_utc()`), captured at log-emission time. - `request_time` is a monotonic duration (`CLOCK_MONOTONIC` via `Instant::now() - start`), captured in `SessionMetrics` *before* the wall-clock reading. The two clocks drift independently (NTP adjustments affect CLOCK_REALTIME but not CLOCK_MONOTONIC), and they are sampled at different instants within the log-emission path. On short-lived requests the resulting error can exceed the request duration itself, producing spans whose computed start time falls after a subsequent request's start — visibly broken traces. This commit adds a `start_wall: Option<SystemTime>` field to `SessionMetrics`, captured at the same point as the existing monotonic `start: Option<Instant>`. A new `start_wall_ns()` accessor converts it to nanoseconds-since-epoch for the access log. Wire changes: - `ProtobufAccessLog.start_time` (field 30, optional `Uint128`) in `command.proto` — backwards-compatible: old consumers ignore it, new consumers with old producers see `None` and can fall back to the subtraction. - `RequestRecord.start_time_ns: Option<i128>` plumbed through all four access-log emission sites (kawa_h1, mux/stream, pipe, tcp). Consumers should now prefer `start_time` over `time - request_time` whenever the field is present. Signed-off-by: miton18 <remi@collignon-ducret.fr>

Commit:3f41a83
Author:Florentin Dubois
Committer:Florentin Dubois

refactor(command,server): remove the proto_version capability handshake Drop the proto_version field and the SetMetricDetail capability partition that gated the dispatch on worker proto version. Production deployments keep master + workers in sync via the existing UpgradeMain hot-upgrade flow, so the mixed-version-fleet state the field was designed for does not occur. The implementation was structurally broken anyway: the master stamped every WorkerInfo.proto_version from its own SOZU_PROTO_VERSION constant at fork time, so the field always read as the master's version and MIN_PROTO_VERSION_FOR_SET_METRIC_DETAIL was effectively unconditional. The proto contract is additive-only; a worker that does not recognise tag 55 (SetMetricDetail) returns WorkerResponse::error("unknown request type") which already surfaces in the standard fan-out error tally (extras.fanout.workers_err counter). MetricDetailStatus.unsupported_workers becomes redundant and is also removed. Deletions: - WorkerInfo.proto_version (tag 4) and MetricDetailStatus.unsupported_workers (tag 5); both replaced by `reserved` markers per project convention. - sozu_command_lib::SOZU_PROTO_VERSION constant. - WorkerSession.proto_version field. - MIN_PROTO_VERSION_FOR_SET_METRIC_DETAIL constant and the capability partition in set_metric_detail_request; SetMetricDetail now fans out unconditionally via the standard scatter path. - Capability-handshake paragraphs in CHANGELOG.md and doc/sozu-top.md. - print_metric_detail_status block that rendered the unsupported_workers table row. Old workers without tag 55 surface as 'succeeded with errors' (the normal fan-out failure shape) rather than as a dedicated capability-skip list. Operator visibility is preserved. Signed-off-by: Florentin Dubois <florentin.dubois@clever.cloud>

Commit:fe3ff05
Author:Florentin Dubois
Committer:Florentin Dubois

feat(command,lib): per-worker WorkerMetricDetailStatus payload Final piece of the PR #1256 follow-through. Previously the capability-aware dispatcher in `bin/src/command/requests.rs` (`SetMetricDetailTask::on_finish`) synthesised `MetricDetailStatus.workers[<worker_id>]` using the master's aggregator view as a stand-in for each worker because workers replied with `WorkerResponse::ok(message.id)` carrying no payload. Each worker holds an independent `Aggregator` with its own lease table, so that stand-in obscured real per-worker drift (different configured floors, different active lease counts after a partial fan-out). Wire it properly end-to-end: - New `ResponseContent::WorkerMetricDetailStatus` oneof variant (tag 17 — proto additive). Carries the worker's own `(configured, effective, previous_effective, active_lease_count)` quartet, semantically distinct from the aggregated `MetricDetailStatus` at tag 16. - New `lib/src/server.rs::worker_metric_detail_status_content` helper that builds the response payload from a `(configured, effective, previous_effective, lease_count)` snapshot captured BEFORE the `METRICS.borrow_mut` scope ends (so the per- request snapshot is consistent with the transition that just happened). - The three ok-paths in the worker's SetMetricDetail arm (clear-Cleared, clear-NotFound, apply-Applied) now reply via `WorkerResponse:: ok_with_content` with the freshly-built payload instead of the payload-less `ok`. The `clear-NotFound` path reports `previous_effective == effective` (no transition). - Master-side `SetMetricDetailTask::on_finish` collects the per-worker payload from `response.content` and only falls back to skipping the worker entry when the response has no payload (e.g. an older worker that never went through `ok_with_content`). Removes the master-view stand-in noted as a follow-up in commit `70cd24af` (`set_metric_detail_request`). - `command/src/proto/display.rs` adds a silent OK match arm for the new variant — the per-worker payload flows master-side and is never printed directly on the operator's terminal. Build/clippy clean; 1075/1075 workspace tests pass. Signed-off-by: Florentin Dubois <florentin.dubois@clever.cloud>

Commit:b24fba3
Author:Florentin Dubois
Committer:Florentin Dubois

feat(metrics,command): bind lease ownership to peer credentials Replace the unauthenticated `lease_clear(client_id)` API with a binding- aware variant so one same-UID operator cannot clear another operator's lease by guessing the `client_id` format. The binding pairs the master- side `actor_pid` (captured via `SO_PEERCRED` at command-socket accept) with the per-connection session ULID; both halves must match the apply- time binding for the worker to authorise the clear. Implementation: - New `PeerBinding { pid, session_ulid }` + `LeaseEntry` + `LeaseClearOutcome` in `lib/src/metrics/mod.rs`. `lease_apply` records the binding alongside `(level, expires_at)`; `lease_clear` returns `Cleared`/`NotFound`/ `Unauthorized` depending on the apply-time binding vs the presented one. - A "binding unknown" apply (pre-binding caller or platform without `SO_PEERCRED`) preserves backward compat: the worker accepts any clear. A fully-known apply rejects every clear whose presented binding does not match, including the default ("unknown") clear. - New proto fields `SetMetricDetail.peer_pid` (tag 6) and `SetMetricDetail.peer_session_ulid` (tag 7), additive. Clients leave them empty; the master populates them in `bin/src/command/requests.rs::worker_request` from the connecting `ClientSession` before fan-out. - Worker parses the presented ULID via `rusty_ulid::Ulid::from_str` with a `0x…` hex fallback, then routes the `LeaseClearOutcome::Unauthorized` outcome to `WorkerResponse::error` so the operator gets a loud failure rather than a silent no-op. - Six new unit tests cover authorised clear, unauthorised mismatch, unknown-apply accepts-any, known-apply rejects default clear, the existing apply/clear/tick paths through the new signature. Build/clippy clean; 26/26 metric tests pass. Signed-off-by: Florentin Dubois <florentin.dubois@clever.cloud>

Commit:8720d28
Author:Florentin Dubois
Committer:Florentin Dubois

feat(command): worker proto version handshake, scaffolding for unsupported_workers Wire the proto-capability layer that the `MetricDetailStatus.unsupported_workers` field needs to be usefully populated. Previously the field was declared but always empty because the master had no way to know which workers could decode the new SetMetricDetail (tag 55) verb. This commit lays the rails; the dispatch-time gating itself remains a follow-up because it requires a dedicated `WorkerTask` impl for SetMetricDetail that synthesises a MetricDetailStatus reply rather than the current generic worker_request flow. The scaffolding shipped here: - New `sozu_command_lib::SOZU_PROTO_VERSION = 1` constant, baked into every Sōzu binary at compile time. Bumped any time a new wire- affecting `RequestType` or proto field needs capability gating. Version 1 covers `SetMetricDetail`, `METRIC_DETAIL_CHANGED`, the worker→master audit IPC, and per-lease peer-credential binding. - New `WorkerInfo.proto_version` proto field (tag 4, optional uint32) so TUI / status consumers can observe each worker's version directly. - New `WorkerSession.proto_version` master-side field, snapshotted at fork time from the binary's `SOZU_PROTO_VERSION` constant. The `to_info()` / `list_workers` paths populate the new proto field. - Proto comment on `MetricDetailStatus.unsupported_workers` updated to describe the capability-gate model AND the inherited-after-UpgradeMain caveat: a re-exec master does NOT yet re-query inherited workers' versions, so they retain the new master's compile-time value until the planned per-worker capability handshake on Status reply lands. Build/clippy clean; 53/53 TUI unit tests pass. Signed-off-by: Florentin Dubois <florentin.dubois@clever.cloud>

Commit:65c9ed0
Author:Florentin Dubois
Committer:Florentin Dubois

feat(server,command): wire worker→master audit IPC for METRIC_DETAIL_CHANGED Previously, worker-local cardinality-lease transitions (TTL janitor expiry, worker-arm apply/clear) left no audit trail because the worker had no IPC path to the master's audit pipeline. Only operator-initiated transitions audited from `bin/src/command/requests.rs::worker_request` produced an audit row. A SOC analyst correlating "who elevated metrics cardinality" could see the apply but not the implicit clear. Close the gap by reusing the existing worker→master `Event` channel: - New proto `MetricDetailTransition` carrying `previous_effective`, `effective`, `transition_kind`, and an optional `client_id` for explicit apply/clear. Folded into `Event.metric_detail` (tag 5) so `EventKind::METRIC_DETAIL_CHANGED` events now carry their full payload through the existing fan-out plumbing. - Worker emits the event from three sites in `lib/src/server.rs::notify`: the polled `lease_tick` janitor (transition_kind = "lease_tick_expired" and `client_id = None` because the janitor may retire multiple leases at once), the SetMetricDetail worker arm on apply ("lease_apply"), and on clear ("lease_clear"). All three callers gate on `previous != effective` so the helper itself is a defence-in-depth no-op when nothing actually changed. - Master's `handle_worker_response` recognises METRIC_DETAIL_CHANGED events and routes them through a dedicated `audit_worker_metric_detail_transition` helper that writes to both audit sinks (text + JSON). The worker is its own actor — `worker_id` takes the `client_id` slot in the envelope; `actor_role=worker`, `actor_comm=sozu-worker`. Subscriber fan-out is unchanged. - The 11 existing `push_event(Event { … })` constructors get `metric_detail: None` to stay compatible with the new field; only the worker's lease-transition emitter populates it. Build/clippy clean; 53/53 TUI unit tests pass. Signed-off-by: Florentin Dubois <florentin.dubois@clever.cloud>

Commit:09de4ce
Author:Florentin Dubois
Committer:Florentin Dubois

style(top,proto): silence the last clippy and rustdoc warnings Clears the 14 warnings remaining after the simplify pass: - Five dead-code items: drop `render_placeholder` (all panes ship, no placeholder needed); drop `ClusterRow.errors_5xx_total` (the renderer reads `error_rate_pct`, never the raw count); drop `ThresholdTable.conn_warn_pct` (no consumer); drop `BackendRow.bytes_in` / `bytes_out` (the BACKENDS pane reads `back_bytes_in` / `back_bytes_out`). `Skin.categorical` gets a short doc + `#[allow(dead_code)]` because the field is read by tests and is the TOML surface for operator skins — the production consumer (cluster-row categorical tinting) lands in a follow-up. - Nine rustdoc warnings in the prost-generated `command.rs`: my earlier H2 commit's numbered-list comment in the `SetMetricDetail` preamble used 4-space indentation on the continuation lines, which prost-build expanded to overindented in the rendered docstring. Trim to 3 spaces so the rendered comment lands on the rustdoc list-item-alignment rule (3 spaces aligns with the text after `1. ` markers). - One rustdoc warning in `bin/src/ctl/top/mod.rs::run_top`: the doc comment had a `+` at line-start that markdown parsed as a list bullet, then complained the next two lines were unindented. Rephrase to join with "and" so the `+` no longer leads a line. Build is now warning-free under `cargo clippy --all-features --all-targets --locked`. Signed-off-by: Florentin Dubois <florentin.dubois@clever.cloud>

Commit:1c26057
Author:Florentin Dubois
Committer:Florentin Dubois

docs(proto,changelog): scope METRIC_DETAIL_CHANGED to operator-initiated transitions The proto documented "every effective-level transition emits an `EventKind::METRIC_DETAIL_CHANGED` event", but only operator-initiated transitions emit today: the master-side audit-log wiring covers the SetMetricDetail fan-out path, but worker-local transitions (lease expiry on the polled janitor, post-fan-out apply/clear in the worker arm) leave a silent gap. Replicating the audit emission inside the worker `notify` arm needs a new IPC back to the master and is deferred to a follow-up. Update the `SetMetricDetail` doc comment and the `EventKind::METRIC_DETAIL_CHANGED` declaration to scope the contract to operator-initiated transitions, and add a CHANGELOG caveat so audit-log consumers know which transitions are not yet surfaced. The three previous `// TODO(sozu-top week 2):` markers in the worker arm now point to the proto doc as the source of truth for the scope. Signed-off-by: Florentin Dubois <florentin.dubois@clever.cloud>

Commit:631d4a1
Author:Florentin Dubois
Committer:Florentin Dubois

feat(proto,command,server): SetMetricDetail TTL lease verb + plumbing Adds a runtime cardinality lease verb so `sozu top` can elevate the metrics drain to `MetricDetailLevel::Backend` for the duration of an interactive session. The lease design (TTL-bounded, `client_id`-keyed, self-expiring) is crash-safe and composes with multiple concurrent clients — see `Aggregator` lease bookkeeping in the previous commit. Proto (additive, backwards-compat): - `Request.request_type::SetMetricDetail = 55` carries `{ client_id, detail?, ttl_seconds?, clear?, reason? }`. - `ResponseContent::metric_detail_status = 16` returns `MetricDetailStatus { configured, effective, previous_effective, workers: map<id, WorkerMetricDetailStatus>, unsupported_workers[] }` for mixed-version-fleet safety. - `EventKind::METRIC_DETAIL_CHANGED = 30` on the `SubscribeEvents` audit stream; distinct from `METRICS_CONFIGURED` (Enabled/Disabled /Clear) since the cause is different. - `command/build.rs` re-attaches `Hash, Eq` for `MetricDetailStatus` (the embedded `map<string, WorkerMetricDetailStatus>` strips the prost auto-derive, which propagates to `ResponseContent.content_type` and `Request.request_type`). - `command/src/proto/display.rs` adds arms for `RequestType:: SetMetricDetail`, `ContentType::MetricDetailStatus` (with a prettytable renderer that lists per-worker configured/effective /previous_effective + unsupported workers), and `EventKind:: MetricDetailChanged`. - `command/src/request.rs` routes `SetMetricDetail` through the worker-level dispatch group (mirrors `ConfigureMetrics`). Master + worker plumbing: - `bin/src/command/requests.rs::is_mutating_verb` learns the new verb so the master brackets it with `RELOADING=1`/`READY=1` systemd hints. - The dispatch match routes through the existing `worker_request` fan-out path (same shape as `ConfigureMetrics`); per-worker `MetricDetailStatus` aggregation lands in week 2 when the TUI starts consuming it. - `lib/src/server.rs::notify` adds two hooks: a polled lease-expiry janitor at the top of every dispatch (gated by `lease_tick_due` so it only walks the lease table every 5 s), and the `SetMetricDetail` arm itself. The arm clamps the TTL via the `Aggregator` setter, decodes the `MetricDetail` enum defensively, and acks with `WorkerResponse:: ok()` for now (week 2 will return the per-worker `WorkerMetricDetailStatus` payload). `MetricDetailChanged` audit emission is left as a `TODO(sozu-top week 2)` at both `lease_apply`/`lease_clear` call sites and the janitor — plumbed through `bin/src/command/requests.rs::audit_emit_inline` once the master collects per-worker effective levels back. Signed-off-by: Florentin Dubois <florentin.dubois@clever.cloud>

Commit:afe7871
Author:Florentin Dubois
Committer:Florentin Dubois

fix(command): default missing repeated/map fields when loading older state files `bin/src/command/requests.rs::load_state` reads each `\n\0`-separated JSON record via `command::parser::parse_several_requests::<WorkerRequest>`, which calls `serde_json::from_slice` per record. The prost-build config in `command/build.rs` attaches `Serialize`/`Deserialize` derives to every generated message but did not attach `#[serde(default)]` anywhere, so missing `repeated`/`map` fields rejected the record (`Vec<T>` and `BTreeMap<K, V>` are required-by-serde without an explicit default). Post-1.1.1 schema additions (`Cluster.answers`, `Cluster.authorized_hashes`, `RequestHttpFrontend.headers`, plus the listener-level `answers` / `alpn_protocols` fields) therefore broke `LoadState` for any older client (e.g. proxy-manager pinned to `sozu-command-lib = "1.1.1"`): the first `AddCluster` or `AddHttpFrontend` failed to deserialize, `parse_several_requests`'s `many0(complete(...))` left the unparsed bytes as the remainder, and the read loop reported `"Error consuming load state message"` to the client at EOF. Each post-1.1.1 `repeated`/`map` field on a state-file-emittable message now carries `#[serde(default)]` via a `field_attribute(... )` line in `command/build.rs`, so missing fields default to empty (mirroring the protobuf wire-format default). Required scalars stay strict on purpose: `ConfigState::add_cluster` keys the cluster map by `cluster_id` without a non-empty check, so a struct-level blanket would silently insert a bogus `""`-keyed entry. The field-level annotation preserves that defense-in-depth. The new contract is documented at the top of `command/src/command.proto` and inline in `command/build.rs`: any new `repeated` or `map` field on a message reachable from a `SaveState`/`LoadState` JSON file (anything emitted by `ConfigState::generate_requests`, or carried in a `RequestType` an external client may build and feed through `LoadState`) must add the matching `field_attribute(...)` line. Regression tests in `command/tests/state_compat_v1_1_1.rs` pin both the 1.1.1-shaped fixture round-trip and the missing-required-scalar rejection contract. Signed-off-by: Florentin Dubois <florentin.dubois@clever-cloud.com> Signed-off-by: Florentin Dubois <florentin.dubois@clever.cloud>