Proto commits in openziti/ziti

These commits are when the Protocol Buffers files have changed: (only the last 100 relevant commits are shown)

2026-09-30

Reconcile the raft advertise address with the controller config on startup. Fixes #4487 - adds a startup reconcile that asks the leader to move this node's stored raft address to its configured advertise address, retrying with jittered backoff until the change is visible locally - wakes the running reconcile on leader events instead of starting a second one, and jitters the first attempt and each wake so controllers reconciling after the same event spread their requests - adds an UpdatePeerAddressRequest peer message carrying the member id, the address the node believes is stored and the new address; the leader accepts it only from the node it names and does not forward it - adds updateMemberAddressAsLeader, which changes a member's address in place, keeping its suffrage, after refusing non-members, addresses held by another member and stale from addresses - probes the new address as a sanity check, refusing the change if a different controller answers and making it with a warning if the address can't be reached, since some controllers only dial out - conditions the update on a configuration index read from the raft log and snapshot stores, since GetConfiguration().Index() is always zero in the pinned raft version - routes AddPeer requests that find an existing member at a new address to the in-place update instead of remove and re-add, identifying the member through GetPeerInfo rather than the request id - falls back to the member id when the raft transport dials a stored address with no peer connected at it - adds GetCachedServers to the FSM so that lookup works before raft has finished starting - adds a probe hello header so probe connections are closed without registering a peer - corrects the inverted voter flag in the peer AddPeer log and removes the unused ctrlAddress field - tests single-node, follower, non-voter and leader address changes, refused and unverified addresses, requests from the wrong sender, membership changes made while a member is offline or during the probe, and configuration lookup across snapshots and log gaps - adds a 2.0.8 changelog entry describing the address update and the leader version requirement

Reconcile the raft advertise address with the controller config on startup. Fixes #3588 - adds a startup reconcile that asks the leader to move this node's stored raft address to its configured advertise address, retrying with jittered backoff until the change is visible locally - wakes the running reconcile on leader events instead of starting a second one, and jitters the first attempt and each wake so controllers reconciling after the same event spread their requests - adds an UpdatePeerAddressRequest peer message carrying the member id, the address the node believes is stored and the new address; the leader accepts it only from the node it names and does not forward it - adds updateMemberAddressAsLeader, which changes a member's address in place, keeping its suffrage, after refusing non-members, addresses held by another member and stale from addresses - probes the new address as a sanity check, refusing the change if a different controller answers and making it with a warning if the address can't be reached, since some controllers only dial out - conditions the update on a configuration index read from the raft log and snapshot stores, since GetConfiguration().Index() is always zero in the pinned raft version - routes AddPeer requests that find an existing member at a new address to the in-place update instead of remove and re-add, identifying the member through GetPeerInfo rather than the request id - falls back to the member id when the raft transport dials a stored address with no peer connected at it - adds a probe hello header so probe connections are closed without registering a peer - excludes self by server id in GetPeerAddresses - corrects the inverted voter flag in the peer AddPeer log and removes the unused ctrlAddress field - tests single-node, follower, non-voter and leader address changes, refused and unverified addresses, requests from the wrong sender, membership changes made while a member is offline or during the probe, and configuration lookup across snapshots and log gaps - adds a changelog entry describing the address update and the leader version requirement

2026-09-29

Reconcile the raft advertise address with the controller config on startup. Fixes #3588 - adds a startup reconcile that asks the leader to move this node's stored raft address to its configured advertise address, retrying with jittered backoff until the change is visible locally - wakes the running reconcile on leader events instead of starting a second one, and jitters the first attempt and each wake so controllers reconciling after the same event spread their requests - adds an UpdatePeerAddressRequest peer message carrying the member id, the address the node believes is stored and the new address; the leader accepts it only from the node it names and does not forward it - adds updateMemberAddressAsLeader, which changes a member's address in place, keeping its suffrage, after refusing non-members, addresses held by another member and stale from addresses - probes the new address as a sanity check, refusing the change if a different controller answers and making it with a warning if the address can't be reached, since some controllers only dial out - conditions the update on a configuration index read from the raft log and snapshot stores, since GetConfiguration().Index() is always zero in the pinned raft version - routes AddPeer requests that find an existing member at a new address to the in-place update instead of remove and re-add, identifying the member through GetPeerInfo rather than the request id - falls back to the member id when the raft transport dials a stored address with no peer connected at it - adds a probe hello header so probe connections are closed without registering a peer - excludes self by server id in GetPeerAddresses - corrects the inverted voter flag in the peer AddPeer log and removes the unused ctrlAddress field - tests single-node, follower, non-voter and leader address changes, refused and unverified addresses, requests from the wrong sender, membership changes made while a member is offline or during the probe, and configuration lookup across snapshots and log gaps

2026-09-28

Merge pull request #4215 from openziti/router-link-runtime-config Apply managed link heartbeat and queue settings at runtime

2026-09-25

Reconcile the raft advertise address with the controller config on startup. Fixes #3588 - adds a startup reconcile that asks the leader to move this node's stored raft address to its configured advertise address, retrying with backoff until the change is visible locally - adds an UpdatePeerAddressRequest peer message carrying the member id, the address the node believes is stored and the new address, so address updates never carry suffrage or add members - adds UpdateMemberAddressAsLeader, which changes a member's address in place, keeping its suffrage, after refusing non-members, addresses held by another member, stale from addresses and new addresses that don't answer with the member's id - conditions the update on a configuration index read from the raft log and snapshot stores, since GetConfiguration().Index() is always zero in the pinned raft version - routes AddPeer requests for an existing member at a new address to the in-place update instead of remove and re-add - falls back to the member id when the raft transport dials a stored address with no peer connected at it - adds a probe hello header so probe connections are closed without registering a peer - excludes self by server id in GetPeerAddresses - shuts down raft before closing the db in Controller.Shutdown, since in cluster mode the db belongs to the raft FSM - corrects the inverted voter flag in the peer AddPeer log and removes the unused ctrlAddress field - tests single-node, follower, non-voter and leader address changes, refused changes, membership changes made while a member is offline or during the probe, and configuration lookup across snapshots and log gaps

Reconcile the raft advertise address with the controller config on startup. Fixes #3588 - adds a startup reconcile that asks the leader to move this node's stored raft address to its configured advertise address, retrying with backoff until the change is visible locally - adds an UpdatePeerAddressRequest peer message carrying the member id, the address the node believes is stored and the new address, so address updates never carry suffrage or add members - adds UpdateMemberAddressAsLeader, which changes a member's address in place, keeping its suffrage, after refusing non-members, addresses held by another member, stale from addresses and new addresses that don't answer with the member's id - conditions the update on a configuration index read from the raft log and snapshot stores, since GetConfiguration().Index() is always zero in the pinned raft version - routes AddPeer requests for an existing member at a new address to the in-place update instead of remove and re-add - falls back to the member id when the raft transport dials a stored address with no peer connected at it - adds a probe hello header so probe connections are closed without registering a peer - excludes self by server id in GetPeerAddresses - shuts down raft before closing the db in Controller.Shutdown, since in cluster mode the db belongs to the raft FSM - corrects the inverted voter flag in the peer AddPeer log and removes the unused ctrlAddress field - tests single-node, follower, non-voter and leader address changes, refused changes, membership changes made while a member is offline or during the probe, and configuration lookup across snapshots and log gaps

2026-09-24

Controller should clear links not in full link sync from router. Fixes #4458 - adds a fullRefresh flag to the router's reconnect link announcement - prunes links sourced from the announcing router that are not in a full refresh set - moves the reroute of a usable faulted link into LinkFaulted - adds LinkManager.IterateLinks - For #3493

Break the SDK terminator removal retry storm. Fixes #4467 Backport of #4159 to release-v2.0.x. Adapted for channel/v4 and sdk-golang v1, 2.0's int-typed capability constants and binding.AddTypedReceiveHandler registration. The ER/T tests that depend on the establish settle-gate fixtures (#4194, not on this branch) are omitted; their unit-level equivalents are kept. ctrl.pb.go is regenerated with protoc-gen-go v1.36.11, matching main and the google.golang.org/protobuf v1.36.12 runtime this branch already depends on; the committed file predated that bump and was still v1.31.0 output. - adds RemoveTerminatorsV2, an asynchronous control-channel removal, so a router no longer holds a request slot open waiting for a synchronous reply; gated on a new capability so routers fall back to the synchronous path against controllers that do not advertise it - paces establish and remove retries with a per-terminator exponential backoff after a controller rejection, so a rejected operation is not re-sent on the next event-loop tick, which is what let a rate-limited removal sustain itself - routes v2 removes to the leader and skips the ordered delete for a terminator the router confirms it created that no longer exists, so a storm of no-op retries cannot consume the command rate limiter - gates that skip on the leader having applied everything committed when it took leadership, since a leader reports itself elected before those applies finish and would otherwise read a committed create as absent and drop its delete - guards the leader term and every write to the caught-up flag with a dedicated lock, so a barrier completing alongside a leadership change cannot record a result for a term that has already ended - scopes v2 removals to the requesting router, so a router can only remove terminators it owns on the asynchronous path as well as the synchronous one - tracks in-flight removal batches on the router, resolving or requeueing them when the response arrives; lost responses self-heal via rate-limiter expiry and the retry scan - waits for an outstanding operation before sending a terminator delete, and checks that before dequeuing, so a delete neither races a create the controller has committed but not applied nor falls out of the queue until the retry scan finds it minutes later - matches a create response to the attempt that produced it, by control channel and reply sequence, since attempts overlap once one overruns the establishment timeout and every attempt carries the same terminator id; only the outstanding attempt's response may confirm the create or report the terminator idle - records the attempt as the sender assigns its sequence, so a response cannot be processed before the request it answers is known, and withdraws an attempt whose send never queued - reports a removal that succeeds past the establishment timeout as congestion rather than success, matching how the establish path classifies its own latency, since both resolve into the same limiter - extracts the transient-error test both remove handlers share, so the v2 path treats a leaderless cluster as busy like v1 does instead of as a permanent failure, and the two cannot drift again - applies the same delete sequencing and response correlation to the router-hosted (ER/T) registry, which cleared the same in-flight marker unguarded and dropped a deferred delete from its queue rather than leaving it there - bounds the v2 response send with a timeout, so a router that stops draining its send queue cannot accumulate blocked responder goroutines on the controller

Break the SDK terminator removal retry storm. Fixes #4156 - adds RemoveTerminatorsV2, an asynchronous control-channel removal, so a router no longer holds a request slot open waiting for a synchronous reply; gated on a new capability so routers fall back to the synchronous path against controllers that do not advertise it - paces establish and remove retries with a per-terminator exponential backoff after a controller rejection, so a rejected operation is not re-sent on the next event-loop tick, which is what let a rate-limited removal sustain itself - routes v2 removes to the leader and skips the ordered delete for a terminator the router confirms it created that no longer exists, so a storm of no-op retries cannot consume the command rate limiter - gates that skip on the leader having applied everything committed when it took leadership, since a leader reports itself elected before those applies finish and would otherwise read a committed create as absent and drop its delete - guards the leader term and every write to the caught-up flag with a dedicated lock, so a barrier completing alongside a leadership change cannot record a result for a term that has already ended - scopes v2 removals to the requesting router, so a router can only remove terminators it owns on the asynchronous path as well as the synchronous one - tracks in-flight removal batches on the router, resolving or requeueing them when the response arrives; lost responses self-heal via rate-limiter expiry and the retry scan - waits for an outstanding operation before sending a terminator delete, and checks that before dequeuing, so a delete neither races a create the controller has committed but not applied nor falls out of the queue until the retry scan finds it minutes later - matches a create response to the attempt that produced it, by control channel and reply sequence, since attempts overlap once one overruns the establishment timeout and every attempt carries the same terminator id; only the outstanding attempt's response may confirm the create or report the terminator idle - records the attempt as the sender assigns its sequence, so a response cannot be processed before the request it answers is known, and withdraws an attempt whose send never queued - reports a removal that succeeds past the establishment timeout as congestion rather than success, matching how the establish path classifies its own latency, since both resolve into the same limiter - extracts the transient-error test both remove handlers share, so the v2 path treats a leaderless cluster as busy like v1 does instead of as a permanent failure, and the two cannot drift again - applies the same delete sequencing and response correlation to the router-hosted (ER/T) registry, which cleared the same in-flight marker unguarded and dropped a deferred delete from its queue rather than leaving it there - bounds the v2 response send with a timeout, so a router that stops draining its send queue cannot accumulate blocked responder goroutines on the controller

Break the SDK terminator removal retry storm. Fixes #4156 - adds RemoveTerminatorsV2, an asynchronous control-channel removal, so a router no longer holds a request slot open waiting for a synchronous reply; gated on a new capability so routers fall back to the synchronous path against controllers that do not advertise it - paces establish and remove retries with a per-terminator exponential backoff after a controller rejection, so a rejected operation is not re-sent on the next event-loop tick, which is what let a rate-limited removal sustain itself - routes v2 removes to the leader and skips the ordered delete for a terminator the router confirms it created that no longer exists, so a storm of no-op retries cannot consume the command rate limiter - gates that skip on the leader having applied everything committed when it took leadership, since a leader reports itself elected before those applies finish and would otherwise read a committed create as absent and drop its delete - guards the leader term and every write to the caught-up flag with a dedicated lock, so a barrier completing alongside a leadership change cannot record a result for a term that has already ended - scopes v2 removals to the requesting router, so a router can only remove terminators it owns on the asynchronous path as well as the synchronous one - tracks in-flight removal batches on the router, resolving or requeueing them when the response arrives; lost responses self-heal via rate-limiter expiry and the retry scan - waits for an outstanding operation before sending a terminator delete, and checks that before dequeuing, so a delete neither races a create the controller has committed but not applied nor falls out of the queue until the retry scan finds it minutes later - matches a create response to the attempt that produced it, by control channel and reply sequence, since attempts overlap once one overruns the establishment timeout and every attempt carries the same terminator id; only the outstanding attempt's response may confirm the create or report the terminator idle - records the attempt as the sender assigns its sequence, so a response cannot be processed before the request it answers is known, and withdraws an attempt whose send never queued - reports a removal that succeeds past the establishment timeout as congestion rather than success, matching how the establish path classifies its own latency, since both resolve into the same limiter - extracts the transient-error test both remove handlers share, so the v2 path treats a leaderless cluster as busy like v1 does instead of as a permanent failure, and the two cannot drift again - applies the same delete sequencing and response correlation to the router-hosted (ER/T) registry, which cleared the same in-flight marker unguarded and dropped a deferred delete from its queue rather than leaving it there

Controller should clear links not in full link sync from router. Fixes #4458 - adds a fullRefresh flag to the router's reconnect link announcement - prunes links sourced from the announcing router that are not in a full refresh set - moves the reroute of a usable faulted link into LinkFaulted - adds LinkManager.IterateLinks - For #3493

2026-09-23

Apply managed link heartbeat and queue settings at runtime. For #3743 - applies the router.link.v1 heartbeat settings (sendInterval, checkInterval, closeUnresponsiveTimeout) and payload/ack sender queue sizes at runtime, replacing the hardcoded 10s/1s link heartbeat intervals and the ignored fields; new links take them at bind time, established links are re-tuned in place through the channel HeartbeatControl, and queue sizes reach new links only - publishes heartbeat settings as an immutable generation in one atomic swap, so no reader pairs one generation's interval with another's timeout, and has each link enforce the timeout belonging to its own intervals - pushes a generation and its intervals as one step under the link's lock and ignores a generation at or below the one applied, so an overtaken update cannot roll a link back; reconciles every channel of a link, including a split link whose channels bind on either side of a config change, to the newest generation - adopts the first registered settings explicitly, since a router with no managed link config publishes its defaults at generation 0 - re-arms a link's liveness deadline when its generation changes, so age accumulated under the previous intervals cannot close a healthy link when a timeout is shortened - honors a closeUnresponsiveTimeout below 30s, which previously sat behind the unhealthy threshold; treats a zero timeout as closing nothing, and reports a closing link unhealthy - scales the unhealthy threshold with the intervals, to max(30s, 2 x (send + check)): reporting unhealthy poisons the latency metric the controller routes on, so a fixed 30s priced healthy links with a long send interval out of routing every cycle - validates heartbeat timing on the controller and the router: durations must be positive, their sum representable, and the timeout at least 10s and greater than sendInterval + checkInterval; a timeout under two heartbeat cycles is accepted with a warning - adds a per-config-type semantic validator registry to ConfigManager, run after schema validation, and moves the router.link config definition to common/config/routerlink so the controller applies the same rules - treats a link section in router YAML, of any content, as local config that takes precedence over the controller's for the whole router.link type, since the loader defaults every field and so now records presence separately; warns at startup when that section has no listeners or dialers - translates local heartbeat timing leniently, since a local failure stops the router from starting: keeps an explicit zero, raises a timeout below the floor to it, and falls back to the defaults for other invalid timing, each with a warning - stamps UpdateLinkListeners with the config generation it came from and has the controller drop superseded sets, which a multi-underlay control channel can deliver out of order and which would otherwise send peers to withdrawn addresses - logs every successful link config apply, including those that dispatch no work, and pins that a queue-size-only change triggers no GC pass - deprecates split links, documents the feature in the changelog and the router config design doc, and brings both up to date Builds on the channel/v5 HeartbeatControl from openziti/channel#265.

2026-09-16

Converge first-party CA public keys across controllers. Fixes #4407 Backport of #4405 to release-v2.0.x. For #4406. - adds a caPem field to the controller record, holding the intermediate CA certs from the controller's edge CA bundle, replicated through raft - has each controller report its own intermediates into its record on the existing self-update path - restricts leader-written peer record updates to hello-derived fields so a peer's caPem report is never overwritten - publishes each first-party root CA key with the deduplicated, byte-ordered union of intermediates from the local bundle and every controller record, so all controllers emit identical content per kid - republishes the root CA keys on controller record create, update and delete so late-joining and removed peers converge - removes a deleted controller's JWT validation key from the router data model - adds cert helpers for filtering intermediates and producing order-independent DER sets - gives each ha-3 test controller only its own signing intermediate and asserts the keys converge across all controllers and the router (cherry picked from commit 4a30738d9eaf2da7db4174f8c3ef1e81b0ce691d)

Converge first-party CA public keys across controllers. Fixes #4406 - adds a caPem field to the controller record, holding the intermediate CA certs from the controller's edge CA bundle, replicated through raft - has each controller report its own intermediates into its record on the existing self-update path - restricts leader-written peer record updates to hello-derived fields so a peer's caPem report is never overwritten - publishes each first-party root CA key with the deduplicated, byte-ordered union of intermediates from the local bundle and every controller record, so all controllers emit identical content per kid - republishes the root CA keys on controller record create, update and delete so late-joining and removed peers converge - removes a deleted controller's JWT validation key from the router data model - adds cert helpers for filtering intermediates and producing order-independent DER sets - gives each ha-3 test controller only its own signing intermediate and asserts the keys converge across all controllers and the router - syncs zititest module dependencies with the root module

2026-09-11

Auto-GC stale router links. Fixes #4005 - adds a structured xlink.LinkKey ({DialerBinding, Protocol, DestId, ListenerBinding}) used by all staleness and GC checks, replacing the pre-rendered key string each side used to pass around - adds the CheckStaleLinks ctrl message and the `ziti ops verify stale-links` CLI for operator-driven, two-sided staleness verification with optional --gc, which acts only when both endpoints agree - implements auto-GC via the router.link.v1 gcMode (preserve/orphaned/changed): the router walks its xlinks after each Apply that mutates listeners, dialers, or gcMode and closes one-sided-stale entries under the configured mode - sweeps a removal under orphaned whatever the removed config said, including preserve, since a removal leaves no listeners and no dialers and every remaining link is one the router can never re-establish - reports staleness as a three-state verdict (unknown/notStale/stale) so a side that cannot judge a link abstains instead of guessing; a destination absent from the registry's listener snapshot means "listeners unknown", not "peer has none", which keeps a briefly unhealthy peer from losing its live links - rebuilds the link surface only when a listener or dialer definition actually moves, so a gcMode-only change no longer rebinds every listen socket - gates CheckStaleLinks on a new RouterStaleLinkCheck capability and explains each unqueryable endpoint, including one left unasked by the filter, so a partial verdict says which of the three causes applies - detects a peer re-advertising the address a link was dialed to from the dialing side, not just from the peer's own listener side - pins every verdict to the link iteration it was computed against, faults that iteration rather than the current one, and takes the conditional removal's result as the decision, so a link re-dialed while the sweep was running is neither closed nor reported as collected - resolves which side a report speaks for from the sending router against the controller's link map rather than from the side the report names, so one router cannot fill both sides and satisfy the both-agree rule --gc depends on - bounds the query fan-out with a semaphore, matching the other validation sweeps, since each query carries a minute-long reply timeout - fails the CLI when the result stream ends before every link is reported, and drains results already buffered before declaring any missing - rejects an unrecognized match mode rather than resolving it to changed, the broadest removal criterion - consolidates the controller-managed router configuration changelog into one section, replacing the status that said routers do not yet consume configs at runtime, and documents the per-config-type opt-in - restores the Current Beta Features section, dropped when the 2.0 changelog was archived, and lists controller-managed router configuration in it - adds unparseable to the codespell ignore list, a spelling variant the dictionary flags as a typo For #3743.

[Backport-2.0] Replicate admin MFA removal across HA cluster members (#4295) * Replicate admin MFA removal across HA cluster members (#4284) * Run admin MFA removal as a single raft command so a lagging follower cannot serve a partial delete * Update CHANGELOG

2026-09-09

Replicate admin MFA removal across HA cluster members (#4284) * Run admin MFA removal as a single raft command so a lagging follower cannot serve a partial delete

2026-09-03

Report clock skew between gossip peers A tombstone now carries the deadline it will be reaped on, so expiry depends on wall clocks one node writes and another reads. Skew is bounded either way and cannot lose or corrupt state, but it does shift when tombstones are collected, and nothing made that visible. - stamps the sender's clock on the anti-entropy digest and on the canary. Those are the two axes a deadline crosses, since routers create tombstones as well as controllers, and both messages already run on a schedule - warns when the difference reaches a tenth of a tombstone lifetime, with a floor so a short lifetime does not make ordinary transit a warning. Expressed against the lifetime because that is what makes skew matter, so retuning one retunes the other - reports at most once per peer per ten minutes. Skew persists, so an unbounded report is a line per exchange for as long as the clocks disagree - names the direction, since a peer ahead of us stamps deadlines we collect late and a peer behind us stamps ones we collect early or that arrive already past It reports and corrects nothing. A one-way delta is skew plus transit and queueing, so it bounds skew rather than measuring it: enough to see the seconds to minutes that shift collection, and unfit to act on. Acting on it would trade a visible bounded problem for an invisible one.

Replicate link state over gossip. Fixes #3726 A router reported its links to every controller, and each controller kept its own picture built only from what routers told it directly. That does not survive routers being connected to a subset of controllers: a controller learns nothing about links whose routers it does not hold a connection to. Link state now lives in the replicated store: a router reports to one controller, that controller writes the entry it owns, and the mesh carries it to the rest. Each link entry is owned by the router that dialled it, so two controllers never contend for the same key, and a controller that has never spoken to a router still converges on its links. - registers a link state type on the gossip store and carries link add, update and removal through it - makes a link's source router an atomic and repoints it when the router connects, since a link can be built from a gossiped entry before its router has connected here, leaving a database-loaded placeholder as the endpoint - reconciles a reconnecting router's gossip entries, marking its links usable again rather than removing them, since a disconnect sets them down instead of deleting them - tombstones a link on disconnect in single-controller mode, where there is no peer to learn the removal from - adds the gossip transport: peer handlers on the controller mesh, router-facing gossip handlers, digest exchange off the receive goroutine, and the pools that bound apply and I/O work - has the digest exchange restamp a key the controller holds a higher version for, above that version, and send the live value. A router's Lamport clock is in memory, so a restart returns it to zero while the controller still holds versions from the previous incarnation under the same key. Link metrics are keyed by link id alone, and that id belongs to the dialer, so an acceptor's restart leaves the key unchanged and its republishes are refused as older. Keeping the stored version sends nothing, and every later digest reaches the same answer, so the exchange that exists to repair divergence would instead hold it in place. Safe because the router is the sole writer of the keys it advertises: it takes the clock from a digest but never a value - advertises a gossip capability so a router reports to one controller only once every controller can replicate, and falls back to reporting to all until then - adds canaries, a per-router sequence carried over the same path, so a router can tell that a controller has stopped applying its state - carries link metrics over gossip alongside the state - keeps the disconnect teardown's reroute ordering: the currency guard wraps it, and inside, the link snapshot and MarkDisconnected stay ahead of the cascade so reroute cannot path through the router being removed

Add a gossip protocol and replicated state store A controller cluster needs state that every member converges on without any member being authoritative for it, and without a full exchange every time something changes. This adds that as a standalone package, with no notion of what is being replicated. Each entry is owned by the peer that wrote it and carries a version from that owner's Lamport clock, so concurrent writes from different owners never contend and a late-arriving older write loses to the newer one it raced. Deletes are tombstones with a TTL, since an absent key and a deleted key are otherwise indistinguishable to a peer that has not seen the delete yet. Steady state is deltas: a change is broadcast once to the mesh rather than once per peer per entry. Anti-entropy covers what a delta missed, opening a round with per-owner hashes so two peers already in agreement exchange a hash rather than every entry version, and descending to entry versions only for the owners whose hashes differ. Each process gets an epoch, a UUIDv7 minted at startup, which gates deletion only: entries from a peer's previous lifetime can be swept when it comes back, while ordering stays with the version. Sweeping is conditional on the entry that was selected still being the entry there, so a replacement that lands mid-sweep is not deleted by it. - adds controller/gossip: the store, owner clocks, tombstones and their reaper, delta broadcast, anti-entropy with owner hashes, epochs, and the mesh abstraction it sends over - has a tombstone carry the deadline it will be reaped on, decided once by whoever created it. Expiring on a locally recorded arrival time instead makes anti-entropy self-defeating: reapers fire on independent tickers, the hash the exchange compares deliberately covers tombstones, so the node that reaps first is sent its tombstone back and starts it over. The two hand it to each other a TTL at a time and the owner's data is never compacted, which is the leak the reaper exists to prevent - bounds both directions of clock skew, since the deadline is now a wall clock one node sets and another reads. A deadline further out than one TTL is clamped on arrival, so a fast clock cannot mint a tombstone nothing ever reaps. An expired one is still applied rather than refused, so a slow clock cannot have its removals dropped on arrival and leave the entry it deletes in place; it is reaped on the next pass instead. Skew shifts when a tombstone goes and cannot lose or corrupt state - adds gossip_pb and the ctrl/mgmt message types the transport and inspection need - is transport-agnostic and carries no link, terminator or other domain concept; callers register a state type and supply their own encoding

2026-09-01

Auto-GC stale router links. Fixes #4005 - adds a structured xlink.LinkKey ({DialerBinding, Protocol, DestId, ListenerBinding}) used by all staleness and GC checks, replacing the pre-rendered key string each side used to pass around - adds the CheckStaleLinks ctrl message and the `ziti ops verify stale-links` CLI for operator-driven, two-sided staleness verification with optional --gc, which acts only when both endpoints agree - implements auto-GC via the router.link.v1 gcMode (preserve/orphaned/changed): the router walks its xlinks after each Apply that mutates listeners, dialers, or gcMode and closes one-sided-stale entries under the configured mode - reports staleness as a three-state verdict (unknown/notStale/stale) so a side that cannot judge a link abstains instead of guessing; a destination absent from the registry's listener snapshot means "listeners unknown", not "peer has none", which keeps a briefly unhealthy peer from losing its live links - rebuilds the link surface only when a listener or dialer definition actually moves, so a gcMode-only change no longer rebinds every listen socket - gates CheckStaleLinks on a new RouterStaleLinkCheck capability and explains each unqueryable endpoint, so a partial verdict distinguishes an offline peer from one too old to answer - detects a peer re-advertising the address a link was dialed to from the dialing side, not just from the peer's own listener side - pins every verdict to the link iteration it was computed against, faults that iteration rather than the current one, and takes the conditional removal's result as the decision, so a link re-dialed while the sweep was running is neither closed nor reported as collected - returns whether LinkManager.Remove actually removed, rather than discarding the link table's iteration guard - fails the CLI when the result stream ends before every link is reported, and drains results already buffered before declaring any missing, so neither a truncated run nor a random select outcome is read as a clean sweep - rejects an unrecognized match mode rather than resolving it to changed, the broadest removal criterion For #3743.

2026-08-29

Auto-GC stale router links. Fixes #4005 - adds a structured xlink.LinkKey ({DialerBinding, Protocol, DestId, ListenerBinding}) used by all staleness and GC checks, replacing the pre-rendered key string each side used to pass around - adds the CheckStaleLinks ctrl message and the `ziti ops verify stale-links` CLI for operator-driven, two-sided staleness verification with optional --gc, which acts only when both endpoints agree - implements auto-GC via the router.link.v1 gcMode (preserve/orphaned/changed): the router walks its xlinks after each Apply that mutates listeners, dialers, or gcMode and closes one-sided-stale entries under the configured mode - reports staleness as a three-state verdict (unknown/notStale/stale) so a side that cannot judge a link abstains instead of guessing; a destination absent from the registry's listener snapshot means "listeners unknown", not "peer has none", which keeps a briefly unhealthy peer from losing its live links - rebuilds the link surface only when a listener or dialer definition actually moves, so a gcMode-only change no longer rebinds every listen socket - gates CheckStaleLinks on a new RouterStaleLinkCheck capability and explains each unqueryable endpoint, so a partial verdict distinguishes an offline peer from one too old to answer - detects a peer re-advertising the address a link was dialed to from the dialing side, not just from the peer's own listener side For #3743.

2026-08-26

backport openziti/ziti#4094 to release-v2.0.x accept first-party certs issued by a separate edge signing CA (#4153) - publishes FirstPartyX509CertValidation/ThirdPartyX509CertValidation usages and intermediates on router data model public keys, deprecating ClientX509CertValidation - builds the router first-party cert pool from RDM first-party keys unioned with ctrl-channel roots; TLS and VerifyClientCert paths share buildClientCertRoots with fallback to the deprecated usage for old controllers - trusts the edge enrollment signing CA when verifying the certificate a router presents on the control channel, so a signing CA outside the controller's own trust bundle no longer refuses every router; the anchors go into a clone of the identity's pool, never the pool its live tls.Configs share - propagates full controller signing cert chains over the mesh via SigningCertChainHeader and persists them in Controller store CertPem - sends stored public keys during router sync instead of rebuilding them; publishes controller certs leaf-only - stops router controller reconnect loops after shutdown - gives each in-process controller its own command decoder registry - adds the ha-3 three-controller harness and first-party cert integration tests - drains the cli test stdout pipe while commands run; anchors the totp token issued-at assertion to the test clock - backports the SPIFFE-capable test PKI from openziti/ziti#3947: --not-before on ziti pki create, tests/testdata/create-pki.sh/.ps1, and the generated PKI under tests/testdata/pki including the separate edge signing root and per-controller signing intermediates; existing config sets stay on the testdata/ca PKI - skips *.pem, *.cert and *.key files in codespell

fixes openziti/ziti#4094 accept first-party certs issued by a separat… (#4140) * fixes openziti/ziti#4094 accept first-party certs issued by a separate edge signing CA - adds FirstPartyX509CertValidation and ThirdPartyX509CertValidation usages and an intermediates field to the router data model public keys, deprecating ClientX509CertValidation - publishes config CA bundle roots as first-party anchors with their intermediates and Ca store entries as third-party anchors; controller certs carry JWT validation only, since a controller identity is never a CA and anchors no client cert chains - builds router first-party and client cert trust pools from the published usages, falling back to the deprecated usage against older controllers - propagates the full signing cert chain between controllers via a new mesh SigningCertChainHeader and persists whole chains in controller records - removes the orphaned InstantStrategy.AddPublicKey, dead since public key sync moved to controller list data - gives each command dispatcher its own decoder registry so multiple in-process controllers no longer decode into the last-started controller's managers - adds a three-controller in-process HA test harness with a split signing PKI, cluster formation and first-party cert integration tests - trusts the edge signing CA when verifying router control channel certs - adds a variadic additionalRoots parameter to VerifyLeafCertChain, applied to a clone of the caller's pool so an identity's live tls.Configs are unaffected - passes the edge enrollment signing CA bundle as additional roots when admitting a router control channel connection, so a deployment whose signing CA sits outside the controller's own trust bundle no longer has every router refused - leaves the fingerprint check bound to the verified leaf, so the wider anchor set changes which chains verify, not which routers are admitted - covers the split-root case and the caller-pool guarantee in common/cert tests

2026-08-25

Auto-GC stale router links. Fixes #4005 - adds a structured xlink.LinkKey ({DialerBinding, Protocol, DestId, ListenerBinding}) used by all staleness and GC checks - adds the CheckStaleLinks ctrl message and the `ziti ops verify stale-links` CLI for operator-driven, two-sided staleness verification with optional --gc - implements auto-GC via the router.link.v1 gcMode (preserve/orphaned/changed): the router walks its xlinks after each Apply that mutates listeners, dialers, or gcMode and closes one-sided-stale entries under the configured mode For #3743.

2026-08-24

Replicate admin MFA removal across HA cluster members (#4284) * Run admin MFA removal as a single raft command so a lagging follower cannot serve a partial delete

Replicate admin MFA removal across HA cluster members (#4284) * Run admin MFA removal as a single raft command so a lagging follower cannot serve a partial delete

2026-08-21

Run admin MFA removal as a single raft command so a lagging follower cannot serve a partial delete

2026-08-14

Auto-GC stale router links. Fixes #4005 - adds a structured xlink.LinkKey ({DialerBinding, Protocol, DestId, ListenerBinding}) used by all staleness and GC checks - adds the CheckStaleLinks ctrl message and the `ziti ops verify stale-links` CLI for operator-driven, two-sided staleness verification with optional --gc - implements auto-GC via the router.link.v1 gcMode (preserve/orphaned/changed): the router walks its xlinks after each Apply that mutates listeners, dialers, or gcMode and closes one-sided-stale entries under the configured mode For #3743.

Replicate link state over gossip. Fixes #3726 A router reported its links to every controller, and each controller kept its own picture built only from what routers told it directly. That does not survive routers being connected to a subset of controllers: a controller learns nothing about links whose routers it does not hold a connection to. Link state now lives in the replicated store: a router reports to one controller, that controller writes the entry it owns, and the mesh carries it to the rest. Each link entry is owned by the router that dialled it, so two controllers never contend for the same key, and a controller that has never spoken to a router still converges on its links. - registers a link state type on the gossip store and carries link add, update and removal through it - makes a link's source router an atomic and repoints it when the router connects, since a link can be built from a gossiped entry before its router has connected here, leaving a database-loaded placeholder as the endpoint - reconciles a reconnecting router's gossip entries, marking its links usable again rather than removing them, since a disconnect sets them down instead of deleting them - tombstones a link on disconnect in single-controller mode, where there is no peer to learn the removal from - adds the gossip transport: peer handlers on the controller mesh, router-facing gossip handlers, digest exchange off the receive goroutine, and the pools that bound apply and I/O work - advertises a gossip capability so a router reports to one controller only once every controller can replicate, and falls back to reporting to all until then - adds canaries, a per-router sequence carried over the same path, so a router can tell that a controller has stopped applying its state - carries link metrics over gossip alongside the state - keeps the disconnect teardown's reroute ordering: the currency guard wraps it, and inside, the link snapshot and MarkDisconnected stay ahead of the cascade so reroute cannot path through the router being removed

Add a gossip protocol and replicated state store A controller cluster needs state that every member converges on without any member being authoritative for it, and without a full exchange every time something changes. This adds that as a standalone package, with no notion of what is being replicated. Each entry is owned by the peer that wrote it and carries a version from that owner's Lamport clock, so concurrent writes from different owners never contend and a late-arriving older write loses to the newer one it raced. Deletes are tombstones with a TTL, since an absent key and a deleted key are otherwise indistinguishable to a peer that has not seen the delete yet. Steady state is deltas: a change is broadcast once to the mesh rather than once per peer per entry. Anti-entropy covers what a delta missed, opening a round with per-owner hashes so two peers already in agreement exchange a hash rather than every entry version, and descending to entry versions only for the owners whose hashes differ. Each process gets an epoch, a UUIDv7 minted at startup, which gates deletion only: entries from a peer's previous lifetime can be swept when it comes back, while ordering stays with the version. Sweeping is conditional on the entry that was selected still being the entry there, so a replacement that lands mid-sweep is not deleted by it. - adds controller/gossip: the store, owner clocks, tombstones and their reaper, delta broadcast, anti-entropy with owner hashes, epochs, and the mesh abstraction it sends over - adds gossip_pb and the ctrl/mgmt message types the transport and inspection need - is transport-agnostic and carries no link, terminator or other domain concept; callers register a state type and supply their own encoding

2026-08-13

Auto-GC stale router links. Fixes #4005 - adds a structured xlink.LinkKey ({DialerBinding, Protocol, DestId, ListenerBinding}) used by all staleness and GC checks - adds the CheckStaleLinks ctrl message and the `ziti ops verify stale-links` CLI for operator-driven, two-sided staleness verification with optional --gc - implements auto-GC via the router.link.v1 gcMode (preserve/orphaned/changed): the router walks its xlinks after each Apply that mutates listeners, dialers, or gcMode and closes one-sided-stale entries under the configured mode For #3743.

Manage router link configuration via the controller. Fixes #4004 - adds a link ConfigHandler (router/link FactoryRegistry) that applies router.link.v1 config: Apply rebuilds the listener/dialer set wholesale, and established Xlinks survive because Listener.Close() only closes the accept loop - translates local link: YAML into router.link.v1 JSON and pushes it through the managed-config registry at startup - adds the UpdateLinkListeners ctrl message so the router republishes its listener set to the controller on change; the controller re-fans via the existing PeerStateChange path - re-evaluates dialers on link group and listener changes via RescanForDialOpportunities For #3743.

2026-08-12

Add leader fast-path for confirmed no-op terminator removals. For #4156 - routes V2 terminator removes to the leader (GetModelUpdateCtrlChannel), where the delete is applied anyway and the fast-path can consult current state - adds a createConfirmed flag (index-aligned with terminatorIds) to RemoveTerminatorsV2Request, set once the router receives the create acknowledgement; backwards compatible, older peers omit it and it decodes as false - on the leader, skips the raft/rate-limited delete for a terminator the router confirms was created but that no longer exists, so a storm of no-op retries can't consume the command rate limiter; unconfirmed or still-present ids take the normal ordered path - tests the fast-path filter across leader/confirmed/present/error and mixed cases

Add asynchronous terminator removal (RemoveTerminatorsV2). For #4156 - adds RemoveTerminatorsV2Request/Response control-channel messages, answered asynchronously so a router no longer holds a request slot open waiting for a synchronous removal reply - adds the ControllerAsyncTerminatorRemove capability so routers use the async path only against controllers that advertise support, falling back to synchronous removal otherwise - adds a controller handler that removes the batch and replies with per-batch success/rate-limited status - tracks in-flight removal batches on the router and, when the response arrives, resolves the batch's rate-limit control and clears or requeues its terminators; lost responses self-heal via the existing rate-limiter expiry and retry scan

Replicate link state over gossip. Fixes #3726 A router reported its links to every controller, and each controller kept its own picture built only from what routers told it directly. That does not survive routers being connected to a subset of controllers: a controller learns nothing about links whose routers it does not hold a connection to. Link state now lives in the replicated store: a router reports to one controller, that controller writes the entry it owns, and the mesh carries it to the rest. Each link entry is owned by the router that dialled it, so two controllers never contend for the same key, and a controller that has never spoken to a router still converges on its links. - registers a link state type on the gossip store and carries link add, update and removal through it - makes a link's source router an atomic and repoints it when the router connects, since a link can be built from a gossiped entry before its router has connected here, leaving a database-loaded placeholder as the endpoint - reconciles a reconnecting router's gossip entries, marking its links usable again rather than removing them, since a disconnect sets them down instead of deleting them - tombstones a link on disconnect in single-controller mode, where there is no peer to learn the removal from - adds the gossip transport: peer handlers on the controller mesh, router-facing gossip handlers, digest exchange off the receive goroutine, and the pools that bound apply and I/O work - advertises a gossip capability so a router reports to one controller only once every controller can replicate, and falls back to reporting to all until then - adds canaries, a per-router sequence carried over the same path, so a router can tell that a controller has stopped applying its state - carries link metrics over gossip alongside the state - keeps the disconnect teardown's reroute ordering: the currency guard wraps it, and inside, the link snapshot and MarkDisconnected stay ahead of the cascade so reroute cannot path through the router being removed

Add a gossip protocol and replicated state store A controller cluster needs state that every member converges on without any member being authoritative for it, and without a full exchange every time something changes. This adds that as a standalone package, with no notion of what is being replicated. Each entry is owned by the peer that wrote it and carries a version from that owner's Lamport clock, so concurrent writes from different owners never contend and a late-arriving older write loses to the newer one it raced. Deletes are tombstones with a TTL, since an absent key and a deleted key are otherwise indistinguishable to a peer that has not seen the delete yet. Steady state is deltas: a change is broadcast once to the mesh rather than once per peer per entry. Anti-entropy covers what a delta missed, opening a round with per-owner hashes so two peers already in agreement exchange a hash rather than every entry version, and descending to entry versions only for the owners whose hashes differ. Each process gets an epoch, a UUIDv7 minted at startup, which gates deletion only: entries from a peer's previous lifetime can be swept when it comes back, while ordering stays with the version. Sweeping is conditional on the entry that was selected still being the entry there, so a replacement that lands mid-sweep is not deleted by it. - adds controller/gossip: the store, owner clocks, tombstones and their reaper, delta broadcast, anti-entropy with owner hashes, epochs, and the mesh abstraction it sends over - adds gossip_pb and the ctrl/mgmt message types the transport and inspection need - is transport-agnostic and carries no link, terminator or other domain concept; callers register a state type and supply their own encoding

Auto-GC stale router links. Fixes #4005 - adds a structured xlink.LinkKey ({DialerBinding, Protocol, DestId, ListenerBinding}) used by all staleness and GC checks - adds the CheckStaleLinks ctrl message and the `ziti ops verify stale-links` CLI for operator-driven, two-sided staleness verification with optional --gc - implements auto-GC via the router.link.v1 gcMode (preserve/orphaned/changed): the router walks its xlinks after each Apply that mutates listeners, dialers, or gcMode and closes one-sided-stale entries under the configured mode For #3743.

Manage router link configuration via the controller. Fixes #4004 - adds a link ConfigHandler (router/link FactoryRegistry) that applies router.link.v1 config: Apply rebuilds the listener/dialer set wholesale, and established Xlinks survive because Listener.Close() only closes the accept loop - translates local link: YAML into router.link.v1 JSON and pushes it through the managed-config registry at startup - adds the UpdateLinkListeners ctrl message so the router republishes its listener set to the controller on change; the controller re-fans via the existing PeerStateChange path - re-evaluates dialers on link group and listener changes via RescanForDialOpportunities For #3743.

Merge pull request #4216 from openziti/router-link-costtags-removal Remove unused link costTags support

2026-08-11

Replicate link state over gossip. Fixes #3726 A router reported its links to every controller, and each controller kept its own picture built only from what routers told it directly. That does not survive routers being connected to a subset of controllers: a controller learns nothing about links whose routers it does not hold a connection to. Link state now lives in the replicated store: a router reports to one controller, that controller writes the entry it owns, and the mesh carries it to the rest. Each link entry is owned by the router that dialled it, so two controllers never contend for the same key, and a controller that has never spoken to a router still converges on its links. - registers a link state type on the gossip store and carries link add, update and removal through it - makes a link's source router an atomic and repoints it when the router connects, since a link can be built from a gossiped entry before its router has connected here, leaving a database-loaded placeholder as the endpoint - reconciles a reconnecting router's gossip entries, marking its links usable again rather than removing them, since a disconnect sets them down instead of deleting them - tombstones a link on disconnect in single-controller mode, where there is no peer to learn the removal from - adds the gossip transport: peer handlers on the controller mesh, router-facing gossip handlers, digest exchange off the receive goroutine, and the pools that bound apply and I/O work - advertises a gossip capability so a router reports to one controller only once every controller can replicate, and falls back to reporting to all until then - adds canaries, a per-router sequence carried over the same path, so a router can tell that a controller has stopped applying its state - carries link metrics over gossip alongside the state - keeps the disconnect teardown's reroute ordering: the currency guard wraps it, and inside, the link snapshot and MarkDisconnected stay ahead of the cascade so reroute cannot path through the router being removed

Add a gossip protocol and replicated state store A controller cluster needs state that every member converges on without any member being authoritative for it, and without a full exchange every time something changes. This adds that as a standalone package, with no notion of what is being replicated. Each entry is owned by the peer that wrote it and carries a version from that owner's Lamport clock, so concurrent writes from different owners never contend and a late-arriving older write loses to the newer one it raced. Deletes are tombstones with a TTL, since an absent key and a deleted key are otherwise indistinguishable to a peer that has not seen the delete yet. Steady state is deltas: a change is broadcast once to the mesh rather than once per peer per entry. Anti-entropy covers what a delta missed, opening a round with per-owner hashes so two peers already in agreement exchange a hash rather than every entry version, and descending to entry versions only for the owners whose hashes differ. Each process gets an epoch, a UUIDv7 minted at startup, which gates deletion only: entries from a peer's previous lifetime can be swept when it comes back, while ordering stays with the version. Sweeping is conditional on the entry that was selected still being the entry there, so a replacement that lands mid-sweep is not deleted by it. - adds controller/gossip: the store, owner clocks, tombstones and their reaper, delta broadcast, anti-entropy with owner hashes, epochs, and the mesh abstraction it sends over - adds gossip_pb and the ctrl/mgmt message types the transport and inspection need - is transport-agnostic and carries no link, terminator or other domain concept; callers register a state type and supply their own encoding

2026-08-06

Persist cluster id across migration snapshot restore. For #4104 - adds a clusterId field to SyncSnapshotCommand and writes it into the database after the migration snapshot restore, so a controller bootstrapped by migrating a database ends up with a durable cluster id instead of an empty one - the snapshot restore replaces the whole FSM database with the migration source, which carries no cluster id, so without this the id written during bootstrap was silently wiped and the node came up with an empty, non-durable cluster id, defeating the mesh cluster-id validation - persists the raft index after the cluster id in RestoreSnapshot so the index remains the completion gate: a failure before it halts (SyncSnapshotCommand is a critical command) and replays/retries on restart rather than skipping the command with a blank cluster id - fails RaftRestoreFromBoltDb when the cluster id is blank after bootstrap - regenerates cmd.pb.go for the new field (cherry picked from commit d8cedeb8ea1813221808f3da556a8fc59efae431)

Merge pull request #4105 from openziti/ha-cluster-bootstrap-guards Controller cluster bootstrapping fixes

Auto-GC stale router links. Fixes #4005 - adds a structured xlink.LinkKey ({DialerBinding, Protocol, DestId, ListenerBinding}) used by all staleness and GC checks - adds the CheckStaleLinks ctrl message and the `ziti ops verify stale-links` CLI for operator-driven, two-sided staleness verification with optional --gc - implements auto-GC via the router.link.v1 gcMode (preserve/orphaned/changed): the router walks its xlinks after each Apply that mutates listeners, dialers, or gcMode and closes one-sided-stale entries under the configured mode For #3743.

Manage router link configuration via the controller. Fixes #4004 - adds a link ConfigHandler (router/link FactoryRegistry) that applies router.link.v1 config: Apply rebuilds the listener/dialer set wholesale, and established Xlinks survive because Listener.Close() only closes the accept loop - translates local link: YAML into router.link.v1 JSON and pushes it through the managed-config registry at startup - adds the UpdateLinkListeners ctrl message so the router republishes its listener set to the controller on change; the controller re-fans via the existing PeerStateChange path - re-evaluates dialers on link group and listener changes via RescanForDialOpportunities For #3743.

2026-08-04

Add leader fast-path for confirmed no-op terminator removals. For #4156 - routes V2 terminator removes to the leader (GetModelUpdateCtrlChannel), where the delete is applied anyway and the fast-path can consult current state - adds a createConfirmed flag (index-aligned with terminatorIds) to RemoveTerminatorsV2Request, set once the router receives the create acknowledgement; backwards compatible, older peers omit it and it decodes as false - on the leader, skips the raft/rate-limited delete for a terminator the router confirms was created but that no longer exists, so a storm of no-op retries can't consume the command rate limiter; unconfirmed or still-present ids take the normal ordered path - tests the fast-path filter across leader/confirmed/present/error and mixed cases

Add asynchronous terminator removal (RemoveTerminatorsV2). For #4156 - adds RemoveTerminatorsV2Request/Response control-channel messages, answered asynchronously so a router no longer holds a request slot open waiting for a synchronous removal reply - adds the ControllerAsyncTerminatorRemove capability so routers use the async path only against controllers that advertise support, falling back to synchronous removal otherwise - adds a controller handler that removes the batch and replies with per-batch success/rate-limited status - tracks in-flight removal batches on the router and, when the response arrives, resolves the batch's rate-limit control and clears or requeues its terminators; lost responses self-heal via the existing rate-limiter expiry and retry scan

Auto-GC stale router links. Fixes #4005 - adds a structured xlink.LinkKey ({DialerBinding, Protocol, DestId, ListenerBinding}) used by all staleness and GC checks - adds the CheckStaleLinks ctrl message and the `ziti ops verify stale-links` CLI for operator-driven, two-sided staleness verification with optional --gc - implements auto-GC via the router.link.v1 gcMode (preserve/orphaned/changed): the router walks its xlinks after each Apply that mutates listeners, dialers, or gcMode and closes one-sided-stale entries under the configured mode For #3743.

Manage router link configuration via the controller. Fixes #4004 - adds a link ConfigHandler (router/link FactoryRegistry) that applies router.link.v1 config: Apply rebuilds the listener/dialer set wholesale, and established Xlinks survive because Listener.Close() only closes the accept loop - translates local link: YAML into router.link.v1 JSON and pushes it through the managed-config registry at startup - adds the UpdateLinkListeners ctrl message so the router republishes its listener set to the controller on change; the controller re-fans via the existing PeerStateChange path - re-evaluates dialers on link group and listener changes via RescanForDialOpportunities For #3743.

2026-08-03

Remove unused link costTags support - removes the costTags option from router link listeners, which was parsed from config, advertised to the controller, and stored on the router model but never used for path selection or any other behavior - drops GetLinkCostTags from the xlink.Listener interface and its transport implementation - reserves the corresponding ctrl_pb.Listener.costTags and RouterLinks.RouterLink.linkCostTags protobuf fields and regenerates ctrl.pb.go - documents the removal in the changelog

2026-07-31

Add controller Limbo state, takeover splice, and reroute token issuance - adds circuit reroute state (active/Limbo) with a grace timer, entered on an ingress ChannelClosed fault for a reroutable circuit and torn down on grace expiry; issues a controller-signed reroute token at circuit creation when reroutable - adds the takeover splice (X1): re-splices a reroutable circuit's ingress to a new router under the circuit's mutation guard, preserving the terminator side, advancing the iteration, and minting a fresh token - stamps the fault reason on router-emitted ingress faults so only a channel close is Limbo-eligible - keeps the D3 idle scanner from prematurely reaping Limbo circuits

Add FaultReason to the Fault message - adds a FaultReason enum (ReasonUnspecified/ChannelClosed/XgClose/AccessLoss) with load-bearing numbering: the proto3 zero value decodes to teardown, and only ChannelClosed is eligible for a Limbo hold and SDK-driven reroute - adds the reason field to Fault so routers can stamp why a circuit faulted - regenerates ctrl.pb.go with protoc-gen-go v1.36.11

2026-07-29

Add link metrics over gossip - replicates per-link latency metrics across controllers via gossip - narrows the metrics firehose to one controller when link latency is in gossip - validates firehose narrowing in the links-test by total ingestion - documents the metrics rollout phases

Add link state replication over gossip. Fixes #3726 - replicates link state across controllers via the gossip mesh so a router can report links to a single controller instead of all - makes Link.Src atomic and repoints stale source routers at connect - adds gossip integration progress/design docs

2026-07-28

Auto-GC stale router links. Fixes #4005 - adds a structured xlink.LinkKey ({DialerBinding, Protocol, DestId, ListenerBinding}) used by all staleness and GC checks - adds the CheckStaleLinks ctrl message and the `ziti ops verify stale-links` CLI for operator-driven, two-sided staleness verification with optional --gc - implements auto-GC via the router.link.v1 gcMode (preserve/orphaned/changed): the router walks its xlinks after each Apply that mutates listeners, dialers, or gcMode and closes one-sided-stale entries under the configured mode For #3743.

Remove unused link costTags support - removes the costTags option from router link listeners, which was parsed from config, advertised to the controller, and stored on the router model but never used for path selection or any other behavior - drops GetLinkCostTags from the xlink.Listener interface, its transport implementation, and test stubs - reserves the corresponding ctrl_pb.Listener.costTags and RouterLinks.RouterLink.linkCostTags protobuf fields and regenerates ctrl.pb.go - documents the removal in the changelog

Manage router link configuration via the controller. Fixes #4004 - adds a link ConfigHandler (router/link FactoryRegistry) that applies router.link.v1 config: Apply rebuilds the listener/dialer set wholesale, and established Xlinks survive because Listener.Close() only closes the accept loop - translates local link: YAML into router.link.v1 JSON and pushes it through the managed-config registry at startup - adds the UpdateLinkListeners ctrl message so the router republishes its listener set to the controller on change; the controller re-fans via the existing PeerStateChange path - re-evaluates dialers on link group and listener changes via RescanForDialOpportunities For #3743.

2026-07-24

Add leader fast-path for confirmed no-op terminator removals. For #4156 - routes V2 terminator removes to the leader (GetModelUpdateCtrlChannel), where the delete is applied anyway and the fast-path can consult current state - adds a createConfirmed flag (index-aligned with terminatorIds) to RemoveTerminatorsV2Request, set once the router receives the create acknowledgement; backwards compatible, older peers omit it and it decodes as false - on the leader, skips the raft/rate-limited delete for a terminator the router confirms was created but that no longer exists, so a storm of no-op retries can't consume the command rate limiter; unconfirmed or still-present ids take the normal ordered path - tests the fast-path filter across leader/confirmed/present/error and mixed cases

Add asynchronous terminator removal (RemoveTerminatorsV2). For #4156 - adds RemoveTerminatorsV2Request/Response control-channel messages, answered asynchronously so a router no longer holds a request slot open waiting for a synchronous removal reply - adds the ControllerAsyncTerminatorRemove capability so routers use the async path only against controllers that advertise support, falling back to synchronous removal otherwise - adds a controller handler that removes the batch and replies with per-batch success/rate-limited status - tracks in-flight removal batches on the router and, when the response arrives, resolves the batch's rate-limit control and clears or requeues its terminators; lost responses self-heal via the existing rate-limiter expiry and retry scan

2026-07-23

backport openziti/ziti#4094 to release-v2.0.x accept first-party certs issued by a separate edge signing CA - publishes FirstPartyX509CertValidation/ThirdPartyX509CertValidation usages and intermediates on router data model public keys, deprecating ClientX509CertValidation - builds the router first-party cert pool from RDM first-party keys unioned with ctrl-channel roots; TLS and VerifyClientCert paths share buildClientCertRoots with fallback to the deprecated usage for old controllers - propagates full controller signing cert chains over the mesh via SigningCertChainHeader and persists them in Controller store CertPem - sends stored public keys during router sync instead of rebuilding them; publishes controller certs leaf-only - stops router controller reconnect loops after shutdown - gives each in-process controller its own command decoder registry - adds the ha-3 three-controller harness and first-party cert integration tests - drains the cli test stdout pipe while commands run; anchors the totp token issued-at assertion to the test clock - backports the SPIFFE-capable test PKI from openziti/ziti#3947: --not-before on ziti pki create, tests/testdata/create-pki.sh/.ps1, and the generated PKI under tests/testdata/pki including the separate edge signing root and per-controller signing intermediates; existing config sets stay on the testdata/ca PKI - skips *.pem, *.cert and *.key files in codespell

2026-07-22

fixes openziti/ziti#4094 accept first-party certs issued by a separate edge signing CA - adds FirstPartyX509CertValidation and ThirdPartyX509CertValidation usages and an intermediates field to the router data model public keys, deprecating ClientX509CertValidation - publishes config CA bundle roots as first-party anchors with their intermediates and Ca store entries as third-party anchors; controller certs carry JWT validation only, since a controller identity is never a CA and anchors no client cert chains - builds router first-party and client cert trust pools from the published usages, falling back to the deprecated usage against older controllers - propagates the full signing cert chain between controllers via a new mesh SigningCertChainHeader and persists whole chains in controller records - removes the orphaned InstantStrategy.AddPublicKey, dead since public key sync moved to controller list data - gives each command dispatcher its own decoder registry so multiple in-process controllers no longer decode into the last-started controller's managers - adds a three-controller in-process HA test harness with a split signing PKI, cluster formation and first-party cert integration tests

fixes openziti/ziti#3990 push service and posture changes to subscrib… (#4057) * fixes openziti/ziti#3990 push service and posture state to subscribed SDKs - pushes indexed atomic ServiceChangeSet envelopes to subscribed SDK connections: a full snapshot on subscribe, incremental service changes per RDM scan pass, posture check definition changes as their own entries, and identity-resolved config bodies, all serialized so envelopes hit the wire in index order - pushes per-connection PostureStateChange state (monotonic seq, resync on request) for posture pass/fail, including flips caused by definition edits that mutate no posture data - registers pending RDM identity subscriptions for identities not yet synced to the router and sends an authoritative full sync plus full posture state when the identity arrives; an active push subscription pins the connection's RDM listener - advertises service subscriptions and router data model support on the control-channel capability bitmask; the controller persists each router's capabilities mask and version on the EdgeRouter entity via raft and renders them on the edge APIs, so SDKs can select capable routers before connecting - submits posture per router and corrects MFA posture semantics: pushed expiry is the earliest of timeout and pending wake/unlock grace deadlines, wake/unlock re-pass satisfies the re-prompt, api session tokens whose amr attests TOTP seed the MFA baseline from auth_time only (never iat), and token exchange carries the subject token's auth_time - sends structured denials on dial and bind refusals: posture failures carry the failing check ids, no-policy denials are access denied, unknown services are invalid service, and session token failures are invalid session; the denial's cause no longer rides the wire as an unserializable error - hard-closes accepted SDK connections on edge listener shutdown so clients observe a router going away immediately - adds integration coverage: subscription snapshots and change delivery, poll and push reconciliation as capable routers come and go, posture state and definition-change push, router views over the public SDK API, typed dial errors, MFA baseline seeding, and OIDC token-exchange auth_time preservation - removed RDM capability from SDK, router/controller only

2026-07-21

Auto-GC stale router links. Fixes #4005 - adds a structured xlink.LinkKey ({DialerBinding, Protocol, DestId, ListenerBinding}) used by all staleness and GC checks - adds the CheckStaleLinks ctrl message and the `ziti ops verify stale-links` CLI for operator-driven, two-sided staleness verification with optional --gc - implements auto-GC via the router.link.v1 gcMode (preserve/orphaned/changed): the router walks its xlinks after each Apply that mutates listeners, dialers, or gcMode and closes one-sided-stale entries under the configured mode For #3743.

Manage router link configuration via the controller. Fixes #4004 - adds a link ConfigHandler (router/link FactoryRegistry) that applies router.link.v1 config: Apply rebuilds the listener/dialer set wholesale, and established Xlinks survive because Listener.Close() only closes the accept loop - translates local link: YAML into router.link.v1 JSON and pushes it through the managed-config registry at startup - adds the UpdateLinkListeners ctrl message so the router republishes its listener set to the controller on change; the controller re-fans via the existing PeerStateChange path - re-evaluates dialers on link group and listener changes via RescanForDialOpportunities For #3743.

2026-07-20

Manage router link configuration via the controller. Fixes #4004 - adds a link ConfigHandler (router/link FactoryRegistry) that applies router.link.v1 config: Apply rebuilds the listener/dialer set wholesale, and established Xlinks survive because Listener.Close() only closes the accept loop - translates local link: YAML into router.link.v1 JSON and pushes it through the managed-config registry at startup - adds the UpdateLinkListeners ctrl message so the router republishes its listener set to the controller on change; the controller re-fans via the existing PeerStateChange path - re-evaluates dialers on link group and listener changes via RescanForDialOpportunities For #3743.

2026-07-18

Manage router link configuration via the controller. Fixes #4004 - adds a link ConfigHandler (router/link FactoryRegistry) that applies router.link.v1 config: Apply rebuilds the listener/dialer set wholesale, and established Xlinks survive because Listener.Close() only closes the accept loop - translates local link: YAML into router.link.v1 JSON and pushes it through the managed-config registry at startup - adds the UpdateLinkListeners ctrl message so the router republishes its listener set to the controller on change; the controller re-fans via the existing PeerStateChange path - re-evaluates dialers on link group and listener changes via RescanForDialOpportunities For #3743.

2026-07-17

Persist cluster id across migration snapshot restore. For #4104 - adds a clusterId field to SyncSnapshotCommand and writes it into the database after the migration snapshot restore, so a controller bootstrapped by migrating a database ends up with a durable cluster id instead of an empty one - the snapshot restore replaces the whole FSM database with the migration source, which carries no cluster id, so without this the id written during bootstrap was silently wiped and the node came up with an empty, non-durable cluster id, defeating the mesh cluster-id validation - persists the raft index after the cluster id in RestoreSnapshot so the index remains the completion gate: a failure before it halts (SyncSnapshotCommand is a critical command) and replays/retries on restart rather than skipping the command with a blank cluster id - fails RaftRestoreFromBoltDb when the cluster id is blank after bootstrap - regenerates cmd.pb.go for the new field

2026-07-07

fixes openziti/ziti#3990 push service and posture state to subscribed SDKs - pushes indexed atomic ServiceChangeSet envelopes to subscribed SDK connections: a full snapshot on subscribe, incremental service changes per RDM scan pass, posture check definition changes as their own entries, and identity-resolved config bodies, all serialized so envelopes hit the wire in index order - pushes per-connection PostureStateChange state (monotonic seq, resync on request) for posture pass/fail, including flips caused by definition edits that mutate no posture data - registers pending RDM identity subscriptions for identities not yet synced to the router and sends an authoritative full sync plus full posture state when the identity arrives; an active push subscription pins the connection's RDM listener - advertises service subscriptions and router data model support on the control-channel capability bitmask; the controller persists each router's capabilities mask and version on the EdgeRouter entity via raft and renders them on the edge APIs, so SDKs can select capable routers before connecting - submits posture per router and corrects MFA posture semantics: pushed expiry is the earliest of timeout and pending wake/unlock grace deadlines, wake/unlock re-pass satisfies the re-prompt, api session tokens whose amr attests TOTP seed the MFA baseline from auth_time only (never iat), and token exchange carries the subject token's auth_time - sends structured denials on dial and bind refusals: posture failures carry the failing check ids, no-policy denials are access denied, unknown services are invalid service, and session token failures are invalid session; the denial's cause no longer rides the wire as an unserializable error - hard-closes accepted SDK connections on edge listener shutdown so clients observe a router going away immediately - adds integration coverage: subscription snapshots and change delivery, poll and push reconciliation as capable routers come and go, posture state and definition-change push, router views over the public SDK API, typed dial errors, MFA baseline seeding, and OIDC token-exchange auth_time preservation

2026-07-03

fixes openziti/ziti#3990 push service and posture changes to subscribed SDKs - routers push service change (ServiceChangeSet) and posture state (PostureStateChange) notifications directly to subscribed SDK connections, removing the controller from this path - adds RouterServiceSubscriptions and RouterDataModel capability bits and advertises them in the control-channel capabilities mask - persists each edge router's reported capabilities mask and version with a write-on-mismatch raft update, and renders them on the edge router APIs as a capability string list and version - normalizes a single PROCESS posture check to a one-element PROCESS_MULTI on the wire - closes accepted edge connections on router shutdown so SDKs detect a disconnect promptly rather than waiting for a heartbeat timeout - adds integration tests for capability reporting, service subscriptions, push/poll reconciliation, and posture-state push - resolves each subscribing identity's effective config set (service base configs merged with the identity's per-service overrides) and includes the config bodies on pushed service defs - adds GetIdentityServiceConfigs and refactors loadServiceConfigs to delegate to it - config edits ride the existing service-change path: a config change pushes a service-changed to every service that references it, carrying the new body - adds an integration test covering snapshot bodies, adding a config to a service, and editing a config shared by two services - asserts the resolved config body is returned by GetService, not just present on the service event, in the snapshot, config-added, and config-edited cases

Make identity revocation timestamps deterministic across controllers. Fixes #4088 - adds a mutation timestamp to the change context (both ChangeContext protos and the change.Context struct), stamped once when a command is dispatched so the replicated command carries the originating controller's time to every raft node, mirroring how raftIndex is threaded - has the identity revocation constraint derive the revocation's ExpiresAt and IssuedBefore from a cluster-consistent time instead of a per-node time.Now(): the disable transition uses the carried DisabledAt (also the correct cutoff), and delete uses the change-context timestamp; falls back to time.Now() only when no change context is present - adds change-context timestamp round-trip tests, including the zero-time mapping

2026-07-02

Auto-GC stale router links. Fixes #4005 - adds a structured xlink.LinkKey ({DialerBinding, Protocol, DestId, ListenerBinding}) used by all staleness and GC checks - adds the CheckStaleLinks ctrl message and the `ziti ops verify stale-links` CLI for operator-driven, two-sided staleness verification with optional --gc - implements auto-GC via the router.link.v1 gcMode (preserve/orphaned/changed): the router walks its xlinks after each Apply that mutates listeners, dialers, or gcMode and closes one-sided-stale entries under the configured mode For #3743.

Manage router link configuration via the controller. Fixes #4004 - adds a link ConfigHandler (router/link FactoryRegistry) that applies router.link.v1 config: Apply rebuilds the listener/dialer set wholesale, and established Xlinks survive because Listener.Close() only closes the accept loop - translates local link: YAML into router.link.v1 JSON and pushes it through the managed-config registry at startup - adds the UpdateLinkListeners ctrl message so the router republishes its listener set to the controller on change; the controller re-fans via the existing PeerStateChange path - re-evaluates dialers on link group and listener changes via RescanForDialOpportunities For #3743.

2026-06-30

Own the metrics wire format in ziti. Fixes #4036 - adds a common/servermetrics package that owns the metrics MetricsMessage wire format and the reporting/usage subsystem (message builder, usage registry, interval and usage counters), wrapping the openziti/metrics Registry for metric collection - moves the controllers metrics reporter into the router package and removes it from the shared metrics package, breaking a common -> router/env import cycle - repoints controller and router consumers to common/servermetrics; base metric collection stays on openziti/metrics - keeps the proto field numbers and the metrics content-type identical so the encoding is byte-compatible across the move, and uses a distinct proto package name so ziti's and the library's messages coexist without a global proto registry clash - adds a round-trip test asserting wire compatibility with the library's MetricsMessage - leaves openziti/metrics unchanged, so sdk-golang and the shared xgress data plane are unaffected

2026-06-29

fixes openziti/ziti#3990 push service and posture changes to subscribed SDKs - routers push service change (ServiceChangeSet) and posture state (PostureStateChange) notifications directly to subscribed SDK connections, removing the controller from this path - adds RouterServiceSubscriptions and RouterDataModel capability bits and advertises them in the control-channel capabilities mask - persists each edge router's reported capabilities mask and version with a write-on-mismatch raft update, and renders them on the edge router APIs as a capability string list and version - normalizes a single PROCESS posture check to a one-element PROCESS_MULTI on the wire - closes accepted edge connections on router shutdown so SDKs detect a disconnect promptly rather than waiting for a heartbeat timeout - adds integration tests for capability reporting, service subscriptions, push/poll reconciliation, and posture-state push - resolves each subscribing identity's effective config set (service base configs merged with the identity's per-service overrides) and includes the config bodies on pushed service defs - adds GetIdentityServiceConfigs and refactors loadServiceConfigs to delegate to it - config edits ride the existing service-change path: a config change pushes a service-changed to every service that references it, carrying the new body - adds an integration test covering snapshot bodies, adding a config to a service, and editing a config shared by two services - asserts the resolved config body is returned by GetService, not just present on the service event, in the snapshot, config-added, and config-edited cases

2026-06-27

Auto-GC stale router links. Fixes #4005 - adds a structured xlink.LinkKey ({DialerBinding, Protocol, DestId, ListenerBinding}) used by all staleness and GC checks - adds the CheckStaleLinks ctrl message and the `ziti ops verify stale-links` CLI for operator-driven, two-sided staleness verification with optional --gc - implements auto-GC via the router.link.v1 gcMode (preserve/orphaned/changed): the router walks its xlinks after each Apply that mutates listeners, dialers, or gcMode and closes one-sided-stale entries under the configured mode For #3743.

Manage router link configuration via the controller. Fixes #4004 - adds a link ConfigHandler (router/link FactoryRegistry) that applies router.link.v1 config: Apply rebuilds the listener/dialer set wholesale, and established Xlinks survive because Listener.Close() only closes the accept loop - translates local link: YAML into router.link.v1 JSON and pushes it through the managed-config registry at startup - adds the UpdateLinkListeners ctrl message so the router republishes its listener set to the controller on change; the controller re-fans via the existing PeerStateChange path - re-evaluates dialers on link group and listener changes via RescanForDialOpportunities For #3743.

2026-06-26

Manage router link configuration via the controller. Fixes #4004 - adds a link ConfigHandler (router/link FactoryRegistry) that applies router.link.v1 config: Apply rebuilds the listener/dialer set wholesale, and established Xlinks survive because Listener.Close() only closes the accept loop - translates local link: YAML into router.link.v1 JSON and pushes it through the managed-config registry at startup - adds the UpdateLinkListeners ctrl message so the router republishes its listener set to the controller on change; the controller re-fans via the existing PeerStateChange path - re-evaluates dialers on link group and listener changes via RescanForDialOpportunities For #3743.

Auto-GC stale router links. Fixes #4005 - adds a structured xlink.LinkKey ({DialerBinding, Protocol, DestId, ListenerBinding}) used by all staleness and GC checks - adds the CheckStaleLinks ctrl message and the `ziti ops verify stale-links` CLI for operator-driven, two-sided staleness verification with optional --gc - implements auto-GC via the router.link.v1 gcMode (preserve/orphaned/changed): the router walks its xlinks after each Apply that mutates listeners, dialers, or gcMode and closes one-sided-stale entries under the configured mode For #3743.

2026-06-25

Add link metrics over gossip Replicates per-link latency across controllers via gossip so routing cost converges HA-wide, replacing per-controller derivation from the metrics firehose. - adds a link-metrics gossip store keyed by link id and owned by the reporting router, with a router publisher (threshold-based, force-publish on re-dial) and a controller listener (exact-iteration apply, both-end reconcile on link create) - adds a linkLatencyInGossip field to the servermetrics metrics message; the router sets it once every connected controller is gossip-capable, and the controller then stops deriving routing latency from that message while still relaying its histograms for observability - adds an inspect surface for the gossip link-metrics store

Add link state replication over gossip. Fixes #3726 Replicates router-reported link state across HA controllers via a gossip store instead of per-controller fault forwarding, so every controller converges on the same view of links. - adds a versioned gossip store (Lamport clocks, deltas, acks, digests, anti-entropy, tombstones) and a generic per-state-type registration model - adds the link-state gossip type, controller listener, and router gossip client with reconcile and superseded-iteration tombstoning - adds a canary gossip type and controller/router canary handlers to detect divergence, plus inspect and `validate gossip` surfaces - advertises the ControllerLinkGossip capability and gates the router's link-state propagation on all controllers supporting it - ports the affected controller and router handlers from the channel v4 receive-handler API to v5 (AddReceiveHandlers / ContentTypeReceiver) - carries the gossip-stability work: bounded peer send timeouts, broadcast and digest on a separate I/O pool off the apply pool, digest debounce, and count-derivation, plus a per-peer broadcast message clone that fixes a concurrent-map-write crash - migrates the link subsystem to slog and adds observability (ctrl.is_leader gauge, messaging pool meters, slow-handler diagnostic) - adds a links-test fablab model and gossip chaos/validation for soak testing

Own the metrics wire format in ziti. Fixes #4036 - adds a common/servermetrics package that owns the metrics MetricsMessage wire format and the reporting/usage subsystem (message builder, usage registry, interval and usage counters), wrapping the openziti/metrics Registry for metric collection - moves the controllers metrics reporter into the router package and removes it from the shared metrics package, breaking a common -> router/env import cycle - repoints controller and router consumers to common/servermetrics; base metric collection stays on openziti/metrics - keeps the proto field numbers and the metrics content-type identical so the encoding is byte-compatible across the move, and uses a distinct proto package name so ziti's and the library's messages coexist without a global proto registry clash - adds a round-trip test asserting wire compatibility with the library's MetricsMessage - leaves openziti/metrics unchanged, so sdk-golang and the shared xgress data plane are unaffected

2026-06-24

Enforce api-session and identity revocations on the router. Fixes #3929 Backport to release-v2.0.x of #3927 (main-line fix in PR #3930). The router's CheckConnections reaper enforced only JWT expiry, so a revoked OIDC api-session, or a disabled/deleted identity, kept its live circuits and hosted terminators until the access token expired. The router now enforces the RouterDataModel revocations directly, tightening access-loss propagation to the reaper interval. - adds a Type field to DataState_Revocation and the raft Revocation command proto, mirroring rest_model.RevocationTypeEnum (API_SESSION/IDENTITY/JTI) so the management API, OIDC producers, sync, and router enforcement share one vocabulary; the common.RevocationType* constants are compile-time bound to the enum to prevent drift - adds an IssuedBefore cutoff so an identity revocation invalidates only sessions issued before it; a session re-authenticated after the cutoff survives the still-lingering revocation. Persists IssuedBefore on the db and model Revocation and carries it (plus the Type) through the single and batched raft marshalling - adds RouterDataModel.IsApiSessionRevoked and IsIdentityRevoked and enforces both in CheckConnections, closing a revoked session's connections - revokes a deleted or disabled identity's live OIDC sessions via an IdentityRevocationConstraint in the db package, run as a store pre-commit constraint so the revocation is written in the same transaction as the identity change and cannot be skipped (self-contained OIDC JWTs aren't otherwise reachable). NewIdentityManager installs it with the revocation type and lifetime - has the OIDC end-session (TerminateSessionFromRequest) revoke the specific api-session named by the z_asid claim, with an identity-scoped fallback; sets IssuedBefore on the identity fallback and the management revocation API; adds RevocationManager.CreateOrReplace, routed through by both the OIDC paths and the management revocation API, so a repeat logout/termination/revocation refreshes the cutoff rather than colliding on the reused id. Expiry derives from the longest configured token duration via a shared common.MaxTokenDuration helper - adds tests/revocation_enforcement_oidc_test.go covering api-session revocation (and a fresh session staying unaffected), identity disable and delete, and the identity cutoff (a post-cutoff session surviving the lingering revocation) - regenerates edge_cmd.pb.go (protoc-gen-go v1.31.0) and edge_ctrl.pb.go (v1.36.11) against the release-v2.0.x base, matching each file's existing generator version

fixes openziti/ziti#3990 push service and posture changes to subscribed SDKs - routers push service change (ServiceChangeSet) and posture state (PostureStateChange) notifications directly to subscribed SDK connections, removing the controller from this path - adds RouterServiceSubscriptions and RouterDataModel capability bits and advertises them in the control-channel capabilities mask - persists each edge router's reported capabilities mask and version with a write-on-mismatch raft update, and renders them on the edge router APIs as a capability string list and version - normalizes a single PROCESS posture check to a one-element PROCESS_MULTI on the wire - closes accepted edge connections on router shutdown so SDKs detect a disconnect promptly rather than waiting for a heartbeat timeout - adds integration tests for capability reporting, service subscriptions, push/poll reconciliation, and posture-state push

Auto-GC stale router links. Fixes #4005 - adds a structured xlink.LinkKey ({DialerBinding, Protocol, DestId, ListenerBinding}) used by all staleness and GC checks - adds the CheckStaleLinks ctrl message and the `ziti ops verify stale-links` CLI for operator-driven, two-sided staleness verification with optional --gc - implements auto-GC via the router.link.v1 gcMode (preserve/orphaned/changed): the router walks its xlinks after each Apply that mutates listeners, dialers, or gcMode and closes one-sided-stale entries under the configured mode For #3743.

Manage router link configuration via the controller. Fixes #4004 - adds a link ConfigHandler (router/link FactoryRegistry) that applies router.link.v1 config: Apply rebuilds the listener/dialer set wholesale, and established Xlinks survive because Listener.Close() only closes the accept loop - translates local link: YAML into router.link.v1 JSON and pushes it through the managed-config registry at startup - adds the UpdateLinkListeners ctrl message so the router republishes its listener set to the controller on change; the controller re-fans via the existing PeerStateChange path - re-evaluates dialers on link group and listener changes via RescanForDialOpportunities For #3743.

2026-06-18

Auto-GC stale router links. Fixes #4005 - adds a structured xlink.LinkKey ({DialerBinding, Protocol, DestId, ListenerBinding}) used by all staleness and GC checks - adds the CheckStaleLinks ctrl message and the `ziti ops verify stale-links` CLI for operator-driven, two-sided staleness verification with optional --gc - implements auto-GC via the router.link.v1 gcMode (preserve/orphaned/changed): the router walks its xlinks after each Apply that mutates listeners, dialers, or gcMode and closes one-sided-stale entries under the configured mode For #3743.

Manage router link configuration via the controller. Fixes #4004 - adds a link ConfigHandler (router/link FactoryRegistry) that applies router.link.v1 config: Apply rebuilds the listener/dialer set wholesale, and established Xlinks survive because Listener.Close() only closes the accept loop - translates local link: YAML into router.link.v1 JSON and pushes it through the managed-config registry at startup - adds the UpdateLinkListeners ctrl message so the router republishes its listener set to the controller on change; the controller re-fans via the existing PeerStateChange path - re-evaluates dialers on link group and listener changes via RescanForDialOpportunities For #3743.

Use gossip to propagate link data between routers and controllers. Fixes #3726 Replaces the legacy link fault/refresh protocol with a gossip-based anti-entropy store so routers and controllers converge on link state. - adds the gossip store, anti-entropy, and link/canary gossip handlers on both controller and router - splits router connects into a separate pool from router gossip/canary events (routerEventsPool, peerEventsPool) so connect bursts can't starve the gossip hot path - tidies canary sender close handling - refactors gossip store data to be organized by owner - adds store metrics, host metrics, and drops router metrics on router delete - adds a per-owner FNV-64a hash digest to anti-entropy so matching owners short-circuit per-entry comparison, plus diagnostic instrumentation (slow-handler dumps, disk gauges, stale-send and iteration-mismatch logging) - fixes a cmap re-entry deadlock in the digest handler by collecting (owner, ownerData) pairs under iteration and hashing outside it - defers link registry side-effect events outside the registry lock (applyLinkLocked) to avoid piling dial/accept workers behind a back-pressured event channel

2026-06-17

Implement stale links checking

Handle changes to link groups and listeners by publishing listener changes and re-evaluating dialers

Distribute routers and router-target configs through the RDM. Fixes #3976 (#3977) - adds a `DataState.Router` event variant (id, name, fingerprint, configs) to the RDM protobuf and the matching ConfigType.Target field flows - loads routers into `RouterDataModelSender` at startup and registers entity-change listeners so router create/update/delete and config reassignment emit RDM events - filters `Config` events per-router at `RouterSender`: each router sees the full `Router` set but only its own router-target Configs - receiver-side: parses and stores `Router` entities; GCs router-target Configs locally when they drop off the router's `Configs` list, so no synthetic remove events are needed on the wire - extends the `router-data-model-test` fablab model with router-config distribution scenarios (assignment, reassignment, controller restart, RDM cache miss) - updates `validate-router-data-model` to recognize the new event shape - refreshes the design doc to describe per-router filtering and the change-notification flow

2026-06-13

Enforce api-session and identity revocations on the router. Fixes #3929 Backport to release-v2.0.x of #3927 (main-line fix in PR #3930). The router's CheckConnections reaper enforced only JWT expiry, so a revoked OIDC api-session, or a disabled/deleted identity, kept its live circuits and hosted terminators until the access token expired. The router now enforces the RouterDataModel revocations directly, tightening access-loss propagation to the reaper interval. - adds a Type field to DataState_Revocation and the raft Revocation command proto, mirroring rest_model.RevocationTypeEnum (API_SESSION/IDENTITY/JTI) so the management API, OIDC producers, sync, and router enforcement share one vocabulary; the common.RevocationType* constants are compile-time bound to the enum to prevent drift - adds an IssuedBefore cutoff so an identity revocation invalidates only sessions issued before it; a session re-authenticated after the cutoff survives the still-lingering revocation. Persists IssuedBefore on the db and model Revocation and carries it (plus the Type) through the single and batched raft marshalling - adds RouterDataModel.IsApiSessionRevoked and IsIdentityRevoked and enforces both in CheckConnections, closing a revoked session's connections - revokes a deleted or disabled identity's live OIDC sessions via an IdentityRevocationConstraint in the db package, run as a store pre-commit constraint so the revocation is written in the same transaction as the identity change and cannot be skipped (self-contained OIDC JWTs aren't otherwise reachable). NewIdentityManager installs it with the revocation type and lifetime - has the OIDC end-session (TerminateSessionFromRequest) revoke the specific api-session named by the z_asid claim, with an identity-scoped fallback; sets IssuedBefore on the identity fallback and the management revocation API; adds RevocationManager.CreateOrReplace, routed through by both the OIDC paths and the management revocation API, so a repeat logout/termination/revocation refreshes the cutoff rather than colliding on the reused id. Expiry derives from the longest configured token duration via a shared common.MaxTokenDuration helper - adds tests/revocation_enforcement_oidc_test.go covering api-session revocation (and a fresh session staying unaffected), identity disable and delete, and the identity cutoff (a post-cutoff session surviving the lingering revocation) - regenerates edge_cmd.pb.go (protoc-gen-go v1.31.0) and edge_ctrl.pb.go (v1.36.11) against the release-v2.0.x base, matching each file's existing generator version

Enforce api-session and identity revocations on the router. Fixes #3927 The router's CheckConnections reaper enforced only JWT expiry, so a revoked OIDC api-session, or a disabled/deleted identity, kept its live circuits and hosted terminators until the access token expired. The router now enforces the RouterDataModel revocations directly, tightening access-loss propagation to the reaper interval. - adds a Type field to DataState_Revocation and the raft Revocation command proto, mirroring rest_model.RevocationTypeEnum (API_SESSION/IDENTITY/JTI) so the management API, OIDC producers, sync, and router enforcement share one vocabulary; the common.RevocationType* constants are compile-time bound to the enum to prevent drift - adds an IssuedBefore cutoff so an identity revocation invalidates only sessions issued before it; a session re-authenticated after the cutoff survives the still-lingering revocation. Persists IssuedBefore on the db and model Revocation and carries it (plus the Type) through the single and batched raft marshalling - adds RouterDataModel.IsApiSessionRevoked and IsIdentityRevoked and enforces both in CheckConnections, closing a revoked session's connections - revokes a deleted or disabled identity's live OIDC sessions via an IdentityRevocationConstraint in the db package, run as a store pre-commit constraint so the revocation is written in the same transaction as the identity change and cannot be skipped (self-contained OIDC JWTs aren't otherwise reachable). NewIdentityManager installs it with the revocation type and lifetime - has the OIDC end-session (TerminateSessionFromRequest) revoke the specific api-session named by the z_asid claim, with an identity-scoped fallback; sets IssuedBefore on the identity fallback and the management revocation API; adds RevocationManager.CreateOrReplace, routed through by both the OIDC paths and the management revocation API, so a repeat logout/termination/revocation refreshes the cutoff rather than colliding on the reused id. Expiry derives from the longest configured token duration via a shared common.MaxTokenDuration helper - adds tests/revocation_enforcement_oidc_test.go covering api-session revocation (and a fresh session staying unaffected), identity disable and delete, and the identity cutoff (a post-cutoff session surviving the lingering revocation)

2026-06-11

Add configs field to routers. Fixes #3780

2026-05-27

Add target field to config types. Fixes #3744

2026-05-06

Update circuit-perf-diff test to report more metrics. Allow deploying ziti-traffic-test from git ref - adds stageziti.StageZitiTrafficTest / StageZitiTrafficTestOnce, which clone openziti/ziti, check out the requested ref (tag, branch, or commit SHA), and build zititest/ziti-traffic-test from source for linux/amd64; output is cached in the kit bin dir as ziti-traffic-test-<version> - extends StageZiti to build the ziti multi-tool from source via the same git-clone path when version is not a release tag (commit hash, branch name, "main"); semver-tagged versions still download the prebuilt GitHub release - introduces gitBuildSpec / buildFromGit in stageziti so the git-build pipeline is shared by both binaries; supports overriding the source repo via ZITI_TRAFFIC_TEST_REPO_URL (a local filesystem path works, letting developers build from a working tree) - adds Loop4SimType.Version: when set, the fablab run builds ziti-traffic-test from that ref before staging; empty preserves the legacy local-path behavior - relaxes canonicalizeGoAppVersion so bare commit hashes and branch names pass through unchanged and can flow into the git-build path, while bare semver still gets the leading "v" prefix; covered by a new unit test - shifts loop4.proto ContentType values from 1-3 to 1000-1002 to avoid collision with channel/v4 reserved types (Ping/Result/Latency/etc.), matching the convention used by ctrl_pb / edge_client_pb - collects xgress-level meters (retransmissions, retx failures, dup acks, dropped payloads, dup payloads, blocked-by-local/remote-window) per data-plane host and emits a "Network health" block in the per-pair report so flow-control regressions are visible alongside service throughput/latency - tags loop-host-xg with sim-services-client and adds a metrics connector block to its config so the listener can push its SDK xgress metrics into fablab via the metrics service - adds tc_actions.go registering netem-based scenarios (tcClear, tcJitter, tcLowLoss, tcHighLatency, tcHighRtt, ...) that apply tc qdiscs to the default interface of every data-plane host; auto- detects the interface so it works across AWS AMI variants without severing ssh on a mgmt NIC - hardens runFullComparison with consecutive-failure tracking: a single failed pair is logged and skipped after a best-effort stopDataPlane recovery, but maxConsecutivePairFailures (5) in a row aborts the run; protects 8-hour comparisons from a transient ssh blip on a single pair - replaces the old GOPATH-bin copy in stageBaselineTrafficTest with a StageZitiTrafficTestOnce call against the new git-build path, so baseline binaries are produced reproducibly from a pinned commit rather than whatever happens to be in GOPATH/bin

2026-04-14

fixes #3680 add revocation management API, CLI, and enforcement (#3789) * fixes #3680 add revocation management API, CLI, and enforcement - adds Management API endpoints for revocations (POST, GET, LIST) with type-aware validation (JTI/API_SESSION require UUID, IDENTITY requires existing identity) - adds CLI commands: ziti edge create revocation identity|api-session|jti - adds revocation checks to resolveOidcSession in security_ctx.go so the REST API returns 401 for revoked OIDC tokens. Previously only ValidateAccessToken (router ctrl channel path) checked revocations, so revoked tokens still received 200 OK from the management and client HTTP APIs - adds api-session revocation check to ValidateAccessToken, which only checked JTI and identity revocations - adds Type field to Revocation model, store, and protobuf message - adds integration tests covering CRUD, input validation, and token enforcement for all three revocation types - use release edge-api@v0.28.1

2026-04-03

Coalesce OIDC JWT revocations to reduce controller write pressure. Fixes #3764 - adds DeleteRevocationsBatchCommand so expired-revocation cleanup goes through raft as a single log entry per batch - adds CreateRevocationsBatchCommand for batched revocation creation through raft - moves refresh-token revocations from synchronous inline creation to a background batcher that flushes on a configurable interval, removing the database and raft as a bottleneck on token refreshes - skips revocation creation for tokens expiring within a configurable threshold (revocationMinTokenLifetime), since they become invalid on their own - validates that revocationMinTokenLifetime is less than 50% of the configured refresh token lifetime - makes the revocation enforcer frequency configurable and restricts it to run only on the raft leader - adds tests for multi-batch delete, batched create with router RDM propagation, and skip-threshold behavior - adds new configuration tunables under edge.oidc: revocationBucketInterval, revocationMinTokenLifetime, revocationBucketMaxSize, revocationMaxQueued, revocationEnforcerFrequency

2026-04-01

Add initial POC work on federated routers