Proto commits in openziti/ziti

These commits are when the Protocol Buffers files have changed: (only the last 100 relevant commits are shown)

Commit:c79f98a
Author:Paul Lorenz
Committer:Paul Lorenz

Record links to destinations this controller cannot yet judge, and settle them once it can A controller that has not caught up on raft cannot tell a destination router that was deleted from one it has not applied yet, and the two want opposite answers. Refusing the report drops a link to a router that does exist, and the reporting router considers it delivered, so nothing ever says it again. One controller ended a soak run 125 links short of its peers that way. - records a report from a router that does not gossip with its destination left unresolved, since the link table is its only copy; an unresolved link is never selected for a path - removes those links once their destination is still absent under a read barrier requested after the destination was first named, and tells the dialer its destination is gone, retrying the notice until it can be sent - keeps that time per destination, the earliest arrival naming it, so reports naming other destinations, or repeats naming the same one, cannot keep a barrier from settling it, and refuses reports naming a settled destination as they arrive - forgets a settled destination some minutes after it was first named; anything still naming it notes it again and waits for another barrier - adds an asynchronous raft read barrier: a follower asks the leader, which commits a raft barrier and answers with its store index, and the leader answers its own requests the same way; requests carry an id, so only the outstanding request's answer is accepted - leaves a gossiped entry naming an unknown destination in the store and asks about it again on the network cycle, refusing it once a barrier covers its destination, and answers a proved absence with a removal notice to its owner - re-reads a destination from the store after publishing a link to it, so a link published while its destination is being deleted is removed rather than outliving the delete's cleanup - retires a deleted router's links by scanning the link table rather than the router's index, which anything finding the router gone may already have dropped - renders a link whose source router cannot be resolved without a source rather than failing on it, since a link records its destination's id but not its source's - renames the peer-state index to say what it is: the raft index the sending controller had applied, not a store index

Commit:6f3cc7e
Author:Paul Lorenz
Committer:Paul Lorenz

Replicate link state over gossip. Fixes #3726 Link state was every controller's own view, built from what routers reported to it directly. A router announces its links once per reconnect, so a controller that missed the announcement had no way to ask again, and controllers in a cluster disagreed about which links existed until something forced a reconnect. - publishes a router's dialed links into the gossip store, keyed by link id and iteration and owned by the dialer, so every controller converges on the same set without the router telling each one - builds links from gossip entries on arrival, and re-walks the owner's entries when a router connects, which covers entries applied while the router was unknown - keeps a gossiped link across its dialer's disconnect from this controller, with or without raft, since the dialer may still be connected to another controller and gossip settles the link once it is not; its circuits still move off it - retires a link by tombstoning its entry, so the removal reaches every controller rather than only the one that heard the fault, and forwards the fault to whichever endpoint is connected here - counts fault forwards dropped for a full I/O pool rather than logging each one, since the forward is queued under the entry's notification lock, which every later change to that entry waits for - sends the fault for a link closed after its state is gone, as when its destination is removed or its dial state was detached, through the notification cycle rather than directly, so it is ordered after the live entry it retires - keeps a router's reports out of the store while it is not a publisher, and retracts what it published when it stops being one - holds a link's source router atomically, since a link built from a gossiped entry carries a database-loaded router until its real connection arrives - refuses a report naming an existing link id from a router that is not its source, which would otherwise let any router take over a link between two others by reporting a higher iteration - answers a refused entry with a removal notice to its dialer, which is the only thing that stops the dialer re-announcing it

Commit:246c4c0
Author:Paul Lorenz
Committer:Paul Lorenz

Check a router's gossip state against the controller's with a canary Anti-entropy repairs divergence it is told about, but nothing noticed when a controller's view of a router's entries had drifted. The canary is that detector: a sequence the router advances and the controller echoes back, so either side can see the other is behind without exchanging the entries themselves. The canary also carries what the router reports about itself, so the raft leader can persist each router's version and capabilities from one ordered source rather than from every controller's hello. - adds a canary state type carrying each router's monotonic sequence, versioned from the router rather than the controller clock, so every controller writing it uses the same version space and a stale update is rejected on version wherever it lands - emits the sequence from the router with the hashes and counts of what it holds, and answers with the controller's view, so a mismatch is visible to both ends - stamps each router lifetime with an epoch, so entries from a previous lifetime can be told from current ones after a restart resets the sequence - drops a router's old-epoch entries across every store type when its epoch changes, read from the hello as well as the canary so the cleanup does not wait for the first sequence - keeps canary state out of the epoch sweep, since the canary is what carries the epoch that detects the change - counts canaries dropped for a full apply pool with the other gossip drops, rather than logging each one - carries the router's version and capabilities on the canary, and retries a router's first canary every second until it goes out, so a restart's report replaces the previous lifetime's promptly - persists each edge router's reported version and capabilities from the caught-up raft leader only, one router at a time, and stops each controller's edge hello writing them from the control channel receive goroutine - judges a router with link gossip by its canary, which reaches the leader whether or not the router is connected to it, and a router without it, or any router while the cluster cannot carry canaries, by the leader's own connection - re-checks a router when its canary reports a change or it connects to or disconnects from the leader, and every router on taking leadership and every five minutes, keeping a router whose write fails marked for the next pass

Commit:1d39c0c
Author:Paul Lorenz
Committer:Paul Lorenz

Carry gossip between controllers and routers over the control channel - adds the router-side gossip client and the controller handlers for deltas, digest requests and digest responses, so a state type has a wire to travel on - opens with a digest to each router on connect, so a reconnecting router reconciles what it holds against the controller rather than waiting for the next change - hash-gates the digest exchange per store type, replying in full only for the types whose hash differs, since the steady state is that they agree - tells a router which capabilities the whole cluster supports, as distinct from what the controllers it happens to be connected to support; a feature needs both answers before it is used - lifts delta application off the receive goroutine onto its own pool, and gives peer events and controller I/O their own pools, so one slow consumer cannot stall the others - counts the work the apply and I/O pools drop when full, on both sides, in a meter per kind of work, and logs a summary of each once a minute rather than a warning per drop, since the overload that fills a pool fills the log queue too and a warning waits for room in it - registers no state type: what travels over this is added by the commits that follow

Commit:c477418
Author:Paul Lorenz
Committer:Paul Lorenz

Add a gossip protocol and replicated state store Link state was every controller's own view, built from what routers reported to it directly. A router announces its links once per reconnect, so a controller that missed the announcement had no way to ask again, and controllers in a cluster disagreed about which links existed until something forced a reconnect. Replicating state between controllers is what this makes possible; nothing uses it yet. - adds a peer-to-peer state store that replicates keyed, owner-scoped entries between controllers, with a Lamport clock for ordering, tombstones for removals, and per-type listeners for consumers - repairs missed updates with an anti-entropy round that exchanges a per-owner digest with a rotating peer, so divergence is corrected without re-sending state that already agrees - derives which capabilities every member of the raft cluster supports, as the intersection of what each last advertised, and withholds an answer until every member has been observed, since a member nothing has been heard from could be running anything - gates each state type on its own capability, so a cluster part-way through an upgrade keeps using what it can carry rather than giving up everything - gives up a type's entries when its capability stops holding, telling listeners the entries are gone without claiming that what they described is gone - documents the protocol, the message sequences and the patterns a state type is expected to follow - documents that listeners run holding the entry's notification lock, so they must not block, which includes logging a warning per entry on a path that load makes frequent

Commit:cbfd3fe
Author:Paul Lorenz
Committer:Paul Lorenz

Order peer state across controllers by the store index it rests on. For #4669 Controllers compute peer state independently and nothing orders their messages against each other, so a router could be told a destination was removed and then told by another controller, from state computed before the removal, that it was healthy again. The destination came back and the router went on dialing a router that was gone. - carries the controller store index a peer state was computed against, and zero where the reporter has none to offer - remembers a removal against its index, so a report computed against an older one cannot recreate the destination - treats a report with no index as older than any indexed removal, since it cannot be ordered against one: a controller from before store indexes sends none, and during a rolling upgrade one it computed before the delete can arrive after the removal - loses a tie: a router being deleted stays in a controller's connected map until its control channel teardown runs, so the same store state can still produce a healthy report - suppresses only existence, never health, since health is each controller's own view of what it is connected to and a controller lagging in replication reports a low index alongside a perfectly good health report - names the reporting controller on each update, so the registry can union health across controllers while ordering existence by index - remembers a removal for two days, since a controller computes peer state against its own store, so one lagging in replication reports from before the removal for as long as it lags; past that, it removes the destination again once it applies the delete

Commit:4da6c79
Author:Paul Lorenz
Committer:Paul Lorenz

Stop routers dropping controllers on a follower's stale controller list. Fixes #4574 - applies controller lists, leader announcements, controller address changes and index resets on one goroutine in arrival order, so checking an update's index, applying it and recording it is one step and no control channel's receive goroutine waits on another controller's update - lets only a leader's list at a newer index replace the controller set; any other list at or above the recorded version only adds, merging into the router's stored details and connected controllers - applies a leader's list at the recorded version as additions, so a router missing controllers recovers when the leader resends - keeps an inbound controller's socket address out of the saved endpoints, and fills in a saved controller that has no endpoints when a later list gives some - orders leadership claims by raft term: adds a raftTerm field to UpdateCtrlAddresses and UpdateClusterLeader, which controllers read together with leadership so both describe the same raft term; claims without one, from older controllers, still apply as they arrive - folds the cluster leader handler into the same worker - asks every connected controller for its list while waiting for the leader's, without waiting on a full send queue, one request run at a time, stopping when the router closes - bounds the waiting updates: past the bound it drops waiting lists and announcements, keeps each controller's latest address change, and requests the leader's list - fixes a router crash applying a controller's new address before any controller details were stored, and keeps connected controllers the router has no details for - stops clustered controllers sending the newListener address, which the cluster's controller lists would undo, warns when it is configured, and notes this in the sample configs - tests list handling, leadership ordering, overflow, the request loop, address changes and the controller's raft term read

Commit:942d1a7
Author:Paul Lorenz
Committer:Paul Lorenz

Stop routers dropping controllers on a follower's stale controller list. Fixes #4573 - applies controller lists, leader announcements, controller address changes and index resets on one goroutine in arrival order, so checking an update's index, applying it and recording it is one step and no control channel's receive goroutine waits on another controller's update - lets only a leader's list at a newer index replace the controller set; any other list at or above the recorded version only adds, merging into the router's stored details and connected controllers - applies a leader's list at the recorded version as additions, so a router missing controllers recovers when the leader resends - keeps an inbound controller's socket address out of the saved endpoints, and fills in a saved controller that has no endpoints when a later list gives some - orders leadership claims by raft term: adds a raftTerm field to UpdateCtrlAddresses and UpdateClusterLeader, which controllers read together with leadership so both describe the same raft term; claims without one, from older controllers, still apply as they arrive - folds the cluster leader handler into the same worker - asks every connected controller for its list while waiting for the leader's, without waiting on a full send queue, one request run at a time, stopping when the router closes - bounds the waiting updates: past the bound it drops waiting lists and announcements, keeps each controller's latest address change, and requests the leader's list - fixes a router crash applying a controller's new address before any controller details were stored, and keeps connected controllers the router has no details for - stops clustered controllers sending the newListener address, which the cluster's controller lists would undo, warns when it is configured, and notes this in the sample configs - tests list handling, leadership ordering, overflow, the request loop, address changes and the controller's raft term read - notes the fix in the changelog

Commit:ae36719
Author:Paul Lorenz
Committer:GitHub

Decide delete existence in Apply and add DeleteIfExists. Fixes #4571 - checks the deleting manager's own store for the target inside the delete transaction, before DeleteById, so a child store delete never reaches the parent's record of another type - moves the edge service manager's fabric-only check from a pre-dispatch read into an ApplyDelete override - makes IsEntityPresent on an extended child store match FindById, so a router created through the fabric is deletable through the transit router manager, which excludes edge routers explicitly - adds a flags field to the delete command and DeleteIfExists on the base manager, under which an absent target is a no-op - routes identity deletes through the base ApplyDelete after the admin precondition, so identities get the presence check and DeleteIfExists - gives the API session, session and API session certificate managers a direct DeleteIfExists, since they register no delete decoder - tests cross-type router deletes, the fabric-only check at apply time, base router deletes, and DeleteIfExists on present, absent, fabric-only, wrong-type, identity and local-only targets, at the model level and through the management API

Commit:02bdbd2
Author:Paul Lorenz
Committer:Paul Lorenz

Record links to destinations this controller cannot yet judge, and settle them once it can A controller that has not caught up on raft cannot tell a destination router that was deleted from one it has not applied yet, and the two want opposite answers. Refusing the report drops a link to a router that does exist, and the reporting router considers it delivered, so nothing ever says it again. One controller ended a soak run 125 links short of its peers that way. - records a report from a router that does not gossip with its destination left unresolved, since the link table is its only copy; an unresolved link is never selected for a path - removes those links once their destination is still absent under a read barrier requested after the destination was first named, and tells the dialer its destination is gone, retrying the notice until it can be sent - keeps that time per destination, the earliest arrival naming it, so reports naming other destinations, or repeats naming the same one, cannot keep a barrier from settling it, and refuses reports naming a settled destination as they arrive - forgets a settled destination some minutes after it was first named; anything still naming it notes it again and waits for another barrier - adds an asynchronous raft read barrier: a follower asks the leader, which commits a raft barrier and answers with its store index, and the leader answers its own requests the same way; requests carry an id, so only the outstanding request's answer is accepted - leaves a gossiped entry naming an unknown destination in the store and asks about it again on the network cycle, refusing it once a barrier covers its destination, and answers a proved absence with a removal notice to its owner - re-reads a destination from the store after publishing a link to it, so a link published while its destination is being deleted is removed rather than outliving the delete's cleanup - retires a deleted router's links by scanning the link table rather than the router's index, which anything finding the router gone may already have dropped - renders a link whose destination cannot be resolved by its id rather than failing the whole link listing - renames the peer-state index to say what it is: the raft index the sending controller had applied, not a store index

Commit:0e7d3f1
Author:Paul Lorenz
Committer:Paul Lorenz

Replicate link state over gossip. Fixes #3726 Link state was every controller's own view, built from what routers reported to it directly. A router announces its links once per reconnect, so a controller that missed the announcement had no way to ask again, and controllers in a cluster disagreed about which links existed until something forced a reconnect. - publishes a router's dialed links into the gossip store, keyed by link id and iteration and owned by the dialer, so every controller converges on the same set without the router telling each one - builds links from gossip entries on arrival, and re-walks the owner's entries when a router connects, which covers entries applied while the router was unknown - keeps a gossiped link across its dialer's disconnect from this controller, with or without raft, since the dialer may still be connected to another controller and gossip settles the link once it is not; its circuits still move off it - retires a link by tombstoning its entry, so the removal reaches every controller rather than only the one that heard the fault, and forwards the fault to whichever endpoint is connected here - counts fault forwards dropped for a full I/O pool rather than logging each one, since the forward is queued under the entry's notification lock, which every later change to that entry waits for - sends the fault for a link closed after its state is gone, as when its destination is removed or its dial state was detached, through the notification cycle rather than directly, so it is ordered after the live entry it retires - keeps a router's reports out of the store while it is not a publisher, and retracts what it published when it stops being one - holds a link's source router atomically, since a link built from a gossiped entry carries a database-loaded router until its real connection arrives - refuses a report naming an existing link id from a router that is not its source, which would otherwise let any router take over a link between two others by reporting a higher iteration - answers a refused entry with a removal notice to its dialer, which is the only thing that stops the dialer re-announcing it

Commit:8b929ce
Author:Paul Lorenz
Committer:Paul Lorenz

Check a router's gossip state against the controller's with a canary Anti-entropy repairs divergence it is told about, but nothing noticed when a controller's view of a router's entries had drifted. The canary is that detector: a sequence the router advances and the controller echoes back, so either side can see the other is behind without exchanging the entries themselves. The canary also carries what the router reports about itself, so the raft leader can persist each router's version and capabilities from one ordered source rather than from every controller's hello. - adds a canary state type carrying each router's monotonic sequence, versioned from the router rather than the controller clock, so every controller writing it uses the same version space and a stale update is rejected on version wherever it lands - emits the sequence from the router with the hashes and counts of what it holds, and answers with the controller's view, so a mismatch is visible to both ends - stamps each router lifetime with an epoch, so entries from a previous lifetime can be told from current ones after a restart resets the sequence - drops a router's old-epoch entries across every store type when its epoch changes, read from the hello as well as the canary so the cleanup does not wait for the first sequence - keeps canary state out of the epoch sweep, since the canary is what carries the epoch that detects the change - counts canaries dropped for a full apply pool with the other gossip drops, rather than logging each one - carries the router's version and capabilities on the canary, and retries a router's first canary every second until it goes out, so a restart's report replaces the previous lifetime's promptly - persists each edge router's reported version and capabilities from the caught-up raft leader only, one router at a time, and stops each controller's edge hello writing them from the control channel receive goroutine - judges a router with link gossip by its canary, which reaches the leader whether or not the router is connected to it, and a router without it, or any router while the cluster cannot carry canaries, by the leader's own connection - re-checks a router when its canary reports a change or it connects to or disconnects from the leader, and every router on taking leadership and every five minutes, keeping a router whose write fails marked for the next pass

Commit:2ba4078
Author:Paul Lorenz
Committer:Paul Lorenz

Carry gossip between controllers and routers over the control channel - adds the router-side gossip client and the controller handlers for deltas, digest requests and digest responses, so a state type has a wire to travel on - opens with a digest to each router on connect, so a reconnecting router reconciles what it holds against the controller rather than waiting for the next change - hash-gates the digest exchange per store type, replying in full only for the types whose hash differs, since the steady state is that they agree - tells a router which capabilities the whole cluster supports, as distinct from what the controllers it happens to be connected to support; a feature needs both answers before it is used - lifts delta application off the receive goroutine onto its own pool, and gives peer events and controller I/O their own pools, so one slow consumer cannot stall the others - counts the work the apply and I/O pools drop when full, on both sides, in a meter per kind of work, and logs a summary of each once a minute rather than a warning per drop, since the overload that fills a pool fills the log queue too and a warning waits for room in it - registers no state type: what travels over this is added by the commits that follow

Commit:12b7900
Author:Paul Lorenz
Committer:Paul Lorenz

Add a gossip protocol and replicated state store Link state was every controller's own view, built from what routers reported to it directly. A router announces its links once per reconnect, so a controller that missed the announcement had no way to ask again, and controllers in a cluster disagreed about which links existed until something forced a reconnect. Replicating state between controllers is what this makes possible; nothing uses it yet. - adds a peer-to-peer state store that replicates keyed, owner-scoped entries between controllers, with a Lamport clock for ordering, tombstones for removals, and per-type listeners for consumers - repairs missed updates with an anti-entropy round that exchanges a per-owner digest with a rotating peer, so divergence is corrected without re-sending state that already agrees - derives which capabilities every member of the raft cluster supports, as the intersection of what each last advertised, and withholds an answer until every member has been observed, since a member nothing has been heard from could be running anything - gates each state type on its own capability, so a cluster part-way through an upgrade keeps using what it can carry rather than giving up everything - gives up a type's entries when its capability stops holding, telling listeners the entries are gone without claiming that what they described is gone - documents the protocol, the message sequences and the patterns a state type is expected to follow - documents that listeners run holding the entry's notification lock, so they must not block, which includes logging a warning per entry on a path that load makes frequent

Commit:3d62517
Author:Paul Lorenz
Committer:Paul Lorenz

Order peer state across controllers by the store index it rests on. For #4669 Controllers compute peer state independently and nothing orders their messages against each other, so a router could be told a destination was removed and then told by another controller, from state computed before the removal, that it was healthy again. The destination came back and the router went on dialing a router that was gone. - carries the controller store index a peer state was computed against, and zero where the reporter has none to offer - remembers a removal against its index, so a report computed against an older one cannot recreate the destination - loses a tie: a router being deleted stays in a controller's connected map until its control channel teardown runs, so the same store state can still produce a healthy report - suppresses only existence, never health, since health is each controller's own view of what it is connected to and a controller lagging in replication reports a low index alongside a perfectly good health report - names the reporting controller on each update, so the registry can union health across controllers while ordering existence by index - forgets a removal after a few minutes, which outlives any report already in flight

Commit:8adcebe
Author:Paul Lorenz
Committer:Paul Lorenz

Stop routers dropping controllers on a follower's stale controller list. Fixes #4573 - applies controller lists, leader announcements, controller address changes and index resets on one goroutine in arrival order, so checking an update's index, applying it and recording it is one step and no control channel's receive goroutine waits on another controller's update - lets only a leader's list at a newer index replace the controller set; any other list at or above the recorded version only adds, merging into the router's stored details and connected controllers - applies a leader's list at the recorded version as additions, so a router missing controllers recovers when the leader resends - keeps an inbound controller's socket address out of the saved endpoints, and fills in a saved controller that has no endpoints when a later list gives some - orders leadership claims by raft term: adds a term to UpdateCtrlAddresses and UpdateClusterLeader, which controllers read together with leadership so both describe the same term; claims without a term, from older controllers, still apply as they arrive - folds the cluster leader handler into the same worker - asks every connected controller for its list while waiting for the leader's, without waiting on a full send queue, one request run at a time, stopping when the router closes - bounds the waiting updates: past the bound it drops waiting lists and announcements, keeps each controller's latest address change, and requests the leader's list - fixes a router crash applying a controller's new address before any controller details were stored, and keeps connected controllers the router has no details for - stops clustered controllers sending the newListener address, which the cluster's controller lists would undo, warns when it is configured, and notes this in the sample configs - tests list handling, leadership ordering, overflow, the request loop, address changes and the controller's term read - notes the fix in the changelog

Commit:dfba224
Author:Paul Lorenz
Committer:Paul Lorenz

Record links to destinations this controller cannot yet judge, and settle them once it can A controller that has not caught up on raft cannot tell a destination router that was deleted from one it has not applied yet, and the two want opposite answers. Refusing the report drops a link to a router that does exist, and the reporting router considers it delivered, so nothing ever says it again. One controller ended a soak run 125 links short of its peers that way. - records a report from a router that does not gossip with its destination left unresolved, since the link table is its only copy; an unresolved link is never selected for a path - removes those links once their destination is still absent under a read barrier requested after the destination was first named, and tells the dialer its destination is gone, retrying the notice until it can be sent - keeps that time per destination, the earliest arrival naming it, so reports naming other destinations, or repeats naming the same one, cannot keep a barrier from settling it, and refuses reports naming a settled destination as they arrive - forgets a settled destination some minutes after it was first named; anything still naming it notes it again and waits for another barrier - adds an asynchronous raft read barrier: a follower asks the leader, which commits a raft barrier and answers with its store index, and the leader answers its own requests the same way; requests carry an id, so only the outstanding request's answer is accepted - leaves a gossiped entry naming an unknown destination in the store and asks about it again on the network cycle, refusing it once a barrier covers its destination, and answers a proved absence with a removal notice to its owner - re-reads a destination from the store after publishing a link to it, so a link published while its destination is being deleted is removed rather than outliving the delete's cleanup - retires a deleted router's links by scanning the link table rather than the router's index, which anything finding the router gone may already have dropped - renders a link whose destination cannot be resolved by its id rather than failing the whole link listing - renames the peer-state index to say what it is: the raft index the sending controller had applied, not a store index

Commit:6925be3
Author:Paul Lorenz
Committer:Paul Lorenz

Replicate link state over gossip. Fixes #3726 Link state was every controller's own view, built from what routers reported to it directly. A router announces its links once per reconnect, so a controller that missed the announcement had no way to ask again, and controllers in a cluster disagreed about which links existed until something forced a reconnect. - publishes a router's dialed links into the gossip store, keyed by link id and iteration and owned by the dialer, so every controller converges on the same set without the router telling each one - builds links from gossip entries on arrival, and re-walks the owner's entries when a router connects, which covers entries applied while the router was unknown - retires a link by tombstoning its entry, so the removal reaches every controller rather than only the one that heard the fault, and forwards the fault to whichever endpoint is connected here - counts fault forwards dropped for a full I/O pool rather than logging each one, since the forward is queued under the entry's notification lock, which every later change to that entry waits for - sends the fault for a link closed after its state is gone, as when its destination is removed or its dial state was detached, through the notification cycle rather than directly, so it is ordered after the live entry it retires - keeps a router's reports out of the store while it is not a publisher, and retracts what it published when it stops being one - holds a link's source router atomically, since a link built from a gossiped entry carries a database-loaded router until its real connection arrives - refuses a report naming an existing link id from a router that is not its source, which would otherwise let any router take over a link between two others by reporting a higher iteration - answers a refused entry with a removal notice to its dialer, which is the only thing that stops the dialer re-announcing it

Commit:3b2fdf2
Author:Paul Lorenz
Committer:Paul Lorenz

Check a router's gossip state against the controller's with a canary Anti-entropy repairs divergence it is told about, but nothing noticed when a controller's view of a router's entries had drifted. The canary is that detector: a sequence the router advances and the controller echoes back, so either side can see the other is behind without exchanging the entries themselves. The canary also carries what the router reports about itself, so the raft leader can persist each router's version and capabilities from one ordered source rather than from every controller's hello. - adds a canary state type carrying each router's monotonic sequence, versioned from the router rather than the controller clock, so every controller writing it uses the same version space and a stale update is rejected on version wherever it lands - emits the sequence from the router with the hashes and counts of what it holds, and answers with the controller's view, so a mismatch is visible to both ends - stamps each router lifetime with an epoch, so entries from a previous lifetime can be told from current ones after a restart resets the sequence - drops a router's old-epoch entries across every store type when its epoch changes, read from the hello as well as the canary so the cleanup does not wait for the first sequence - keeps canary state out of the epoch sweep, since the canary is what carries the epoch that detects the change - counts canaries dropped for a full apply pool with the other gossip drops, rather than logging each one - carries the router's version and capabilities on the canary, and retries a router's first canary every second until it goes out, so a restart's report replaces the previous lifetime's promptly - persists each edge router's reported version and capabilities from the caught-up raft leader only, one router at a time, and stops each controller's edge hello writing them from the control channel receive goroutine - judges a router with link gossip by its canary, which reaches the leader whether or not the router is connected to it, and a router without it, or any router while the cluster cannot carry canaries, by the leader's own connection - re-checks a router when its canary reports a change or it connects to or disconnects from the leader, and every router on taking leadership and every five minutes, keeping a router whose write fails marked for the next pass

Commit:e3f8de6
Author:Paul Lorenz
Committer:Paul Lorenz

Carry gossip between controllers and routers over the control channel - adds the router-side gossip client and the controller handlers for deltas, digest requests and digest responses, so a state type has a wire to travel on - opens with a digest to each router on connect, so a reconnecting router reconciles what it holds against the controller rather than waiting for the next change - hash-gates the digest exchange per store type, replying in full only for the types whose hash differs, since the steady state is that they agree - tells a router which capabilities the whole cluster supports, as distinct from what the controllers it happens to be connected to support; a feature needs both answers before it is used - lifts delta application off the receive goroutine onto its own pool, and gives peer events and controller I/O their own pools, so one slow consumer cannot stall the others - counts the work the apply and I/O pools drop when full, on both sides, in a meter per kind of work, and logs a summary of each once a minute rather than a warning per drop, since the overload that fills a pool fills the log queue too and a warning waits for room in it - registers no state type: what travels over this is added by the commits that follow

Commit:08cd7d5
Author:Paul Lorenz
Committer:Paul Lorenz

Add a gossip protocol and replicated state store Link state was every controller's own view, built from what routers reported to it directly. A router announces its links once per reconnect, so a controller that missed the announcement had no way to ask again, and controllers in a cluster disagreed about which links existed until something forced a reconnect. Replicating state between controllers is what this makes possible; nothing uses it yet. - adds a peer-to-peer state store that replicates keyed, owner-scoped entries between controllers, with a Lamport clock for ordering, tombstones for removals, and per-type listeners for consumers - repairs missed updates with an anti-entropy round that exchanges a per-owner digest with a rotating peer, so divergence is corrected without re-sending state that already agrees - derives which capabilities every member of the raft cluster supports, as the intersection of what each last advertised, and withholds an answer until every member has been observed, since a member nothing has been heard from could be running anything - gates each state type on its own capability, so a cluster part-way through an upgrade keeps using what it can carry rather than giving up everything - gives up a type's entries when its capability stops holding, telling listeners the entries are gone without claiming that what they described is gone - documents the protocol, the message sequences and the patterns a state type is expected to follow - documents that listeners run holding the entry's notification lock, so they must not block, which includes logging a warning per entry on a path that load makes frequent

Commit:3c593d8
Author:Paul Lorenz
Committer:Paul Lorenz

Order peer state across controllers by the store index it rests on. For #4669 Controllers compute peer state independently and nothing orders their messages against each other, so a router could be told a destination was removed and then told by another controller, from state computed before the removal, that it was healthy again. The destination came back and the router went on dialing a router that was gone. - carries the controller store index a peer state was computed against, and zero where the reporter has none to offer - remembers a removal against its index, so a report computed against an older one cannot recreate the destination - loses a tie: a router being deleted stays in a controller's connected map until its control channel teardown runs, so the same store state can still produce a healthy report - suppresses only existence, never health, since health is each controller's own view of what it is connected to and a controller lagging in replication reports a low index alongside a perfectly good health report - names the reporting controller on each update, so the registry can union health across controllers while ordering existence by index - forgets a removal after a few minutes, which outlives any report already in flight

Commit:dc54137
Author:Paul Lorenz
Committer:Paul Lorenz

Record links to destinations this controller cannot yet judge, and settle them once it can A controller that has not caught up on raft cannot tell a destination router that was deleted from one it has not applied yet, and the two want opposite answers. Refusing the report drops a link to a router that does exist, and the reporting router considers it delivered, so nothing ever says it again. One controller ended a soak run 125 links short of its peers that way. - records a report from a router that does not gossip with its destination left unresolved, since the link table is its only copy; an unresolved link is never selected for a path - removes those links once their destination is still absent under a read barrier requested after the destination was first named, and tells the dialer its destination is gone, retrying the notice until it can be sent - keeps that time per destination, the earliest arrival naming it, so reports naming other destinations, or repeats naming the same one, cannot keep a barrier from settling it, and refuses reports naming a settled destination as they arrive - forgets a settled destination some minutes after it was first named; anything still naming it notes it again and waits for another barrier - adds an asynchronous raft read barrier: a follower asks the leader, which commits a raft barrier and answers with its store index, and the leader answers its own requests the same way; requests carry an id, so only the outstanding request's answer is accepted - leaves a gossiped entry naming an unknown destination in the store and asks about it again on the network cycle, refusing it once a barrier covers its destination, and answers a proved absence with a removal notice to its owner - re-reads a destination from the store after publishing a link to it, so a link published while its destination is being deleted is removed rather than outliving the delete's cleanup - retires a deleted router's links by scanning the link table rather than the router's index, which anything finding the router gone may already have dropped - renders a link whose destination cannot be resolved by its id rather than failing the whole link listing - renames the peer-state index to say what it is: the raft index the sending controller had applied, not a store index

Commit:2eefec9
Author:Paul Lorenz
Committer:Paul Lorenz

Replicate link state over gossip. Fixes #3726 Link state was every controller's own view, built from what routers reported to it directly. A router announces its links once per reconnect, so a controller that missed the announcement had no way to ask again, and controllers in a cluster disagreed about which links existed until something forced a reconnect. - publishes a router's dialed links into the gossip store, keyed by link id and iteration and owned by the dialer, so every controller converges on the same set without the router telling each one - builds links from gossip entries on arrival, and re-walks the owner's entries when a router connects, which covers entries applied while the router was unknown - retires a link by tombstoning its entry, so the removal reaches every controller rather than only the one that heard the fault, and forwards the fault to whichever endpoint is connected here - counts fault forwards dropped for a full I/O pool rather than logging each one, since the forward is queued under the entry's notification lock, which every later change to that entry waits for - sends the fault for a link closed after its state is gone, as when its destination is removed or its dial state was detached, through the notification cycle rather than directly, so it is ordered after the live entry it retires - keeps a router's reports out of the store while it is not a publisher, and retracts what it published when it stops being one - holds a link's source router atomically, since a link built from a gossiped entry carries a database-loaded router until its real connection arrives - refuses a report naming an existing link id from a router that is not its source, which would otherwise let any router take over a link between two others by reporting a higher iteration - answers a refused entry with a removal notice to its dialer, which is the only thing that stops the dialer re-announcing it

Commit:c21a46f
Author:Paul Lorenz
Committer:Paul Lorenz

Check a router's gossip state against the controller's with a canary Anti-entropy repairs divergence it is told about, but nothing noticed when a controller's view of a router's entries had drifted. The canary is that detector: a sequence the router advances and the controller echoes back, so either side can see the other is behind without exchanging the entries themselves. The canary also carries what the router reports about itself, so the raft leader can persist each router's version and capabilities from one ordered source rather than from every controller's hello. - adds a canary state type carrying each router's monotonic sequence, versioned from the router rather than the controller clock, so every controller writing it uses the same version space and a stale update is rejected on version wherever it lands - emits the sequence from the router with the hashes and counts of what it holds, and answers with the controller's view, so a mismatch is visible to both ends - stamps each router lifetime with an epoch, so entries from a previous lifetime can be told from current ones after a restart resets the sequence - drops a router's old-epoch entries across every store type when its epoch changes, read from the hello as well as the canary so the cleanup does not wait for the first sequence - keeps canary state out of the epoch sweep, since the canary is what carries the epoch that detects the change - counts canaries dropped for a full apply pool with the other gossip drops, rather than logging each one - carries the router's version and capabilities on the canary, and retries a router's first canary every second until it goes out, so a restart's report replaces the previous lifetime's promptly - persists each edge router's reported version and capabilities from the caught-up raft leader only, one router at a time, and stops each controller's edge hello writing them from the control channel receive goroutine - judges a router with link gossip by its canary, which reaches the leader whether or not the router is connected to it, and a router without it, or any router while the cluster cannot carry canaries, by the leader's own connection - re-checks a router when its canary reports a change or it connects to or disconnects from the leader, and every router on taking leadership and every five minutes, keeping a router whose write fails marked for the next pass

Commit:aaeabeb
Author:Paul Lorenz
Committer:Paul Lorenz

Carry gossip between controllers and routers over the control channel - adds the router-side gossip client and the controller handlers for deltas, digest requests and digest responses, so a state type has a wire to travel on - opens with a digest to each router on connect, so a reconnecting router reconciles what it holds against the controller rather than waiting for the next change - hash-gates the digest exchange per store type, replying in full only for the types whose hash differs, since the steady state is that they agree - tells a router which capabilities the whole cluster supports, as distinct from what the controllers it happens to be connected to support; a feature needs both answers before it is used - lifts delta application off the receive goroutine onto its own pool, and gives peer events and controller I/O their own pools, so one slow consumer cannot stall the others - counts the work the apply and I/O pools drop when full, on both sides, in a meter per kind of work, and logs a summary of each once a minute rather than a warning per drop, since the overload that fills a pool fills the log queue too and a warning waits for room in it - registers no state type: what travels over this is added by the commits that follow

Commit:c2a767c
Author:Paul Lorenz
Committer:Paul Lorenz

Add a gossip protocol and replicated state store Link state was every controller's own view, built from what routers reported to it directly. A router announces its links once per reconnect, so a controller that missed the announcement had no way to ask again, and controllers in a cluster disagreed about which links existed until something forced a reconnect. Replicating state between controllers is what this makes possible; nothing uses it yet. - adds a peer-to-peer state store that replicates keyed, owner-scoped entries between controllers, with a Lamport clock for ordering, tombstones for removals, and per-type listeners for consumers - repairs missed updates with an anti-entropy round that exchanges a per-owner digest with a rotating peer, so divergence is corrected without re-sending state that already agrees - derives which capabilities every member of the raft cluster supports, as the intersection of what each last advertised, and withholds an answer until every member has been observed, since a member nothing has been heard from could be running anything - gates each state type on its own capability, so a cluster part-way through an upgrade keeps using what it can carry rather than giving up everything - gives up a type's entries when its capability stops holding, telling listeners the entries are gone without claiming that what they described is gone - documents the protocol, the message sequences and the patterns a state type is expected to follow - documents that listeners run holding the entry's notification lock, so they must not block, which includes logging a warning per entry on a path that load makes frequent

Commit:1c48ead
Author:Paul Lorenz
Committer:Paul Lorenz

Order peer state across controllers by the store index it rests on. For #4669 Controllers compute peer state independently and nothing orders their messages against each other, so a router could be told a destination was removed and then told by another controller, from state computed before the removal, that it was healthy again. The destination came back and the router went on dialing a router that was gone. - carries the controller store index a peer state was computed against, and zero where the reporter has none to offer - remembers a removal against its index, so a report computed against an older one cannot recreate the destination - loses a tie: a router being deleted stays in a controller's connected map until its control channel teardown runs, so the same store state can still produce a healthy report - suppresses only existence, never health, since health is each controller's own view of what it is connected to and a controller lagging in replication reports a low index alongside a perfectly good health report - names the reporting controller on each update, so the registry can union health across controllers while ordering existence by index - forgets a removal after a few minutes, which outlives any report already in flight

Commit:c979786
Author:Paul Lorenz
Committer:Paul Lorenz

Reconcile the raft advertise address with the controller config on startup. Fixes #4487 - adds a startup reconcile that asks the leader to move this node's stored raft address to its configured advertise address, retrying with jittered backoff until the change is visible locally - wakes the running reconcile on leader events instead of starting a second one, and jitters the first attempt and each wake so controllers reconciling after the same event spread their requests - keeps checking a node whose copy of the configuration doesn't list it yet, as after a restart before any of the log reached it, instead of treating it as reconciled - adds an UpdatePeerAddressRequest peer message carrying the member id, the address the node believes is stored and the new address; the leader accepts it only from the node it names and does not forward it - adds updateMemberAddressAsLeader, which changes a member's address in place, keeping its suffrage, after refusing non-members, addresses held by another member and stale from addresses - probes the new address as a sanity check, refusing the change if a different controller answers and making it with a warning if the address can't be reached, since some controllers only dial out - conditions the update on a configuration index read from the raft log and snapshot stores, since GetConfiguration().Index() is always zero in the pinned raft version - routes AddPeer requests that find an existing member at a new address to the in-place update instead of remove and re-add, identifying the member through GetPeerInfo rather than the request id - falls back to the member id when the raft transport dials a stored address with no peer connected at it - adds GetCachedServers to the FSM so that lookup works before raft has finished starting - adds a probe hello header so probe connections are closed without registering a peer - corrects the inverted voter flag in the peer AddPeer log and removes the unused ctrlAddress field - tests single-node, follower, non-voter and leader address changes, refused and unverified addresses, requests from the wrong sender, membership changes made while a member is offline or during the probe, and configuration lookup across snapshots and log gaps - tests the reconcile starting on an empty log and waiting until the configuration lists the node - waits in the HA test harness for a restarted peer's own configuration to list it before restarting it - adds a 2.0.9 changelog entry describing the address update and the leader version requirement

Commit:7f2f619
Author:Paul Lorenz
Committer:Paul Lorenz

Reconcile the raft advertise address with the controller config on startup. Fixes #3588 - adds a startup reconcile that asks the leader to move this node's stored raft address to its configured advertise address, retrying with jittered backoff until the change is visible locally - wakes the running reconcile on leader events instead of starting a second one, and jitters the first attempt and each wake so controllers reconciling after the same event spread their requests - keeps checking a node whose copy of the configuration doesn't list it yet, as after a restart before any of the log reached it, instead of treating it as reconciled - adds an UpdatePeerAddressRequest peer message carrying the member id, the address the node believes is stored and the new address; the leader accepts it only from the node it names and does not forward it - adds updateMemberAddressAsLeader, which changes a member's address in place, keeping its suffrage, after refusing non-members, addresses held by another member and stale from addresses - probes the new address as a sanity check, refusing the change if a different controller answers and making it with a warning if the address can't be reached, since some controllers only dial out - conditions the update on a configuration index read from the raft log and snapshot stores, since GetConfiguration().Index() is always zero in the pinned raft version - routes AddPeer requests that find an existing member at a new address to the in-place update instead of remove and re-add, identifying the member through GetPeerInfo rather than the request id - falls back to the member id when the raft transport dials a stored address with no peer connected at it - adds a probe hello header so probe connections are closed without registering a peer - excludes self by server id in GetPeerAddresses - corrects the inverted voter flag in the peer AddPeer log and removes the unused ctrlAddress field - tests single-node, follower, non-voter and leader address changes, refused and unverified addresses, requests from the wrong sender, membership changes made while a member is offline or during the probe, and configuration lookup across snapshots and log gaps - tests the reconcile starting on an empty log and waiting until the configuration lists the node - waits in the HA test harness for a restarted peer's own configuration to list it before restarting it - adds a changelog entry describing the address update and the leader version requirement

Commit:84881c6
Author:Paul Lorenz
Committer:Paul Lorenz

Record links to destinations this controller cannot yet judge, and settle them once it can A controller that has not caught up on raft cannot tell a destination router that was deleted from one it has not applied yet, and the two want opposite answers. Refusing the report drops a link to a router that does exist, and the reporting router considers it delivered, so nothing ever says it again. One controller ended a soak run 125 links short of its peers that way. - records a report from a router that does not gossip with its destination left unresolved, since the link table is its only copy; an unresolved link is never selected for a path - removes those links once their destination is still absent under a read barrier requested after the destination was first named, and tells the dialer its destination is gone, retrying the notice until it can be sent - keeps that time per destination, the earliest arrival naming it, so reports naming other destinations, or repeats naming the same one, cannot keep a barrier from settling it, and refuses reports naming a settled destination as they arrive - forgets a settled destination some minutes after it was first named; anything still naming it notes it again and waits for another barrier - adds an asynchronous raft read barrier: a follower asks the leader, which commits a raft barrier and answers with its store index, and the leader answers its own requests the same way; requests carry an id, so only the outstanding request's answer is accepted - leaves a gossiped entry naming an unknown destination in the store and asks about it again on the network cycle, refusing it once a barrier covers its destination, and answers a proved absence with a removal notice to its owner - re-reads a destination from the store after publishing a link to it, so a link published while its destination is being deleted is removed rather than outliving the delete's cleanup - retires a deleted router's links by scanning the link table rather than the router's index, which anything finding the router gone may already have dropped - renders a link whose destination cannot be resolved by its id rather than failing the whole link listing - renames the peer-state index to say what it is: the raft index the sending controller had applied, not a store index

Commit:73bb151
Author:Paul Lorenz
Committer:Paul Lorenz

Replicate link state over gossip. Fixes #3726 Link state was every controller's own view, built from what routers reported to it directly. A router announces its links once per reconnect, so a controller that missed the announcement had no way to ask again, and controllers in a cluster disagreed about which links existed until something forced a reconnect. - publishes a router's dialed links into the gossip store, keyed by link id and iteration and owned by the dialer, so every controller converges on the same set without the router telling each one - builds links from gossip entries on arrival, and re-walks the owner's entries when a router connects, which covers entries applied while the router was unknown - retires a link by tombstoning its entry, so the removal reaches every controller rather than only the one that heard the fault, and forwards the fault to whichever endpoint is connected here - counts fault forwards dropped for a full I/O pool rather than logging each one, since the forward is queued under the entry's notification lock, which every later change to that entry waits for - sends the fault for a link closed after its state is gone, as when its destination is removed or its dial state was detached, through the notification cycle rather than directly, so it is ordered after the live entry it retires - keeps a router's reports out of the store while it is not a publisher, and retracts what it published when it stops being one - holds a link's source router atomically, since a link built from a gossiped entry carries a database-loaded router until its real connection arrives - refuses a report naming an existing link id from a router that is not its source, which would otherwise let any router take over a link between two others by reporting a higher iteration - answers a refused entry with a removal notice to its dialer, which is the only thing that stops the dialer re-announcing it

Commit:c2c9bf4
Author:Paul Lorenz
Committer:Paul Lorenz

Carry gossip between controllers and routers over the control channel - adds the router-side gossip client and the controller handlers for deltas, digest requests and digest responses, so a state type has a wire to travel on - opens with a digest to each router on connect, so a reconnecting router reconciles what it holds against the controller rather than waiting for the next change - hash-gates the digest exchange per store type, replying in full only for the types whose hash differs, since the steady state is that they agree - tells a router which capabilities the whole cluster supports, as distinct from what the controllers it happens to be connected to support; a feature needs both answers before it is used - lifts delta application off the receive goroutine onto its own pool, and gives peer events and controller I/O their own pools, so one slow consumer cannot stall the others - counts the work the apply and I/O pools drop when full, on both sides, in a meter per kind of work, and logs a summary of each once a minute rather than a warning per drop, since the overload that fills a pool fills the log queue too and a warning waits for room in it - registers no state type: what travels over this is added by the commits that follow

Commit:e9c084d
Author:Paul Lorenz
Committer:Paul Lorenz

Add a gossip protocol and replicated state store Link state was every controller's own view, built from what routers reported to it directly. A router announces its links once per reconnect, so a controller that missed the announcement had no way to ask again, and controllers in a cluster disagreed about which links existed until something forced a reconnect. Replicating state between controllers is what this makes possible; nothing uses it yet. - adds a peer-to-peer state store that replicates keyed, owner-scoped entries between controllers, with a Lamport clock for ordering, tombstones for removals, and per-type listeners for consumers - repairs missed updates with an anti-entropy round that exchanges a per-owner digest with a rotating peer, so divergence is corrected without re-sending state that already agrees - derives which capabilities every member of the raft cluster supports, as the intersection of what each last advertised, and withholds an answer until every member has been observed, since a member nothing has been heard from could be running anything - gates each state type on its own capability, so a cluster part-way through an upgrade keeps using what it can carry rather than giving up everything - gives up a type's entries when its capability stops holding, telling listeners the entries are gone without claiming that what they described is gone - documents the protocol, the message sequences and the patterns a state type is expected to follow - documents that listeners run holding the entry's notification lock, so they must not block, which includes logging a warning per entry on a path that load makes frequent

Commit:6423d2c
Author:Paul Lorenz
Committer:Paul Lorenz

Order peer state across controllers by the store index it rests on Controllers compute peer state independently and nothing orders their messages against each other, so a router could be told a destination was removed and then told by another controller, from state computed before the removal, that it was healthy again. The destination came back and the router went on dialing a router that was gone. - carries the controller store index a peer state was computed against, and zero where the reporter has none to offer - remembers a removal against its index, so a report computed against an older one cannot recreate the destination - loses a tie: a router being deleted stays in a controller's connected map until its control channel teardown runs, so the same store state can still produce a healthy report - suppresses only existence, never health, since health is each controller's own view of what it is connected to and a controller lagging in replication reports a low index alongside a perfectly good health report - names the reporting controller on each update, so the registry can union health across controllers while ordering existence by index - forgets a removal after a few minutes, which outlives any report already in flight

Commit:896c8ca
Author:Paul Lorenz
Committer:Paul Lorenz

Hold a link report until this controller can judge its destination A controller that has not caught up on raft cannot tell a destination router that was deleted from one it has not applied yet, and the two want opposite answers. Refusing the report drops a link to a router that does exist, and the reporting router considers it delivered, so nothing ever says it again. One controller ended a soak run 125 links short of its peers that way. - holds a report whose destination is unknown while this controller is behind, and judges it again once it has applied everything the cluster had committed when the pass started - takes that barrier once per pass rather than per arrival, so a cluster that keeps committing cannot keep pushing the target away - drops a held report once its destination is still absent after several passes, since deferring must not become permanent holding, and stays silent while leaderless, where a frozen view would be mistaken for deletion - says which of the ways a report was not held applies, and exposes the held set as a gauge and an inspect key, so a controller shedding reports does not look like one holding none - keeps a report that was deferred again while a pass was running, counting the holds it has had, since the held report is rewritten in place and pointer identity cannot tell the two apart - works from a value snapshot taken under the lock, so a pass never reads a struct a concurrent re-report is rewriting - confines deferral to reports from routers that do not gossip, which have no other copy anywhere; a gossiped report is already held by the store - asks the store again, on the network cycle, about the entries whose destination could not be judged, naming them rather than copying them, and reads and applies each under the store's delivery lock so a removal cannot land between the two - renames the peer-state index to say what it is: the raft index the sending controller had applied, not a store index

Commit:cf4897e
Author:Paul Lorenz
Committer:Paul Lorenz

Replicate link state over gossip. Fixes #3726 Link state was every controller's own view, built from what routers reported to it directly. A router announces its links once per reconnect, so a controller that missed the announcement had no way to ask again, and controllers in a cluster disagreed about which links existed until something forced a reconnect. - publishes a router's dialed links into the gossip store, keyed by link id and iteration and owned by the dialer, so every controller converges on the same set without the router telling each one - builds links from gossip entries on arrival, and re-walks the owner's entries when a router connects, which covers entries applied while the router was unknown - retires a link by tombstoning its entry, so the removal reaches every controller rather than only the one that heard the fault, and forwards the fault to whichever endpoint is connected here - keeps a router's reports out of the store while it is not a publisher, and retracts what it published when it stops being one - holds a link's source router atomically, since a link built from a gossiped entry carries a database-loaded router until its real connection arrives - refuses a report naming an existing link id from a router that is not its source, which would otherwise let any router take over a link between two others by reporting a higher iteration - answers a refused entry with a removal notice to its dialer, which is the only thing that stops the dialer re-announcing it

Commit:9171ef5
Author:Paul Lorenz
Committer:Paul Lorenz

Carry gossip between controllers and routers over the control channel - adds the router-side gossip client and the controller handlers for deltas, digest requests and digest responses, so a state type has a wire to travel on - opens with a digest to each router on connect, so a reconnecting router reconciles what it holds against the controller rather than waiting for the next change - hash-gates the digest exchange per store type, replying in full only for the types whose hash differs, since the steady state is that they agree - tells a router which capabilities the whole cluster supports, as distinct from what the controllers it happens to be connected to support; a feature needs both answers before it is used - lifts delta application off the receive goroutine onto its own pool, and gives peer events and controller I/O their own pools, so one slow consumer cannot stall the others - registers no state type: what travels over this is added by the commits that follow

Commit:4900adb
Author:Paul Lorenz
Committer:Paul Lorenz

Add a gossip protocol and replicated state store Link state was every controller's own view, built from what routers reported to it directly. A router announces its links once per reconnect, so a controller that missed the announcement had no way to ask again, and controllers in a cluster disagreed about which links existed until something forced a reconnect. Replicating state between controllers is what this makes possible; nothing uses it yet. - adds a peer-to-peer state store that replicates keyed, owner-scoped entries between controllers, with a Lamport clock for ordering, tombstones for removals, and per-type listeners for consumers - repairs missed updates with an anti-entropy round that exchanges a per-owner digest with a rotating peer, so divergence is corrected without re-sending state that already agrees - derives which capabilities every member of the raft cluster supports, as the intersection of what each last advertised, and withholds an answer until every member has been observed, since a member nothing has been heard from could be running anything - gates each state type on its own capability, so a cluster part-way through an upgrade keeps using what it can carry rather than giving up everything - gives up a type's entries when its capability stops holding, telling listeners the entries are gone without claiming that what they described is gone - documents the protocol, the message sequences and the patterns a state type is expected to follow

Commit:8a41c18
Author:Paul Lorenz
Committer:Paul Lorenz

Order peer state across controllers by the store index it rests on Controllers compute peer state independently and nothing orders their messages against each other, so a router could be told a destination was removed and then told by another controller, from state computed before the removal, that it was healthy again. The destination came back and the router went on dialing a router that was gone. - carries the controller store index a peer state was computed against, and zero where the reporter has none to offer - remembers a removal against its index, so a report computed against an older one cannot recreate the destination - loses a tie: a router being deleted stays in a controller's connected map until its control channel teardown runs, so the same store state can still produce a healthy report - suppresses only existence, never health, since health is each controller's own view of what it is connected to and a controller lagging in replication reports a low index alongside a perfectly good health report - names the reporting controller on each update, so the registry can union health across controllers while ordering existence by index - forgets a removal after a few minutes, which outlives any report already in flight

Commit:805027e
Author:Paul Lorenz

Hold a link report until this controller can judge its destination A controller that has not caught up on raft cannot tell a destination router that was deleted from one it has not applied yet, and the two want opposite answers. Refusing the report drops a link to a router that does exist, and the reporting router considers it delivered, so nothing ever says it again. One controller ended a soak run 125 links short of its peers that way. - holds a report whose destination is unknown while this controller is behind, and judges it again once it has applied everything the cluster had committed when the pass started - takes that barrier once per pass rather than per arrival, so a cluster that keeps committing cannot keep pushing the target away - drops a held report once its destination is still absent after several passes, since deferring must not become permanent holding, and stays silent while leaderless, where a frozen view would be mistaken for deletion - says which of the ways a report was not held applies, and exposes the held set as a gauge and an inspect key, so a controller shedding reports does not look like one holding none - keeps a report that was deferred again while a pass was running, counting the holds it has had, since the held report is rewritten in place and pointer identity cannot tell the two apart - works from a value snapshot taken under the lock, so a pass never reads a struct a concurrent re-report is rewriting - confines deferral to reports from routers that do not gossip, which have no other copy anywhere; a gossiped report is already held by the store - asks the store again, on the network cycle, about the entries whose destination could not be judged, naming them rather than copying them, and reads and applies each under the store's delivery lock so a removal cannot land between the two - renames the peer-state index to say what it is: the raft index the sending controller had applied, not a store index

Commit:233dbe0
Author:Paul Lorenz

Replicate link state over gossip. Fixes #3726 Link state was every controller's own view, built from what routers reported to it directly. A router announces its links once per reconnect, so a controller that missed the announcement had no way to ask again, and controllers in a cluster disagreed about which links existed until something forced a reconnect. - publishes a router's dialed links into the gossip store, keyed by link id and iteration and owned by the dialer, so every controller converges on the same set without the router telling each one - builds links from gossip entries on arrival, and re-walks the owner's entries when a router connects, which covers entries applied while the router was unknown - retires a link by tombstoning its entry, so the removal reaches every controller rather than only the one that heard the fault, and forwards the fault to whichever endpoint is connected here - keeps a router's reports out of the store while it is not a publisher, and retracts what it published when it stops being one - holds a link's source router atomically, since a link built from a gossiped entry carries a database-loaded router until its real connection arrives - refuses a report naming an existing link id from a router that is not its source, which would otherwise let any router take over a link between two others by reporting a higher iteration - answers a refused entry with a removal notice to its dialer, which is the only thing that stops the dialer re-announcing it

Commit:69361e8
Author:Paul Lorenz

Carry gossip between controllers and routers over the control channel - adds the router-side gossip client and the controller handlers for deltas, digest requests and digest responses, so a state type has a wire to travel on - opens with a digest to each router on connect, so a reconnecting router reconciles what it holds against the controller rather than waiting for the next change - hash-gates the digest exchange per store type, replying in full only for the types whose hash differs, since the steady state is that they agree - tells a router which capabilities the whole cluster supports, as distinct from what the controllers it happens to be connected to support; a feature needs both answers before it is used - lifts delta application off the receive goroutine onto its own pool, and gives peer events and controller I/O their own pools, so one slow consumer cannot stall the others - registers no state type: what travels over this is added by the commits that follow

Commit:2a94c3f
Author:Paul Lorenz
Committer:Paul Lorenz

Add a gossip protocol and replicated state store Link state was every controller's own view, built from what routers reported to it directly. A router announces its links once per reconnect, so a controller that missed the announcement had no way to ask again, and controllers in a cluster disagreed about which links existed until something forced a reconnect. Replicating state between controllers is what this makes possible; nothing uses it yet. - adds a peer-to-peer state store that replicates keyed, owner-scoped entries between controllers, with a Lamport clock for ordering, tombstones for removals, and per-type listeners for consumers - repairs missed updates with an anti-entropy round that exchanges a per-owner digest with a rotating peer, so divergence is corrected without re-sending state that already agrees - derives which capabilities every member of the raft cluster supports, as the intersection of what each last advertised, and withholds an answer until every member has been observed, since a member nothing has been heard from could be running anything - gates each state type on its own capability, so a cluster part-way through an upgrade keeps using what it can carry rather than giving up everything - gives up a type's entries when its capability stops holding, telling listeners the entries are gone without claiming that what they described is gone - documents the protocol, the message sequences and the patterns a state type is expected to follow

Commit:cc2f277
Author:Paul Lorenz

Order peer state across controllers by the store index it rests on Controllers compute peer state independently and nothing orders their messages against each other, so a router could be told a destination was removed and then told by another controller, from state computed before the removal, that it was healthy again. The destination came back and the router went on dialing a router that was gone. - carries the controller store index a peer state was computed against, and zero where the reporter has none to offer - remembers a removal against its index, so a report computed against an older one cannot recreate the destination - loses a tie: a router being deleted stays in a controller's connected map until its control channel teardown runs, so the same store state can still produce a healthy report - suppresses only existence, never health, since health is each controller's own view of what it is connected to and a controller lagging in replication reports a low index alongside a perfectly good health report - names the reporting controller on each update, so the registry can union health across controllers while ordering existence by index - forgets a removal after a few minutes, which outlives any report already in flight

Commit:dd37e75
Author:Paul Lorenz
Committer:Paul Lorenz

Stop routers dropping controllers on a follower's stale controller list. Fixes #4574 - applies controller lists, leader announcements, controller address changes and index resets on one goroutine in arrival order, so checking an update's index, applying it and recording it is one step and no control channel's receive goroutine waits on another controller's update - lets only a leader's list at a newer index replace the controller set; any other list at or above the recorded version only adds, merging into the router's stored details and connected controllers - applies a leader's list at the recorded version as additions, so a router missing controllers recovers when the leader resends - keeps an inbound controller's socket address out of the saved endpoints, and fills in a saved controller that has no endpoints when a later list gives some - orders leadership claims by raft term: adds a term to UpdateCtrlAddresses and UpdateClusterLeader, which controllers read together with leadership so both describe the same term; claims without a term, from older controllers, still apply as they arrive - folds the cluster leader handler into the same worker - asks every connected controller for its list while waiting for the leader's, without waiting on a full send queue, one request run at a time, stopping when the router closes - bounds the waiting updates: past the bound it drops waiting lists and announcements, keeps each controller's latest address change, and requests the leader's list - fixes a router crash applying a controller's new address before any controller details were stored, and keeps connected controllers the router has no details for - stops clustered controllers sending the newListener address, which the cluster's controller lists would undo, warns when it is configured, and notes this in the sample configs - tests list handling, leadership ordering, overflow, the request loop, address changes and the controller's term read

Commit:b129fa4
Author:Paul Lorenz
Committer:Paul Lorenz

Stop routers dropping controllers on a follower's stale controller list. Fixes #4573 - applies controller lists, leader announcements, controller address changes and index resets on one goroutine in arrival order, so checking an update's index, applying it and recording it is one step and no control channel's receive goroutine waits on another controller's update - lets only a leader's list at a newer index replace the controller set; any other list at or above the recorded version only adds, merging into the router's stored details and connected controllers - applies a leader's list at the recorded version as additions, so a router missing controllers recovers when the leader resends - keeps an inbound controller's socket address out of the saved endpoints, and fills in a saved controller that has no endpoints when a later list gives some - orders leadership claims by raft term: adds a term to UpdateCtrlAddresses and UpdateClusterLeader, which controllers read together with leadership so both describe the same term; claims without a term, from older controllers, still apply as they arrive - folds the cluster leader handler into the same worker - asks every connected controller for its list while waiting for the leader's, without waiting on a full send queue, one request run at a time, stopping when the router closes - bounds the waiting updates: past the bound it drops waiting lists and announcements, keeps each controller's latest address change, and requests the leader's list - fixes a router crash applying a controller's new address before any controller details were stored, and keeps connected controllers the router has no details for - stops clustered controllers sending the newListener address, which the cluster's controller lists would undo, warns when it is configured, and notes this in the sample configs - tests list handling, leadership ordering, overflow, the request loop, address changes and the controller's term read - notes the fix in the changelog

Commit:4ff5ff9
Author:Paul Lorenz
Committer:Paul Lorenz

Decide delete existence in Apply and add DeleteIfExists. Fixes #4571 - checks the deleting manager's own store for the target inside the delete transaction, before DeleteById, so a child store delete never reaches the parent's record of another type - moves the edge service manager's fabric-only check from a pre-dispatch read into an ApplyDelete override - makes IsEntityPresent on an extended child store match FindById, so a router created through the fabric is deletable through the transit router manager, which excludes edge routers explicitly - adds a flags field to the delete command and DeleteIfExists on the base manager, under which an absent target is a no-op - routes identity deletes through the base ApplyDelete after the admin precondition, so identities get the presence check and DeleteIfExists - gives the API session, session and API session certificate managers a direct DeleteIfExists, since they register no delete decoder - tests cross-type router deletes, the fabric-only check at apply time, base router deletes, and DeleteIfExists on present, absent, fabric-only, wrong-type, identity and local-only targets, at the model level and through the management API

Commit:4cfc691
Author:Paul Lorenz

Stop a digest exchange renewing the tombstones a controller already holds - marks each digest entry as live or tombstone, so the receiver can tell a key the sender still holds from one it has already retired - skips a key the controller holds as a tombstone when retiring entries the router's source of truth no longer backs, while still dropping that key from the advertised set - keeps tombstones in the version map, since a key the router still advertises that the controller holds as a tombstone is the divergence the re-stamp exists to repair - fixes every exchange stamping a fresh lifetime on each tombstone the controller holds for the router and broadcasting it to every peer again, which left the tombstones of a router that churns enough to keep failing the hash gate unable to ever be reaped - tests the skip, the live-entry retirement it must not suppress, and the re-stamp over a tombstone for a key still advertised - records in the gossip design doc why both halves of the comparison need the flag

Commit:f22d845
Author:Paul Lorenz
Committer:Paul Lorenz

Reconcile the raft advertise address with the controller config on startup. Fixes #4487 - adds a startup reconcile that asks the leader to move this node's stored raft address to its configured advertise address, retrying with jittered backoff until the change is visible locally - wakes the running reconcile on leader events instead of starting a second one, and jitters the first attempt and each wake so controllers reconciling after the same event spread their requests - adds an UpdatePeerAddressRequest peer message carrying the member id, the address the node believes is stored and the new address; the leader accepts it only from the node it names and does not forward it - adds updateMemberAddressAsLeader, which changes a member's address in place, keeping its suffrage, after refusing non-members, addresses held by another member and stale from addresses - probes the new address as a sanity check, refusing the change if a different controller answers and making it with a warning if the address can't be reached, since some controllers only dial out - conditions the update on a configuration index read from the raft log and snapshot stores, since GetConfiguration().Index() is always zero in the pinned raft version - routes AddPeer requests that find an existing member at a new address to the in-place update instead of remove and re-add, identifying the member through GetPeerInfo rather than the request id - falls back to the member id when the raft transport dials a stored address with no peer connected at it - adds GetCachedServers to the FSM so that lookup works before raft has finished starting - adds a probe hello header so probe connections are closed without registering a peer - corrects the inverted voter flag in the peer AddPeer log and removes the unused ctrlAddress field - tests single-node, follower, non-voter and leader address changes, refused and unverified addresses, requests from the wrong sender, membership changes made while a member is offline or during the probe, and configuration lookup across snapshots and log gaps - adds a 2.0.8 changelog entry describing the address update and the leader version requirement

Commit:eb029f9
Author:Paul Lorenz
Committer:Paul Lorenz

Reconcile the raft advertise address with the controller config on startup. Fixes #3588 - adds a startup reconcile that asks the leader to move this node's stored raft address to its configured advertise address, retrying with jittered backoff until the change is visible locally - wakes the running reconcile on leader events instead of starting a second one, and jitters the first attempt and each wake so controllers reconciling after the same event spread their requests - adds an UpdatePeerAddressRequest peer message carrying the member id, the address the node believes is stored and the new address; the leader accepts it only from the node it names and does not forward it - adds updateMemberAddressAsLeader, which changes a member's address in place, keeping its suffrage, after refusing non-members, addresses held by another member and stale from addresses - probes the new address as a sanity check, refusing the change if a different controller answers and making it with a warning if the address can't be reached, since some controllers only dial out - conditions the update on a configuration index read from the raft log and snapshot stores, since GetConfiguration().Index() is always zero in the pinned raft version - routes AddPeer requests that find an existing member at a new address to the in-place update instead of remove and re-add, identifying the member through GetPeerInfo rather than the request id - falls back to the member id when the raft transport dials a stored address with no peer connected at it - adds a probe hello header so probe connections are closed without registering a peer - excludes self by server id in GetPeerAddresses - corrects the inverted voter flag in the peer AddPeer log and removes the unused ctrlAddress field - tests single-node, follower, non-voter and leader address changes, refused and unverified addresses, requests from the wrong sender, membership changes made while a member is offline or during the probe, and configuration lookup across snapshots and log gaps - adds a changelog entry describing the address update and the leader version requirement

Commit:aac89f2
Author:Paul Lorenz

Reconcile the raft advertise address with the controller config on startup. Fixes #3588 - adds a startup reconcile that asks the leader to move this node's stored raft address to its configured advertise address, retrying with jittered backoff until the change is visible locally - wakes the running reconcile on leader events instead of starting a second one, and jitters the first attempt and each wake so controllers reconciling after the same event spread their requests - adds an UpdatePeerAddressRequest peer message carrying the member id, the address the node believes is stored and the new address; the leader accepts it only from the node it names and does not forward it - adds updateMemberAddressAsLeader, which changes a member's address in place, keeping its suffrage, after refusing non-members, addresses held by another member and stale from addresses - probes the new address as a sanity check, refusing the change if a different controller answers and making it with a warning if the address can't be reached, since some controllers only dial out - conditions the update on a configuration index read from the raft log and snapshot stores, since GetConfiguration().Index() is always zero in the pinned raft version - routes AddPeer requests that find an existing member at a new address to the in-place update instead of remove and re-add, identifying the member through GetPeerInfo rather than the request id - falls back to the member id when the raft transport dials a stored address with no peer connected at it - adds a probe hello header so probe connections are closed without registering a peer - excludes self by server id in GetPeerAddresses - corrects the inverted voter flag in the peer AddPeer log and removes the unused ctrlAddress field - tests single-node, follower, non-voter and leader address changes, refused and unverified addresses, requests from the wrong sender, membership changes made while a member is offline or during the probe, and configuration lookup across snapshots and log gaps

Commit:c4b5a96
Author:Paul Lorenz
Committer:GitHub

Merge pull request #4215 from openziti/router-link-runtime-config Apply managed link heartbeat and queue settings at runtime

Commit:85637c2
Author:Paul Lorenz
Committer:Paul Lorenz

Reconcile the raft advertise address with the controller config on startup. Fixes #3588 - adds a startup reconcile that asks the leader to move this node's stored raft address to its configured advertise address, retrying with backoff until the change is visible locally - adds an UpdatePeerAddressRequest peer message carrying the member id, the address the node believes is stored and the new address, so address updates never carry suffrage or add members - adds UpdateMemberAddressAsLeader, which changes a member's address in place, keeping its suffrage, after refusing non-members, addresses held by another member, stale from addresses and new addresses that don't answer with the member's id - conditions the update on a configuration index read from the raft log and snapshot stores, since GetConfiguration().Index() is always zero in the pinned raft version - routes AddPeer requests for an existing member at a new address to the in-place update instead of remove and re-add - falls back to the member id when the raft transport dials a stored address with no peer connected at it - adds a probe hello header so probe connections are closed without registering a peer - excludes self by server id in GetPeerAddresses - shuts down raft before closing the db in Controller.Shutdown, since in cluster mode the db belongs to the raft FSM - corrects the inverted voter flag in the peer AddPeer log and removes the unused ctrlAddress field - tests single-node, follower, non-voter and leader address changes, refused changes, membership changes made while a member is offline or during the probe, and configuration lookup across snapshots and log gaps

Commit:a9c7ee5
Author:Paul Lorenz
Committer:Paul Lorenz

Reconcile the raft advertise address with the controller config on startup. Fixes #3588 - adds a startup reconcile that asks the leader to move this node's stored raft address to its configured advertise address, retrying with backoff until the change is visible locally - adds an UpdatePeerAddressRequest peer message carrying the member id, the address the node believes is stored and the new address, so address updates never carry suffrage or add members - adds UpdateMemberAddressAsLeader, which changes a member's address in place, keeping its suffrage, after refusing non-members, addresses held by another member, stale from addresses and new addresses that don't answer with the member's id - conditions the update on a configuration index read from the raft log and snapshot stores, since GetConfiguration().Index() is always zero in the pinned raft version - routes AddPeer requests for an existing member at a new address to the in-place update instead of remove and re-add - falls back to the member id when the raft transport dials a stored address with no peer connected at it - adds a probe hello header so probe connections are closed without registering a peer - excludes self by server id in GetPeerAddresses - shuts down raft before closing the db in Controller.Shutdown, since in cluster mode the db belongs to the raft FSM - corrects the inverted voter flag in the peer AddPeer log and removes the unused ctrlAddress field - tests single-node, follower, non-voter and leader address changes, refused changes, membership changes made while a member is offline or during the probe, and configuration lookup across snapshots and log gaps

Commit:8fe27b1
Author:Paul Lorenz
Committer:Paul Lorenz

Controller should clear links not in full link sync from router. Fixes #4458 - adds a fullRefresh flag to the router's reconnect link announcement - prunes links sourced from the announcing router that are not in a full refresh set - moves the reroute of a usable faulted link into LinkFaulted - adds LinkManager.IterateLinks - For #3493

Commit:0e78817
Author:Paul Lorenz
Committer:Paul Lorenz

Break the SDK terminator removal retry storm. Fixes #4467 Backport of #4159 to release-v2.0.x. Adapted for channel/v4 and sdk-golang v1, 2.0's int-typed capability constants and binding.AddTypedReceiveHandler registration. The ER/T tests that depend on the establish settle-gate fixtures (#4194, not on this branch) are omitted; their unit-level equivalents are kept. ctrl.pb.go is regenerated with protoc-gen-go v1.36.11, matching main and the google.golang.org/protobuf v1.36.12 runtime this branch already depends on; the committed file predated that bump and was still v1.31.0 output. - adds RemoveTerminatorsV2, an asynchronous control-channel removal, so a router no longer holds a request slot open waiting for a synchronous reply; gated on a new capability so routers fall back to the synchronous path against controllers that do not advertise it - paces establish and remove retries with a per-terminator exponential backoff after a controller rejection, so a rejected operation is not re-sent on the next event-loop tick, which is what let a rate-limited removal sustain itself - routes v2 removes to the leader and skips the ordered delete for a terminator the router confirms it created that no longer exists, so a storm of no-op retries cannot consume the command rate limiter - gates that skip on the leader having applied everything committed when it took leadership, since a leader reports itself elected before those applies finish and would otherwise read a committed create as absent and drop its delete - guards the leader term and every write to the caught-up flag with a dedicated lock, so a barrier completing alongside a leadership change cannot record a result for a term that has already ended - scopes v2 removals to the requesting router, so a router can only remove terminators it owns on the asynchronous path as well as the synchronous one - tracks in-flight removal batches on the router, resolving or requeueing them when the response arrives; lost responses self-heal via rate-limiter expiry and the retry scan - waits for an outstanding operation before sending a terminator delete, and checks that before dequeuing, so a delete neither races a create the controller has committed but not applied nor falls out of the queue until the retry scan finds it minutes later - matches a create response to the attempt that produced it, by control channel and reply sequence, since attempts overlap once one overruns the establishment timeout and every attempt carries the same terminator id; only the outstanding attempt's response may confirm the create or report the terminator idle - records the attempt as the sender assigns its sequence, so a response cannot be processed before the request it answers is known, and withdraws an attempt whose send never queued - reports a removal that succeeds past the establishment timeout as congestion rather than success, matching how the establish path classifies its own latency, since both resolve into the same limiter - extracts the transient-error test both remove handlers share, so the v2 path treats a leaderless cluster as busy like v1 does instead of as a permanent failure, and the two cannot drift again - applies the same delete sequencing and response correlation to the router-hosted (ER/T) registry, which cleared the same in-flight marker unguarded and dropped a deferred delete from its queue rather than leaving it there - bounds the v2 response send with a timeout, so a router that stops draining its send queue cannot accumulate blocked responder goroutines on the controller

Commit:444999b
Author:Paul Lorenz
Committer:Paul Lorenz

Break the SDK terminator removal retry storm. Fixes #4156 - adds RemoveTerminatorsV2, an asynchronous control-channel removal, so a router no longer holds a request slot open waiting for a synchronous reply; gated on a new capability so routers fall back to the synchronous path against controllers that do not advertise it - paces establish and remove retries with a per-terminator exponential backoff after a controller rejection, so a rejected operation is not re-sent on the next event-loop tick, which is what let a rate-limited removal sustain itself - routes v2 removes to the leader and skips the ordered delete for a terminator the router confirms it created that no longer exists, so a storm of no-op retries cannot consume the command rate limiter - gates that skip on the leader having applied everything committed when it took leadership, since a leader reports itself elected before those applies finish and would otherwise read a committed create as absent and drop its delete - guards the leader term and every write to the caught-up flag with a dedicated lock, so a barrier completing alongside a leadership change cannot record a result for a term that has already ended - scopes v2 removals to the requesting router, so a router can only remove terminators it owns on the asynchronous path as well as the synchronous one - tracks in-flight removal batches on the router, resolving or requeueing them when the response arrives; lost responses self-heal via rate-limiter expiry and the retry scan - waits for an outstanding operation before sending a terminator delete, and checks that before dequeuing, so a delete neither races a create the controller has committed but not applied nor falls out of the queue until the retry scan finds it minutes later - matches a create response to the attempt that produced it, by control channel and reply sequence, since attempts overlap once one overruns the establishment timeout and every attempt carries the same terminator id; only the outstanding attempt's response may confirm the create or report the terminator idle - records the attempt as the sender assigns its sequence, so a response cannot be processed before the request it answers is known, and withdraws an attempt whose send never queued - reports a removal that succeeds past the establishment timeout as congestion rather than success, matching how the establish path classifies its own latency, since both resolve into the same limiter - extracts the transient-error test both remove handlers share, so the v2 path treats a leaderless cluster as busy like v1 does instead of as a permanent failure, and the two cannot drift again - applies the same delete sequencing and response correlation to the router-hosted (ER/T) registry, which cleared the same in-flight marker unguarded and dropped a deferred delete from its queue rather than leaving it there - bounds the v2 response send with a timeout, so a router that stops draining its send queue cannot accumulate blocked responder goroutines on the controller

Commit:6bb287a
Author:Paul Lorenz

Break the SDK terminator removal retry storm. Fixes #4156 - adds RemoveTerminatorsV2, an asynchronous control-channel removal, so a router no longer holds a request slot open waiting for a synchronous reply; gated on a new capability so routers fall back to the synchronous path against controllers that do not advertise it - paces establish and remove retries with a per-terminator exponential backoff after a controller rejection, so a rejected operation is not re-sent on the next event-loop tick, which is what let a rate-limited removal sustain itself - routes v2 removes to the leader and skips the ordered delete for a terminator the router confirms it created that no longer exists, so a storm of no-op retries cannot consume the command rate limiter - gates that skip on the leader having applied everything committed when it took leadership, since a leader reports itself elected before those applies finish and would otherwise read a committed create as absent and drop its delete - guards the leader term and every write to the caught-up flag with a dedicated lock, so a barrier completing alongside a leadership change cannot record a result for a term that has already ended - scopes v2 removals to the requesting router, so a router can only remove terminators it owns on the asynchronous path as well as the synchronous one - tracks in-flight removal batches on the router, resolving or requeueing them when the response arrives; lost responses self-heal via rate-limiter expiry and the retry scan - waits for an outstanding operation before sending a terminator delete, and checks that before dequeuing, so a delete neither races a create the controller has committed but not applied nor falls out of the queue until the retry scan finds it minutes later - matches a create response to the attempt that produced it, by control channel and reply sequence, since attempts overlap once one overruns the establishment timeout and every attempt carries the same terminator id; only the outstanding attempt's response may confirm the create or report the terminator idle - records the attempt as the sender assigns its sequence, so a response cannot be processed before the request it answers is known, and withdraws an attempt whose send never queued - reports a removal that succeeds past the establishment timeout as congestion rather than success, matching how the establish path classifies its own latency, since both resolve into the same limiter - extracts the transient-error test both remove handlers share, so the v2 path treats a leaderless cluster as busy like v1 does instead of as a permanent failure, and the two cannot drift again - applies the same delete sequencing and response correlation to the router-hosted (ER/T) registry, which cleared the same in-flight marker unguarded and dropped a deferred delete from its queue rather than leaving it there

Commit:fd64d83
Author:Paul Lorenz

Controller should clear links not in full link sync from router. Fixes #4458 - adds a fullRefresh flag to the router's reconnect link announcement - prunes links sourced from the announcing router that are not in a full refresh set - moves the reroute of a usable faulted link into LinkFaulted - adds LinkManager.IterateLinks - For #3493

Commit:54d5a9e
Author:Paul Lorenz
Committer:Paul Lorenz

Hold a link report naming a destination this controller has not caught up to - defers a report whose destination is absent while this controller is behind, instead of discarding it, since an entity not yet applied is indistinguishable from one that was deleted and the two want opposite answers - re-evaluates held reports on the existing network cycle, applying those whose destination has since appeared and dropping those still absent after several passes - waits on a raft index captured once per pass rather than on being fully caught up, which on a busy cluster can be false indefinitely and would hold reports for ever - skips evaluation entirely without a leader, where the local view is frozen rather than behind and an absence says nothing - bounds the held set, since a controller that never catches up cannot route and is re-announced to on reconnect - gates the destination-removed notice on being caught up, so a leader still applying entries from before its election cannot tell routers to drop links to routers that exist - adds GetRaftCommitIndex, HasRaftLeader and IsRaftCaughtUp to the controller env, and replaces the deprecated raft Leader call with LeaderWithID - renames the peer state index from storeIndex to raftIndex, matching what the change context, the command protos and the persisted field already call it - logs every detach of a link pairing, with its source, status and whether it held a link, since a silent detach leaves a dial in flight with nowhere to land and no record of why - fixes a controller ending permanently short of its peers, which a soak run reproduced as one controller holding 125 fewer links than the other two after discarding 181 reports during bring-up

Commit:6c20e68
Author:Paul Lorenz
Committer:Paul Lorenz

Apply managed link heartbeat and queue settings at runtime. For #3743 - applies the router.link.v1 heartbeat settings (sendInterval, checkInterval, closeUnresponsiveTimeout) and payload/ack sender queue sizes at runtime, replacing the hardcoded 10s/1s link heartbeat intervals and the ignored fields; new links take them at bind time, established links are re-tuned in place through the channel HeartbeatControl, and queue sizes reach new links only - publishes heartbeat settings as an immutable generation in one atomic swap, so no reader pairs one generation's interval with another's timeout, and has each link enforce the timeout belonging to its own intervals - pushes a generation and its intervals as one step under the link's lock and ignores a generation at or below the one applied, so an overtaken update cannot roll a link back; reconciles every channel of a link, including a split link whose channels bind on either side of a config change, to the newest generation - adopts the first registered settings explicitly, since a router with no managed link config publishes its defaults at generation 0 - re-arms a link's liveness deadline when its generation changes, so age accumulated under the previous intervals cannot close a healthy link when a timeout is shortened - honors a closeUnresponsiveTimeout below 30s, which previously sat behind the unhealthy threshold; treats a zero timeout as closing nothing, and reports a closing link unhealthy - scales the unhealthy threshold with the intervals, to max(30s, 2 x (send + check)): reporting unhealthy poisons the latency metric the controller routes on, so a fixed 30s priced healthy links with a long send interval out of routing every cycle - validates heartbeat timing on the controller and the router: durations must be positive, their sum representable, and the timeout at least 10s and greater than sendInterval + checkInterval; a timeout under two heartbeat cycles is accepted with a warning - adds a per-config-type semantic validator registry to ConfigManager, run after schema validation, and moves the router.link config definition to common/config/routerlink so the controller applies the same rules - treats a link section in router YAML, of any content, as local config that takes precedence over the controller's for the whole router.link type, since the loader defaults every field and so now records presence separately; warns at startup when that section has no listeners or dialers - translates local heartbeat timing leniently, since a local failure stops the router from starting: keeps an explicit zero, raises a timeout below the floor to it, and falls back to the defaults for other invalid timing, each with a warning - stamps UpdateLinkListeners with the config generation it came from and has the controller drop superseded sets, which a multi-underlay control channel can deliver out of order and which would otherwise send peers to withdrawn addresses - logs every successful link config apply, including those that dispatch no work, and pins that a queue-size-only change triggers no GC pass - deprecates split links, documents the feature in the changelog and the router config design doc, and brings both up to date Builds on the channel/v5 HeartbeatControl from openziti/channel#265.

Commit:b90c912
Author:Paul Lorenz

Order peer state across controllers by the store index it rests on - stamps each peer state change with the sending controller's store index, so a router can tell an older statement about whether a router exists from a newer one, which nothing else sequenced - exposes the index as GetStoreIndex on the controller env, reporting zero without raft, where there is no index and one controller's sends to a router are already ordered - settles existence from the store before consulting the connected map, so a router deleted since it connected is never reported as reachable - reads the index after the existence check for a removal and before it for the other states, making a removal as strong and the rest as weak as each can truthfully be, since a router id is never reused - computes each router's state once per pass and reuses it, so every recipient hears the same statement and a delete landing mid-pass cannot split the fleet's view - remembers on the router, per destination, the index a removal was computed against, and ignores a report of that destination computed against an older index - keeps ordering to existence alone, so health from a controller behind on replication is still applied - tests the stamping order, the deleted-but-connected case, the one-statement-per-pass rule, and the router-side resurrection it all prevents

Commit:2608e99
Author:Paul Lorenz
Committer:Paul Lorenz

Replicate link state over gossip. Fixes #3726 Link state was every controller's own view, built from what routers reported to it directly. A router announces its links once per reconnect, so a controller that missed the announcement had no way to ask again, and controllers in a cluster disagreed about which links existed until something forced a reconnect. Replicating that state between controllers is what this changes. - adds a gossip protocol and replicated state store: per-owner entries with tombstones, digest exchange and anti-entropy, over the raft mesh - publishes link state, per-link latency and router canaries through it, with each state type gated on its own cluster capability so a part-way upgrade carries what it can and nothing it cannot - narrows the router metrics firehose to a single subscription controller once link latency travels over gossip, instead of a copy per controller - keeps a router's link reports out of gossip while it is not a publisher, and retracts what it published when it stops being one - gives up the whole store when the cluster stops supporting gossip, refusing entries and ignoring digests while that holds, so a downgraded cluster behaves like one that never had gossip Cross-version behaviour, which is most of the risk here: - derives gossip availability from raft membership, and says nothing about cluster capabilities until every member is known - tells routers which capabilities the whole cluster supports, so a router decides from the cluster rather than from whichever controller it asked - retires links for routers that do not gossip the pre-gossip way, since those routers re-announce on reconnect and nothing else would remove what they left - drops links to routers that no longer exist, from a router data model subscription on the router and a refusal on the controller Also here: - reports clock skew between gossip peers, and labels a link's conn state with the iteration its connections were read at - bounds and instruments the wait for a router connect setup - adds inspect targets and a validate-gossip command for the replicated state - adds a links-test model exercising link state at scale, across a version mix, through a controller upgrade cycle, with router delete and recreate churn

Commit:dddecd7
Author:Paul Lorenz
Committer:Paul Lorenz

Converge first-party CA public keys across controllers. Fixes #4407 Backport of #4405 to release-v2.0.x. For #4406. - adds a caPem field to the controller record, holding the intermediate CA certs from the controller's edge CA bundle, replicated through raft - has each controller report its own intermediates into its record on the existing self-update path - restricts leader-written peer record updates to hello-derived fields so a peer's caPem report is never overwritten - publishes each first-party root CA key with the deduplicated, byte-ordered union of intermediates from the local bundle and every controller record, so all controllers emit identical content per kid - republishes the root CA keys on controller record create, update and delete so late-joining and removed peers converge - removes a deleted controller's JWT validation key from the router data model - adds cert helpers for filtering intermediates and producing order-independent DER sets - gives each ha-3 test controller only its own signing intermediate and asserts the keys converge across all controllers and the router (cherry picked from commit 4a30738d9eaf2da7db4174f8c3ef1e81b0ce691d)

Commit:4a30738
Author:Paul Lorenz
Committer:Paul Lorenz

Converge first-party CA public keys across controllers. Fixes #4406 - adds a caPem field to the controller record, holding the intermediate CA certs from the controller's edge CA bundle, replicated through raft - has each controller report its own intermediates into its record on the existing self-update path - restricts leader-written peer record updates to hello-derived fields so a peer's caPem report is never overwritten - publishes each first-party root CA key with the deduplicated, byte-ordered union of intermediates from the local bundle and every controller record, so all controllers emit identical content per kid - republishes the root CA keys on controller record create, update and delete so late-joining and removed peers converge - removes a deleted controller's JWT validation key from the router data model - adds cert helpers for filtering intermediates and producing order-independent DER sets - gives each ha-3 test controller only its own signing intermediate and asserts the keys converge across all controllers and the router - syncs zititest module dependencies with the root module

Commit:a0e53e2
Author:Paul Lorenz
Committer:Paul Lorenz

Auto-GC stale router links. Fixes #4005 - adds a structured xlink.LinkKey ({DialerBinding, Protocol, DestId, ListenerBinding}) used by all staleness and GC checks, replacing the pre-rendered key string each side used to pass around - adds the CheckStaleLinks ctrl message and the `ziti ops verify stale-links` CLI for operator-driven, two-sided staleness verification with optional --gc, which acts only when both endpoints agree - implements auto-GC via the router.link.v1 gcMode (preserve/orphaned/changed): the router walks its xlinks after each Apply that mutates listeners, dialers, or gcMode and closes one-sided-stale entries under the configured mode - sweeps a removal under orphaned whatever the removed config said, including preserve, since a removal leaves no listeners and no dialers and every remaining link is one the router can never re-establish - reports staleness as a three-state verdict (unknown/notStale/stale) so a side that cannot judge a link abstains instead of guessing; a destination absent from the registry's listener snapshot means "listeners unknown", not "peer has none", which keeps a briefly unhealthy peer from losing its live links - rebuilds the link surface only when a listener or dialer definition actually moves, so a gcMode-only change no longer rebinds every listen socket - gates CheckStaleLinks on a new RouterStaleLinkCheck capability and explains each unqueryable endpoint, including one left unasked by the filter, so a partial verdict says which of the three causes applies - detects a peer re-advertising the address a link was dialed to from the dialing side, not just from the peer's own listener side - pins every verdict to the link iteration it was computed against, faults that iteration rather than the current one, and takes the conditional removal's result as the decision, so a link re-dialed while the sweep was running is neither closed nor reported as collected - resolves which side a report speaks for from the sending router against the controller's link map rather than from the side the report names, so one router cannot fill both sides and satisfy the both-agree rule --gc depends on - bounds the query fan-out with a semaphore, matching the other validation sweeps, since each query carries a minute-long reply timeout - fails the CLI when the result stream ends before every link is reported, and drains results already buffered before declaring any missing - rejects an unrecognized match mode rather than resolving it to changed, the broadest removal criterion - consolidates the controller-managed router configuration changelog into one section, replacing the status that said routers do not yet consume configs at runtime, and documents the per-config-type opt-in - restores the Current Beta Features section, dropped when the 2.0 changelog was archived, and lists controller-managed router configuration in it - adds unparseable to the codespell ignore list, a spelling variant the dictionary flags as a typo For #3743.

Commit:8f72323
Author:Christopher Britton
Committer:GitHub

[Backport-2.0] Replicate admin MFA removal across HA cluster members (#4295) * Replicate admin MFA removal across HA cluster members (#4284) * Run admin MFA removal as a single raft command so a lagging follower cannot serve a partial delete * Update CHANGELOG

Commit:6e5ea1f
Author:Christopher Britton
Committer:Christopher Britton

Replicate admin MFA removal across HA cluster members (#4284) * Run admin MFA removal as a single raft command so a lagging follower cannot serve a partial delete

Commit:dc777e0
Author:Paul Lorenz
Committer:Paul Lorenz

Report clock skew between gossip peers A tombstone now carries the deadline it will be reaped on, so expiry depends on wall clocks one node writes and another reads. Skew is bounded either way and cannot lose or corrupt state, but it does shift when tombstones are collected, and nothing made that visible. - stamps the sender's clock on the anti-entropy digest and on the canary. Those are the two axes a deadline crosses, since routers create tombstones as well as controllers, and both messages already run on a schedule - warns when the difference reaches a tenth of a tombstone lifetime, with a floor so a short lifetime does not make ordinary transit a warning. Expressed against the lifetime because that is what makes skew matter, so retuning one retunes the other - reports at most once per peer per ten minutes. Skew persists, so an unbounded report is a line per exchange for as long as the clocks disagree - names the direction, since a peer ahead of us stamps deadlines we collect late and a peer behind us stamps ones we collect early or that arrive already past It reports and corrects nothing. A one-way delta is skew plus transit and queueing, so it bounds skew rather than measuring it: enough to see the seconds to minutes that shift collection, and unfit to act on. Acting on it would trade a visible bounded problem for an invisible one.

Commit:9595e10
Author:Paul Lorenz
Committer:Paul Lorenz

Replicate link state over gossip. Fixes #3726 A router reported its links to every controller, and each controller kept its own picture built only from what routers told it directly. That does not survive routers being connected to a subset of controllers: a controller learns nothing about links whose routers it does not hold a connection to. Link state now lives in the replicated store: a router reports to one controller, that controller writes the entry it owns, and the mesh carries it to the rest. Each link entry is owned by the router that dialled it, so two controllers never contend for the same key, and a controller that has never spoken to a router still converges on its links. - registers a link state type on the gossip store and carries link add, update and removal through it - makes a link's source router an atomic and repoints it when the router connects, since a link can be built from a gossiped entry before its router has connected here, leaving a database-loaded placeholder as the endpoint - reconciles a reconnecting router's gossip entries, marking its links usable again rather than removing them, since a disconnect sets them down instead of deleting them - tombstones a link on disconnect in single-controller mode, where there is no peer to learn the removal from - adds the gossip transport: peer handlers on the controller mesh, router-facing gossip handlers, digest exchange off the receive goroutine, and the pools that bound apply and I/O work - has the digest exchange restamp a key the controller holds a higher version for, above that version, and send the live value. A router's Lamport clock is in memory, so a restart returns it to zero while the controller still holds versions from the previous incarnation under the same key. Link metrics are keyed by link id alone, and that id belongs to the dialer, so an acceptor's restart leaves the key unchanged and its republishes are refused as older. Keeping the stored version sends nothing, and every later digest reaches the same answer, so the exchange that exists to repair divergence would instead hold it in place. Safe because the router is the sole writer of the keys it advertises: it takes the clock from a digest but never a value - advertises a gossip capability so a router reports to one controller only once every controller can replicate, and falls back to reporting to all until then - adds canaries, a per-router sequence carried over the same path, so a router can tell that a controller has stopped applying its state - carries link metrics over gossip alongside the state - keeps the disconnect teardown's reroute ordering: the currency guard wraps it, and inside, the link snapshot and MarkDisconnected stay ahead of the cascade so reroute cannot path through the router being removed

Commit:1d4c666
Author:Paul Lorenz
Committer:Paul Lorenz

Add a gossip protocol and replicated state store A controller cluster needs state that every member converges on without any member being authoritative for it, and without a full exchange every time something changes. This adds that as a standalone package, with no notion of what is being replicated. Each entry is owned by the peer that wrote it and carries a version from that owner's Lamport clock, so concurrent writes from different owners never contend and a late-arriving older write loses to the newer one it raced. Deletes are tombstones with a TTL, since an absent key and a deleted key are otherwise indistinguishable to a peer that has not seen the delete yet. Steady state is deltas: a change is broadcast once to the mesh rather than once per peer per entry. Anti-entropy covers what a delta missed, opening a round with per-owner hashes so two peers already in agreement exchange a hash rather than every entry version, and descending to entry versions only for the owners whose hashes differ. Each process gets an epoch, a UUIDv7 minted at startup, which gates deletion only: entries from a peer's previous lifetime can be swept when it comes back, while ordering stays with the version. Sweeping is conditional on the entry that was selected still being the entry there, so a replacement that lands mid-sweep is not deleted by it. - adds controller/gossip: the store, owner clocks, tombstones and their reaper, delta broadcast, anti-entropy with owner hashes, epochs, and the mesh abstraction it sends over - has a tombstone carry the deadline it will be reaped on, decided once by whoever created it. Expiring on a locally recorded arrival time instead makes anti-entropy self-defeating: reapers fire on independent tickers, the hash the exchange compares deliberately covers tombstones, so the node that reaps first is sent its tombstone back and starts it over. The two hand it to each other a TTL at a time and the owner's data is never compacted, which is the leak the reaper exists to prevent - bounds both directions of clock skew, since the deadline is now a wall clock one node sets and another reads. A deadline further out than one TTL is clamped on arrival, so a fast clock cannot mint a tombstone nothing ever reaps. An expired one is still applied rather than refused, so a slow clock cannot have its removals dropped on arrival and leave the entry it deletes in place; it is reaped on the next pass instead. Skew shifts when a tombstone goes and cannot lose or corrupt state - adds gossip_pb and the ctrl/mgmt message types the transport and inspection need - is transport-agnostic and carries no link, terminator or other domain concept; callers register a state type and supply their own encoding

Commit:030e048
Author:Paul Lorenz
Committer:Paul Lorenz

Auto-GC stale router links. Fixes #4005 - adds a structured xlink.LinkKey ({DialerBinding, Protocol, DestId, ListenerBinding}) used by all staleness and GC checks, replacing the pre-rendered key string each side used to pass around - adds the CheckStaleLinks ctrl message and the `ziti ops verify stale-links` CLI for operator-driven, two-sided staleness verification with optional --gc, which acts only when both endpoints agree - implements auto-GC via the router.link.v1 gcMode (preserve/orphaned/changed): the router walks its xlinks after each Apply that mutates listeners, dialers, or gcMode and closes one-sided-stale entries under the configured mode - reports staleness as a three-state verdict (unknown/notStale/stale) so a side that cannot judge a link abstains instead of guessing; a destination absent from the registry's listener snapshot means "listeners unknown", not "peer has none", which keeps a briefly unhealthy peer from losing its live links - rebuilds the link surface only when a listener or dialer definition actually moves, so a gcMode-only change no longer rebinds every listen socket - gates CheckStaleLinks on a new RouterStaleLinkCheck capability and explains each unqueryable endpoint, so a partial verdict distinguishes an offline peer from one too old to answer - detects a peer re-advertising the address a link was dialed to from the dialing side, not just from the peer's own listener side - pins every verdict to the link iteration it was computed against, faults that iteration rather than the current one, and takes the conditional removal's result as the decision, so a link re-dialed while the sweep was running is neither closed nor reported as collected - returns whether LinkManager.Remove actually removed, rather than discarding the link table's iteration guard - fails the CLI when the result stream ends before every link is reported, and drains results already buffered before declaring any missing, so neither a truncated run nor a random select outcome is read as a clean sweep - rejects an unrecognized match mode rather than resolving it to changed, the broadest removal criterion For #3743.

Commit:a7e8e01
Author:Paul Lorenz

Auto-GC stale router links. Fixes #4005 - adds a structured xlink.LinkKey ({DialerBinding, Protocol, DestId, ListenerBinding}) used by all staleness and GC checks, replacing the pre-rendered key string each side used to pass around - adds the CheckStaleLinks ctrl message and the `ziti ops verify stale-links` CLI for operator-driven, two-sided staleness verification with optional --gc, which acts only when both endpoints agree - implements auto-GC via the router.link.v1 gcMode (preserve/orphaned/changed): the router walks its xlinks after each Apply that mutates listeners, dialers, or gcMode and closes one-sided-stale entries under the configured mode - reports staleness as a three-state verdict (unknown/notStale/stale) so a side that cannot judge a link abstains instead of guessing; a destination absent from the registry's listener snapshot means "listeners unknown", not "peer has none", which keeps a briefly unhealthy peer from losing its live links - rebuilds the link surface only when a listener or dialer definition actually moves, so a gcMode-only change no longer rebinds every listen socket - gates CheckStaleLinks on a new RouterStaleLinkCheck capability and explains each unqueryable endpoint, so a partial verdict distinguishes an offline peer from one too old to answer - detects a peer re-advertising the address a link was dialed to from the dialing side, not just from the peer's own listener side For #3743.

Commit:8c919df
Author:Andrew Martinez
Committer:GitHub

backport openziti/ziti#4094 to release-v2.0.x accept first-party certs issued by a separate edge signing CA (#4153) - publishes FirstPartyX509CertValidation/ThirdPartyX509CertValidation usages and intermediates on router data model public keys, deprecating ClientX509CertValidation - builds the router first-party cert pool from RDM first-party keys unioned with ctrl-channel roots; TLS and VerifyClientCert paths share buildClientCertRoots with fallback to the deprecated usage for old controllers - trusts the edge enrollment signing CA when verifying the certificate a router presents on the control channel, so a signing CA outside the controller's own trust bundle no longer refuses every router; the anchors go into a clone of the identity's pool, never the pool its live tls.Configs share - propagates full controller signing cert chains over the mesh via SigningCertChainHeader and persists them in Controller store CertPem - sends stored public keys during router sync instead of rebuilding them; publishes controller certs leaf-only - stops router controller reconnect loops after shutdown - gives each in-process controller its own command decoder registry - adds the ha-3 three-controller harness and first-party cert integration tests - drains the cli test stdout pipe while commands run; anchors the totp token issued-at assertion to the test clock - backports the SPIFFE-capable test PKI from openziti/ziti#3947: --not-before on ziti pki create, tests/testdata/create-pki.sh/.ps1, and the generated PKI under tests/testdata/pki including the separate edge signing root and per-controller signing intermediates; existing config sets stay on the testdata/ca PKI - skips *.pem, *.cert and *.key files in codespell

Commit:948735d
Author:Andrew Martinez
Committer:GitHub

fixes openziti/ziti#4094 accept first-party certs issued by a separat… (#4140) * fixes openziti/ziti#4094 accept first-party certs issued by a separate edge signing CA - adds FirstPartyX509CertValidation and ThirdPartyX509CertValidation usages and an intermediates field to the router data model public keys, deprecating ClientX509CertValidation - publishes config CA bundle roots as first-party anchors with their intermediates and Ca store entries as third-party anchors; controller certs carry JWT validation only, since a controller identity is never a CA and anchors no client cert chains - builds router first-party and client cert trust pools from the published usages, falling back to the deprecated usage against older controllers - propagates the full signing cert chain between controllers via a new mesh SigningCertChainHeader and persists whole chains in controller records - removes the orphaned InstantStrategy.AddPublicKey, dead since public key sync moved to controller list data - gives each command dispatcher its own decoder registry so multiple in-process controllers no longer decode into the last-started controller's managers - adds a three-controller in-process HA test harness with a split signing PKI, cluster formation and first-party cert integration tests - trusts the edge signing CA when verifying router control channel certs - adds a variadic additionalRoots parameter to VerifyLeafCertChain, applied to a clone of the caller's pool so an identity's live tls.Configs are unaffected - passes the edge enrollment signing CA bundle as additional roots when admitting a router control channel connection, so a deployment whose signing CA sits outside the controller's own trust bundle no longer has every router refused - leaves the fingerprint check bound to the verified leaf, so the wider anchor set changes which chains verify, not which routers are admitted - covers the split-root case and the caller-pool guarantee in common/cert tests

Commit:576bf7d
Author:Paul Lorenz
Committer:Paul Lorenz

Auto-GC stale router links. Fixes #4005 - adds a structured xlink.LinkKey ({DialerBinding, Protocol, DestId, ListenerBinding}) used by all staleness and GC checks - adds the CheckStaleLinks ctrl message and the `ziti ops verify stale-links` CLI for operator-driven, two-sided staleness verification with optional --gc - implements auto-GC via the router.link.v1 gcMode (preserve/orphaned/changed): the router walks its xlinks after each Apply that mutates listeners, dialers, or gcMode and closes one-sided-stale entries under the configured mode For #3743.

Commit:08ac26d
Author:Christopher Britton
Committer:Christopher Britton

Replicate admin MFA removal across HA cluster members (#4284) * Run admin MFA removal as a single raft command so a lagging follower cannot serve a partial delete

Commit:c639518
Author:Christopher Britton
Committer:GitHub

Replicate admin MFA removal across HA cluster members (#4284) * Run admin MFA removal as a single raft command so a lagging follower cannot serve a partial delete

Commit:5857168
Author:Christopher Britton

Run admin MFA removal as a single raft command so a lagging follower cannot serve a partial delete

Commit:8b93c7d
Author:Paul Lorenz
Committer:Paul Lorenz

Auto-GC stale router links. Fixes #4005 - adds a structured xlink.LinkKey ({DialerBinding, Protocol, DestId, ListenerBinding}) used by all staleness and GC checks - adds the CheckStaleLinks ctrl message and the `ziti ops verify stale-links` CLI for operator-driven, two-sided staleness verification with optional --gc - implements auto-GC via the router.link.v1 gcMode (preserve/orphaned/changed): the router walks its xlinks after each Apply that mutates listeners, dialers, or gcMode and closes one-sided-stale entries under the configured mode For #3743.

Commit:b072268
Author:Paul Lorenz
Committer:Paul Lorenz

Replicate link state over gossip. Fixes #3726 A router reported its links to every controller, and each controller kept its own picture built only from what routers told it directly. That does not survive routers being connected to a subset of controllers: a controller learns nothing about links whose routers it does not hold a connection to. Link state now lives in the replicated store: a router reports to one controller, that controller writes the entry it owns, and the mesh carries it to the rest. Each link entry is owned by the router that dialled it, so two controllers never contend for the same key, and a controller that has never spoken to a router still converges on its links. - registers a link state type on the gossip store and carries link add, update and removal through it - makes a link's source router an atomic and repoints it when the router connects, since a link can be built from a gossiped entry before its router has connected here, leaving a database-loaded placeholder as the endpoint - reconciles a reconnecting router's gossip entries, marking its links usable again rather than removing them, since a disconnect sets them down instead of deleting them - tombstones a link on disconnect in single-controller mode, where there is no peer to learn the removal from - adds the gossip transport: peer handlers on the controller mesh, router-facing gossip handlers, digest exchange off the receive goroutine, and the pools that bound apply and I/O work - advertises a gossip capability so a router reports to one controller only once every controller can replicate, and falls back to reporting to all until then - adds canaries, a per-router sequence carried over the same path, so a router can tell that a controller has stopped applying its state - carries link metrics over gossip alongside the state - keeps the disconnect teardown's reroute ordering: the currency guard wraps it, and inside, the link snapshot and MarkDisconnected stay ahead of the cascade so reroute cannot path through the router being removed

Commit:24f5056
Author:Paul Lorenz
Committer:Paul Lorenz

Add a gossip protocol and replicated state store A controller cluster needs state that every member converges on without any member being authoritative for it, and without a full exchange every time something changes. This adds that as a standalone package, with no notion of what is being replicated. Each entry is owned by the peer that wrote it and carries a version from that owner's Lamport clock, so concurrent writes from different owners never contend and a late-arriving older write loses to the newer one it raced. Deletes are tombstones with a TTL, since an absent key and a deleted key are otherwise indistinguishable to a peer that has not seen the delete yet. Steady state is deltas: a change is broadcast once to the mesh rather than once per peer per entry. Anti-entropy covers what a delta missed, opening a round with per-owner hashes so two peers already in agreement exchange a hash rather than every entry version, and descending to entry versions only for the owners whose hashes differ. Each process gets an epoch, a UUIDv7 minted at startup, which gates deletion only: entries from a peer's previous lifetime can be swept when it comes back, while ordering stays with the version. Sweeping is conditional on the entry that was selected still being the entry there, so a replacement that lands mid-sweep is not deleted by it. - adds controller/gossip: the store, owner clocks, tombstones and their reaper, delta broadcast, anti-entropy with owner hashes, epochs, and the mesh abstraction it sends over - adds gossip_pb and the ctrl/mgmt message types the transport and inspection need - is transport-agnostic and carries no link, terminator or other domain concept; callers register a state type and supply their own encoding

Commit:02a80d0
Author:Paul Lorenz
Committer:GitHub

Auto-GC stale router links. Fixes #4005 - adds a structured xlink.LinkKey ({DialerBinding, Protocol, DestId, ListenerBinding}) used by all staleness and GC checks - adds the CheckStaleLinks ctrl message and the `ziti ops verify stale-links` CLI for operator-driven, two-sided staleness verification with optional --gc - implements auto-GC via the router.link.v1 gcMode (preserve/orphaned/changed): the router walks its xlinks after each Apply that mutates listeners, dialers, or gcMode and closes one-sided-stale entries under the configured mode For #3743.

Commit:48e4d4a
Author:Paul Lorenz
Committer:Paul Lorenz

Manage router link configuration via the controller. Fixes #4004 - adds a link ConfigHandler (router/link FactoryRegistry) that applies router.link.v1 config: Apply rebuilds the listener/dialer set wholesale, and established Xlinks survive because Listener.Close() only closes the accept loop - translates local link: YAML into router.link.v1 JSON and pushes it through the managed-config registry at startup - adds the UpdateLinkListeners ctrl message so the router republishes its listener set to the controller on change; the controller re-fans via the existing PeerStateChange path - re-evaluates dialers on link group and listener changes via RescanForDialOpportunities For #3743.

Commit:a071cd3
Author:Paul Lorenz
Committer:Paul Lorenz

Add leader fast-path for confirmed no-op terminator removals. For #4156 - routes V2 terminator removes to the leader (GetModelUpdateCtrlChannel), where the delete is applied anyway and the fast-path can consult current state - adds a createConfirmed flag (index-aligned with terminatorIds) to RemoveTerminatorsV2Request, set once the router receives the create acknowledgement; backwards compatible, older peers omit it and it decodes as false - on the leader, skips the raft/rate-limited delete for a terminator the router confirms was created but that no longer exists, so a storm of no-op retries can't consume the command rate limiter; unconfirmed or still-present ids take the normal ordered path - tests the fast-path filter across leader/confirmed/present/error and mixed cases

Commit:a76b537
Author:Paul Lorenz
Committer:Paul Lorenz

Add asynchronous terminator removal (RemoveTerminatorsV2). For #4156 - adds RemoveTerminatorsV2Request/Response control-channel messages, answered asynchronously so a router no longer holds a request slot open waiting for a synchronous removal reply - adds the ControllerAsyncTerminatorRemove capability so routers use the async path only against controllers that advertise support, falling back to synchronous removal otherwise - adds a controller handler that removes the batch and replies with per-batch success/rate-limited status - tracks in-flight removal batches on the router and, when the response arrives, resolves the batch's rate-limit control and clears or requeues its terminators; lost responses self-heal via the existing rate-limiter expiry and retry scan

Commit:7c6a2a9
Author:Paul Lorenz
Committer:Paul Lorenz

Replicate link state over gossip. Fixes #3726 A router reported its links to every controller, and each controller kept its own picture built only from what routers told it directly. That does not survive routers being connected to a subset of controllers: a controller learns nothing about links whose routers it does not hold a connection to. Link state now lives in the replicated store: a router reports to one controller, that controller writes the entry it owns, and the mesh carries it to the rest. Each link entry is owned by the router that dialled it, so two controllers never contend for the same key, and a controller that has never spoken to a router still converges on its links. - registers a link state type on the gossip store and carries link add, update and removal through it - makes a link's source router an atomic and repoints it when the router connects, since a link can be built from a gossiped entry before its router has connected here, leaving a database-loaded placeholder as the endpoint - reconciles a reconnecting router's gossip entries, marking its links usable again rather than removing them, since a disconnect sets them down instead of deleting them - tombstones a link on disconnect in single-controller mode, where there is no peer to learn the removal from - adds the gossip transport: peer handlers on the controller mesh, router-facing gossip handlers, digest exchange off the receive goroutine, and the pools that bound apply and I/O work - advertises a gossip capability so a router reports to one controller only once every controller can replicate, and falls back to reporting to all until then - adds canaries, a per-router sequence carried over the same path, so a router can tell that a controller has stopped applying its state - carries link metrics over gossip alongside the state - keeps the disconnect teardown's reroute ordering: the currency guard wraps it, and inside, the link snapshot and MarkDisconnected stay ahead of the cascade so reroute cannot path through the router being removed

Commit:c3bf72d
Author:Paul Lorenz
Committer:Paul Lorenz

Add a gossip protocol and replicated state store A controller cluster needs state that every member converges on without any member being authoritative for it, and without a full exchange every time something changes. This adds that as a standalone package, with no notion of what is being replicated. Each entry is owned by the peer that wrote it and carries a version from that owner's Lamport clock, so concurrent writes from different owners never contend and a late-arriving older write loses to the newer one it raced. Deletes are tombstones with a TTL, since an absent key and a deleted key are otherwise indistinguishable to a peer that has not seen the delete yet. Steady state is deltas: a change is broadcast once to the mesh rather than once per peer per entry. Anti-entropy covers what a delta missed, opening a round with per-owner hashes so two peers already in agreement exchange a hash rather than every entry version, and descending to entry versions only for the owners whose hashes differ. Each process gets an epoch, a UUIDv7 minted at startup, which gates deletion only: entries from a peer's previous lifetime can be swept when it comes back, while ordering stays with the version. Sweeping is conditional on the entry that was selected still being the entry there, so a replacement that lands mid-sweep is not deleted by it. - adds controller/gossip: the store, owner clocks, tombstones and their reaper, delta broadcast, anti-entropy with owner hashes, epochs, and the mesh abstraction it sends over - adds gossip_pb and the ctrl/mgmt message types the transport and inspection need - is transport-agnostic and carries no link, terminator or other domain concept; callers register a state type and supply their own encoding

Commit:c4f992e
Author:Paul Lorenz
Committer:GitHub

Auto-GC stale router links. Fixes #4005 - adds a structured xlink.LinkKey ({DialerBinding, Protocol, DestId, ListenerBinding}) used by all staleness and GC checks - adds the CheckStaleLinks ctrl message and the `ziti ops verify stale-links` CLI for operator-driven, two-sided staleness verification with optional --gc - implements auto-GC via the router.link.v1 gcMode (preserve/orphaned/changed): the router walks its xlinks after each Apply that mutates listeners, dialers, or gcMode and closes one-sided-stale entries under the configured mode For #3743.

Commit:a43811e
Author:Paul Lorenz
Committer:GitHub

Manage router link configuration via the controller. Fixes #4004 - adds a link ConfigHandler (router/link FactoryRegistry) that applies router.link.v1 config: Apply rebuilds the listener/dialer set wholesale, and established Xlinks survive because Listener.Close() only closes the accept loop - translates local link: YAML into router.link.v1 JSON and pushes it through the managed-config registry at startup - adds the UpdateLinkListeners ctrl message so the router republishes its listener set to the controller on change; the controller re-fans via the existing PeerStateChange path - re-evaluates dialers on link group and listener changes via RescanForDialOpportunities For #3743.

Commit:ea92d78
Author:Paul Lorenz
Committer:GitHub

Merge pull request #4216 from openziti/router-link-costtags-removal Remove unused link costTags support

Commit:f9600f2
Author:Paul Lorenz
Committer:Paul Lorenz

Replicate link state over gossip. Fixes #3726 A router reported its links to every controller, and each controller kept its own picture built only from what routers told it directly. That does not survive routers being connected to a subset of controllers: a controller learns nothing about links whose routers it does not hold a connection to. Link state now lives in the replicated store: a router reports to one controller, that controller writes the entry it owns, and the mesh carries it to the rest. Each link entry is owned by the router that dialled it, so two controllers never contend for the same key, and a controller that has never spoken to a router still converges on its links. - registers a link state type on the gossip store and carries link add, update and removal through it - makes a link's source router an atomic and repoints it when the router connects, since a link can be built from a gossiped entry before its router has connected here, leaving a database-loaded placeholder as the endpoint - reconciles a reconnecting router's gossip entries, marking its links usable again rather than removing them, since a disconnect sets them down instead of deleting them - tombstones a link on disconnect in single-controller mode, where there is no peer to learn the removal from - adds the gossip transport: peer handlers on the controller mesh, router-facing gossip handlers, digest exchange off the receive goroutine, and the pools that bound apply and I/O work - advertises a gossip capability so a router reports to one controller only once every controller can replicate, and falls back to reporting to all until then - adds canaries, a per-router sequence carried over the same path, so a router can tell that a controller has stopped applying its state - carries link metrics over gossip alongside the state - keeps the disconnect teardown's reroute ordering: the currency guard wraps it, and inside, the link snapshot and MarkDisconnected stay ahead of the cascade so reroute cannot path through the router being removed

Commit:d30d35b
Author:Paul Lorenz
Committer:Paul Lorenz

Add a gossip protocol and replicated state store A controller cluster needs state that every member converges on without any member being authoritative for it, and without a full exchange every time something changes. This adds that as a standalone package, with no notion of what is being replicated. Each entry is owned by the peer that wrote it and carries a version from that owner's Lamport clock, so concurrent writes from different owners never contend and a late-arriving older write loses to the newer one it raced. Deletes are tombstones with a TTL, since an absent key and a deleted key are otherwise indistinguishable to a peer that has not seen the delete yet. Steady state is deltas: a change is broadcast once to the mesh rather than once per peer per entry. Anti-entropy covers what a delta missed, opening a round with per-owner hashes so two peers already in agreement exchange a hash rather than every entry version, and descending to entry versions only for the owners whose hashes differ. Each process gets an epoch, a UUIDv7 minted at startup, which gates deletion only: entries from a peer's previous lifetime can be swept when it comes back, while ordering stays with the version. Sweeping is conditional on the entry that was selected still being the entry there, so a replacement that lands mid-sweep is not deleted by it. - adds controller/gossip: the store, owner clocks, tombstones and their reaper, delta broadcast, anti-entropy with owner hashes, epochs, and the mesh abstraction it sends over - adds gossip_pb and the ctrl/mgmt message types the transport and inspection need - is transport-agnostic and carries no link, terminator or other domain concept; callers register a state type and supply their own encoding

Commit:1a58248
Author:Paul Lorenz
Committer:Paul Lorenz

Persist cluster id across migration snapshot restore. For #4104 - adds a clusterId field to SyncSnapshotCommand and writes it into the database after the migration snapshot restore, so a controller bootstrapped by migrating a database ends up with a durable cluster id instead of an empty one - the snapshot restore replaces the whole FSM database with the migration source, which carries no cluster id, so without this the id written during bootstrap was silently wiped and the node came up with an empty, non-durable cluster id, defeating the mesh cluster-id validation - persists the raft index after the cluster id in RestoreSnapshot so the index remains the completion gate: a failure before it halts (SyncSnapshotCommand is a critical command) and replays/retries on restart rather than skipping the command with a blank cluster id - fails RaftRestoreFromBoltDb when the cluster id is blank after bootstrap - regenerates cmd.pb.go for the new field (cherry picked from commit d8cedeb8ea1813221808f3da556a8fc59efae431)

Commit:f3fb011
Author:Paul Lorenz
Committer:GitHub

Merge pull request #4105 from openziti/ha-cluster-bootstrap-guards Controller cluster bootstrapping fixes

Commit:74e0be0
Author:Paul Lorenz
Committer:Paul Lorenz

Auto-GC stale router links. Fixes #4005 - adds a structured xlink.LinkKey ({DialerBinding, Protocol, DestId, ListenerBinding}) used by all staleness and GC checks - adds the CheckStaleLinks ctrl message and the `ziti ops verify stale-links` CLI for operator-driven, two-sided staleness verification with optional --gc - implements auto-GC via the router.link.v1 gcMode (preserve/orphaned/changed): the router walks its xlinks after each Apply that mutates listeners, dialers, or gcMode and closes one-sided-stale entries under the configured mode For #3743.

Commit:e4f2d18
Author:Paul Lorenz
Committer:Paul Lorenz

Manage router link configuration via the controller. Fixes #4004 - adds a link ConfigHandler (router/link FactoryRegistry) that applies router.link.v1 config: Apply rebuilds the listener/dialer set wholesale, and established Xlinks survive because Listener.Close() only closes the accept loop - translates local link: YAML into router.link.v1 JSON and pushes it through the managed-config registry at startup - adds the UpdateLinkListeners ctrl message so the router republishes its listener set to the controller on change; the controller re-fans via the existing PeerStateChange path - re-evaluates dialers on link group and listener changes via RescanForDialOpportunities For #3743.

Commit:b80ba43
Author:Paul Lorenz
Committer:Paul Lorenz

Add leader fast-path for confirmed no-op terminator removals. For #4156 - routes V2 terminator removes to the leader (GetModelUpdateCtrlChannel), where the delete is applied anyway and the fast-path can consult current state - adds a createConfirmed flag (index-aligned with terminatorIds) to RemoveTerminatorsV2Request, set once the router receives the create acknowledgement; backwards compatible, older peers omit it and it decodes as false - on the leader, skips the raft/rate-limited delete for a terminator the router confirms was created but that no longer exists, so a storm of no-op retries can't consume the command rate limiter; unconfirmed or still-present ids take the normal ordered path - tests the fast-path filter across leader/confirmed/present/error and mixed cases

Commit:700a795
Author:Paul Lorenz
Committer:Paul Lorenz

Add asynchronous terminator removal (RemoveTerminatorsV2). For #4156 - adds RemoveTerminatorsV2Request/Response control-channel messages, answered asynchronously so a router no longer holds a request slot open waiting for a synchronous removal reply - adds the ControllerAsyncTerminatorRemove capability so routers use the async path only against controllers that advertise support, falling back to synchronous removal otherwise - adds a controller handler that removes the batch and replies with per-batch success/rate-limited status - tracks in-flight removal batches on the router and, when the response arrives, resolves the batch's rate-limit control and clears or requeues its terminators; lost responses self-heal via the existing rate-limiter expiry and retry scan