These commits are when the Protocol Buffers files have changed: (only the last 100 relevant commits are shown)
| Commit: | 5857168 | |
|---|---|---|
| Author: | Christopher Britton | |
Run admin MFA removal as a single raft command so a lagging follower cannot serve a partial delete
| Commit: | 8b93c7d | |
|---|---|---|
| Author: | Paul Lorenz | |
| Committer: | Paul Lorenz | |
Auto-GC stale router links. Fixes #4005 - adds a structured xlink.LinkKey ({DialerBinding, Protocol, DestId, ListenerBinding}) used by all staleness and GC checks - adds the CheckStaleLinks ctrl message and the `ziti ops verify stale-links` CLI for operator-driven, two-sided staleness verification with optional --gc - implements auto-GC via the router.link.v1 gcMode (preserve/orphaned/changed): the router walks its xlinks after each Apply that mutates listeners, dialers, or gcMode and closes one-sided-stale entries under the configured mode For #3743.
| Commit: | b072268 | |
|---|---|---|
| Author: | Paul Lorenz | |
| Committer: | Paul Lorenz | |
Replicate link state over gossip. Fixes #3726 A router reported its links to every controller, and each controller kept its own picture built only from what routers told it directly. That does not survive routers being connected to a subset of controllers: a controller learns nothing about links whose routers it does not hold a connection to. Link state now lives in the replicated store: a router reports to one controller, that controller writes the entry it owns, and the mesh carries it to the rest. Each link entry is owned by the router that dialled it, so two controllers never contend for the same key, and a controller that has never spoken to a router still converges on its links. - registers a link state type on the gossip store and carries link add, update and removal through it - makes a link's source router an atomic and repoints it when the router connects, since a link can be built from a gossiped entry before its router has connected here, leaving a database-loaded placeholder as the endpoint - reconciles a reconnecting router's gossip entries, marking its links usable again rather than removing them, since a disconnect sets them down instead of deleting them - tombstones a link on disconnect in single-controller mode, where there is no peer to learn the removal from - adds the gossip transport: peer handlers on the controller mesh, router-facing gossip handlers, digest exchange off the receive goroutine, and the pools that bound apply and I/O work - advertises a gossip capability so a router reports to one controller only once every controller can replicate, and falls back to reporting to all until then - adds canaries, a per-router sequence carried over the same path, so a router can tell that a controller has stopped applying its state - carries link metrics over gossip alongside the state - keeps the disconnect teardown's reroute ordering: the currency guard wraps it, and inside, the link snapshot and MarkDisconnected stay ahead of the cascade so reroute cannot path through the router being removed
| Commit: | 24f5056 | |
|---|---|---|
| Author: | Paul Lorenz | |
| Committer: | Paul Lorenz | |
Add a gossip protocol and replicated state store A controller cluster needs state that every member converges on without any member being authoritative for it, and without a full exchange every time something changes. This adds that as a standalone package, with no notion of what is being replicated. Each entry is owned by the peer that wrote it and carries a version from that owner's Lamport clock, so concurrent writes from different owners never contend and a late-arriving older write loses to the newer one it raced. Deletes are tombstones with a TTL, since an absent key and a deleted key are otherwise indistinguishable to a peer that has not seen the delete yet. Steady state is deltas: a change is broadcast once to the mesh rather than once per peer per entry. Anti-entropy covers what a delta missed, opening a round with per-owner hashes so two peers already in agreement exchange a hash rather than every entry version, and descending to entry versions only for the owners whose hashes differ. Each process gets an epoch, a UUIDv7 minted at startup, which gates deletion only: entries from a peer's previous lifetime can be swept when it comes back, while ordering stays with the version. Sweeping is conditional on the entry that was selected still being the entry there, so a replacement that lands mid-sweep is not deleted by it. - adds controller/gossip: the store, owner clocks, tombstones and their reaper, delta broadcast, anti-entropy with owner hashes, epochs, and the mesh abstraction it sends over - adds gossip_pb and the ctrl/mgmt message types the transport and inspection need - is transport-agnostic and carries no link, terminator or other domain concept; callers register a state type and supply their own encoding
| Commit: | 02a80d0 | |
|---|---|---|
| Author: | Paul Lorenz | |
| Committer: | GitHub | |
Auto-GC stale router links. Fixes #4005 - adds a structured xlink.LinkKey ({DialerBinding, Protocol, DestId, ListenerBinding}) used by all staleness and GC checks - adds the CheckStaleLinks ctrl message and the `ziti ops verify stale-links` CLI for operator-driven, two-sided staleness verification with optional --gc - implements auto-GC via the router.link.v1 gcMode (preserve/orphaned/changed): the router walks its xlinks after each Apply that mutates listeners, dialers, or gcMode and closes one-sided-stale entries under the configured mode For #3743.
| Commit: | 48e4d4a | |
|---|---|---|
| Author: | Paul Lorenz | |
| Committer: | Paul Lorenz | |
Manage router link configuration via the controller. Fixes #4004 - adds a link ConfigHandler (router/link FactoryRegistry) that applies router.link.v1 config: Apply rebuilds the listener/dialer set wholesale, and established Xlinks survive because Listener.Close() only closes the accept loop - translates local link: YAML into router.link.v1 JSON and pushes it through the managed-config registry at startup - adds the UpdateLinkListeners ctrl message so the router republishes its listener set to the controller on change; the controller re-fans via the existing PeerStateChange path - re-evaluates dialers on link group and listener changes via RescanForDialOpportunities For #3743.
| Commit: | a071cd3 | |
|---|---|---|
| Author: | Paul Lorenz | |
| Committer: | Paul Lorenz | |
Add leader fast-path for confirmed no-op terminator removals. For #4156 - routes V2 terminator removes to the leader (GetModelUpdateCtrlChannel), where the delete is applied anyway and the fast-path can consult current state - adds a createConfirmed flag (index-aligned with terminatorIds) to RemoveTerminatorsV2Request, set once the router receives the create acknowledgement; backwards compatible, older peers omit it and it decodes as false - on the leader, skips the raft/rate-limited delete for a terminator the router confirms was created but that no longer exists, so a storm of no-op retries can't consume the command rate limiter; unconfirmed or still-present ids take the normal ordered path - tests the fast-path filter across leader/confirmed/present/error and mixed cases
| Commit: | a76b537 | |
|---|---|---|
| Author: | Paul Lorenz | |
| Committer: | Paul Lorenz | |
Add asynchronous terminator removal (RemoveTerminatorsV2). For #4156 - adds RemoveTerminatorsV2Request/Response control-channel messages, answered asynchronously so a router no longer holds a request slot open waiting for a synchronous removal reply - adds the ControllerAsyncTerminatorRemove capability so routers use the async path only against controllers that advertise support, falling back to synchronous removal otherwise - adds a controller handler that removes the batch and replies with per-batch success/rate-limited status - tracks in-flight removal batches on the router and, when the response arrives, resolves the batch's rate-limit control and clears or requeues its terminators; lost responses self-heal via the existing rate-limiter expiry and retry scan
| Commit: | 7c6a2a9 | |
|---|---|---|
| Author: | Paul Lorenz | |
| Committer: | Paul Lorenz | |
Replicate link state over gossip. Fixes #3726 A router reported its links to every controller, and each controller kept its own picture built only from what routers told it directly. That does not survive routers being connected to a subset of controllers: a controller learns nothing about links whose routers it does not hold a connection to. Link state now lives in the replicated store: a router reports to one controller, that controller writes the entry it owns, and the mesh carries it to the rest. Each link entry is owned by the router that dialled it, so two controllers never contend for the same key, and a controller that has never spoken to a router still converges on its links. - registers a link state type on the gossip store and carries link add, update and removal through it - makes a link's source router an atomic and repoints it when the router connects, since a link can be built from a gossiped entry before its router has connected here, leaving a database-loaded placeholder as the endpoint - reconciles a reconnecting router's gossip entries, marking its links usable again rather than removing them, since a disconnect sets them down instead of deleting them - tombstones a link on disconnect in single-controller mode, where there is no peer to learn the removal from - adds the gossip transport: peer handlers on the controller mesh, router-facing gossip handlers, digest exchange off the receive goroutine, and the pools that bound apply and I/O work - advertises a gossip capability so a router reports to one controller only once every controller can replicate, and falls back to reporting to all until then - adds canaries, a per-router sequence carried over the same path, so a router can tell that a controller has stopped applying its state - carries link metrics over gossip alongside the state - keeps the disconnect teardown's reroute ordering: the currency guard wraps it, and inside, the link snapshot and MarkDisconnected stay ahead of the cascade so reroute cannot path through the router being removed
| Commit: | c3bf72d | |
|---|---|---|
| Author: | Paul Lorenz | |
| Committer: | Paul Lorenz | |
Add a gossip protocol and replicated state store A controller cluster needs state that every member converges on without any member being authoritative for it, and without a full exchange every time something changes. This adds that as a standalone package, with no notion of what is being replicated. Each entry is owned by the peer that wrote it and carries a version from that owner's Lamport clock, so concurrent writes from different owners never contend and a late-arriving older write loses to the newer one it raced. Deletes are tombstones with a TTL, since an absent key and a deleted key are otherwise indistinguishable to a peer that has not seen the delete yet. Steady state is deltas: a change is broadcast once to the mesh rather than once per peer per entry. Anti-entropy covers what a delta missed, opening a round with per-owner hashes so two peers already in agreement exchange a hash rather than every entry version, and descending to entry versions only for the owners whose hashes differ. Each process gets an epoch, a UUIDv7 minted at startup, which gates deletion only: entries from a peer's previous lifetime can be swept when it comes back, while ordering stays with the version. Sweeping is conditional on the entry that was selected still being the entry there, so a replacement that lands mid-sweep is not deleted by it. - adds controller/gossip: the store, owner clocks, tombstones and their reaper, delta broadcast, anti-entropy with owner hashes, epochs, and the mesh abstraction it sends over - adds gossip_pb and the ctrl/mgmt message types the transport and inspection need - is transport-agnostic and carries no link, terminator or other domain concept; callers register a state type and supply their own encoding
| Commit: | c4f992e | |
|---|---|---|
| Author: | Paul Lorenz | |
| Committer: | GitHub | |
Auto-GC stale router links. Fixes #4005 - adds a structured xlink.LinkKey ({DialerBinding, Protocol, DestId, ListenerBinding}) used by all staleness and GC checks - adds the CheckStaleLinks ctrl message and the `ziti ops verify stale-links` CLI for operator-driven, two-sided staleness verification with optional --gc - implements auto-GC via the router.link.v1 gcMode (preserve/orphaned/changed): the router walks its xlinks after each Apply that mutates listeners, dialers, or gcMode and closes one-sided-stale entries under the configured mode For #3743.
| Commit: | a43811e | |
|---|---|---|
| Author: | Paul Lorenz | |
| Committer: | GitHub | |
Manage router link configuration via the controller. Fixes #4004 - adds a link ConfigHandler (router/link FactoryRegistry) that applies router.link.v1 config: Apply rebuilds the listener/dialer set wholesale, and established Xlinks survive because Listener.Close() only closes the accept loop - translates local link: YAML into router.link.v1 JSON and pushes it through the managed-config registry at startup - adds the UpdateLinkListeners ctrl message so the router republishes its listener set to the controller on change; the controller re-fans via the existing PeerStateChange path - re-evaluates dialers on link group and listener changes via RescanForDialOpportunities For #3743.
| Commit: | ea92d78 | |
|---|---|---|
| Author: | Paul Lorenz | |
| Committer: | GitHub | |
Merge pull request #4216 from openziti/router-link-costtags-removal Remove unused link costTags support
| Commit: | f9600f2 | |
|---|---|---|
| Author: | Paul Lorenz | |
| Committer: | Paul Lorenz | |
Replicate link state over gossip. Fixes #3726 A router reported its links to every controller, and each controller kept its own picture built only from what routers told it directly. That does not survive routers being connected to a subset of controllers: a controller learns nothing about links whose routers it does not hold a connection to. Link state now lives in the replicated store: a router reports to one controller, that controller writes the entry it owns, and the mesh carries it to the rest. Each link entry is owned by the router that dialled it, so two controllers never contend for the same key, and a controller that has never spoken to a router still converges on its links. - registers a link state type on the gossip store and carries link add, update and removal through it - makes a link's source router an atomic and repoints it when the router connects, since a link can be built from a gossiped entry before its router has connected here, leaving a database-loaded placeholder as the endpoint - reconciles a reconnecting router's gossip entries, marking its links usable again rather than removing them, since a disconnect sets them down instead of deleting them - tombstones a link on disconnect in single-controller mode, where there is no peer to learn the removal from - adds the gossip transport: peer handlers on the controller mesh, router-facing gossip handlers, digest exchange off the receive goroutine, and the pools that bound apply and I/O work - advertises a gossip capability so a router reports to one controller only once every controller can replicate, and falls back to reporting to all until then - adds canaries, a per-router sequence carried over the same path, so a router can tell that a controller has stopped applying its state - carries link metrics over gossip alongside the state - keeps the disconnect teardown's reroute ordering: the currency guard wraps it, and inside, the link snapshot and MarkDisconnected stay ahead of the cascade so reroute cannot path through the router being removed
| Commit: | d30d35b | |
|---|---|---|
| Author: | Paul Lorenz | |
| Committer: | Paul Lorenz | |
Add a gossip protocol and replicated state store A controller cluster needs state that every member converges on without any member being authoritative for it, and without a full exchange every time something changes. This adds that as a standalone package, with no notion of what is being replicated. Each entry is owned by the peer that wrote it and carries a version from that owner's Lamport clock, so concurrent writes from different owners never contend and a late-arriving older write loses to the newer one it raced. Deletes are tombstones with a TTL, since an absent key and a deleted key are otherwise indistinguishable to a peer that has not seen the delete yet. Steady state is deltas: a change is broadcast once to the mesh rather than once per peer per entry. Anti-entropy covers what a delta missed, opening a round with per-owner hashes so two peers already in agreement exchange a hash rather than every entry version, and descending to entry versions only for the owners whose hashes differ. Each process gets an epoch, a UUIDv7 minted at startup, which gates deletion only: entries from a peer's previous lifetime can be swept when it comes back, while ordering stays with the version. Sweeping is conditional on the entry that was selected still being the entry there, so a replacement that lands mid-sweep is not deleted by it. - adds controller/gossip: the store, owner clocks, tombstones and their reaper, delta broadcast, anti-entropy with owner hashes, epochs, and the mesh abstraction it sends over - adds gossip_pb and the ctrl/mgmt message types the transport and inspection need - is transport-agnostic and carries no link, terminator or other domain concept; callers register a state type and supply their own encoding
| Commit: | 1a58248 | |
|---|---|---|
| Author: | Paul Lorenz | |
| Committer: | Paul Lorenz | |
Persist cluster id across migration snapshot restore. For #4104 - adds a clusterId field to SyncSnapshotCommand and writes it into the database after the migration snapshot restore, so a controller bootstrapped by migrating a database ends up with a durable cluster id instead of an empty one - the snapshot restore replaces the whole FSM database with the migration source, which carries no cluster id, so without this the id written during bootstrap was silently wiped and the node came up with an empty, non-durable cluster id, defeating the mesh cluster-id validation - persists the raft index after the cluster id in RestoreSnapshot so the index remains the completion gate: a failure before it halts (SyncSnapshotCommand is a critical command) and replays/retries on restart rather than skipping the command with a blank cluster id - fails RaftRestoreFromBoltDb when the cluster id is blank after bootstrap - regenerates cmd.pb.go for the new field (cherry picked from commit d8cedeb8ea1813221808f3da556a8fc59efae431)
| Commit: | f3fb011 | |
|---|---|---|
| Author: | Paul Lorenz | |
| Committer: | GitHub | |
Merge pull request #4105 from openziti/ha-cluster-bootstrap-guards Controller cluster bootstrapping fixes
| Commit: | 74e0be0 | |
|---|---|---|
| Author: | Paul Lorenz | |
| Committer: | Paul Lorenz | |
Auto-GC stale router links. Fixes #4005 - adds a structured xlink.LinkKey ({DialerBinding, Protocol, DestId, ListenerBinding}) used by all staleness and GC checks - adds the CheckStaleLinks ctrl message and the `ziti ops verify stale-links` CLI for operator-driven, two-sided staleness verification with optional --gc - implements auto-GC via the router.link.v1 gcMode (preserve/orphaned/changed): the router walks its xlinks after each Apply that mutates listeners, dialers, or gcMode and closes one-sided-stale entries under the configured mode For #3743.
| Commit: | e4f2d18 | |
|---|---|---|
| Author: | Paul Lorenz | |
| Committer: | Paul Lorenz | |
Manage router link configuration via the controller. Fixes #4004 - adds a link ConfigHandler (router/link FactoryRegistry) that applies router.link.v1 config: Apply rebuilds the listener/dialer set wholesale, and established Xlinks survive because Listener.Close() only closes the accept loop - translates local link: YAML into router.link.v1 JSON and pushes it through the managed-config registry at startup - adds the UpdateLinkListeners ctrl message so the router republishes its listener set to the controller on change; the controller re-fans via the existing PeerStateChange path - re-evaluates dialers on link group and listener changes via RescanForDialOpportunities For #3743.
| Commit: | b80ba43 | |
|---|---|---|
| Author: | Paul Lorenz | |
| Committer: | Paul Lorenz | |
Add leader fast-path for confirmed no-op terminator removals. For #4156 - routes V2 terminator removes to the leader (GetModelUpdateCtrlChannel), where the delete is applied anyway and the fast-path can consult current state - adds a createConfirmed flag (index-aligned with terminatorIds) to RemoveTerminatorsV2Request, set once the router receives the create acknowledgement; backwards compatible, older peers omit it and it decodes as false - on the leader, skips the raft/rate-limited delete for a terminator the router confirms was created but that no longer exists, so a storm of no-op retries can't consume the command rate limiter; unconfirmed or still-present ids take the normal ordered path - tests the fast-path filter across leader/confirmed/present/error and mixed cases
| Commit: | 700a795 | |
|---|---|---|
| Author: | Paul Lorenz | |
| Committer: | Paul Lorenz | |
Add asynchronous terminator removal (RemoveTerminatorsV2). For #4156 - adds RemoveTerminatorsV2Request/Response control-channel messages, answered asynchronously so a router no longer holds a request slot open waiting for a synchronous removal reply - adds the ControllerAsyncTerminatorRemove capability so routers use the async path only against controllers that advertise support, falling back to synchronous removal otherwise - adds a controller handler that removes the batch and replies with per-batch success/rate-limited status - tracks in-flight removal batches on the router and, when the response arrives, resolves the batch's rate-limit control and clears or requeues its terminators; lost responses self-heal via the existing rate-limiter expiry and retry scan
| Commit: | 8f5fd9a | |
|---|---|---|
| Author: | Paul Lorenz | |
| Committer: | Paul Lorenz | |
Auto-GC stale router links. Fixes #4005 - adds a structured xlink.LinkKey ({DialerBinding, Protocol, DestId, ListenerBinding}) used by all staleness and GC checks - adds the CheckStaleLinks ctrl message and the `ziti ops verify stale-links` CLI for operator-driven, two-sided staleness verification with optional --gc - implements auto-GC via the router.link.v1 gcMode (preserve/orphaned/changed): the router walks its xlinks after each Apply that mutates listeners, dialers, or gcMode and closes one-sided-stale entries under the configured mode For #3743.
| Commit: | 3e6ca28 | |
|---|---|---|
| Author: | Paul Lorenz | |
| Committer: | Paul Lorenz | |
Manage router link configuration via the controller. Fixes #4004 - adds a link ConfigHandler (router/link FactoryRegistry) that applies router.link.v1 config: Apply rebuilds the listener/dialer set wholesale, and established Xlinks survive because Listener.Close() only closes the accept loop - translates local link: YAML into router.link.v1 JSON and pushes it through the managed-config registry at startup - adds the UpdateLinkListeners ctrl message so the router republishes its listener set to the controller on change; the controller re-fans via the existing PeerStateChange path - re-evaluates dialers on link group and listener changes via RescanForDialOpportunities For #3743.
| Commit: | a003f18 | |
|---|---|---|
| Author: | Paul Lorenz | |
| Committer: | Paul Lorenz | |
Remove unused link costTags support - removes the costTags option from router link listeners, which was parsed from config, advertised to the controller, and stored on the router model but never used for path selection or any other behavior - drops GetLinkCostTags from the xlink.Listener interface and its transport implementation - reserves the corresponding ctrl_pb.Listener.costTags and RouterLinks.RouterLink.linkCostTags protobuf fields and regenerates ctrl.pb.go - documents the removal in the changelog
| Commit: | 6326203 | |
|---|---|---|
| Author: | Paul Lorenz | |
Add controller Limbo state, takeover splice, and reroute token issuance - adds circuit reroute state (active/Limbo) with a grace timer, entered on an ingress ChannelClosed fault for a reroutable circuit and torn down on grace expiry; issues a controller-signed reroute token at circuit creation when reroutable - adds the takeover splice (X1): re-splices a reroutable circuit's ingress to a new router under the circuit's mutation guard, preserving the terminator side, advancing the iteration, and minting a fresh token - stamps the fault reason on router-emitted ingress faults so only a channel close is Limbo-eligible - keeps the D3 idle scanner from prematurely reaping Limbo circuits
| Commit: | f6e5a49 | |
|---|---|---|
| Author: | Paul Lorenz | |
| Committer: | Paul Lorenz | |
Add FaultReason to the Fault message - adds a FaultReason enum (ReasonUnspecified/ChannelClosed/XgClose/AccessLoss) with load-bearing numbering: the proto3 zero value decodes to teardown, and only ChannelClosed is eligible for a Limbo hold and SDK-driven reroute - adds the reason field to Fault so routers can stamp why a circuit faulted - regenerates ctrl.pb.go with protoc-gen-go v1.36.11
| Commit: | 7f9b970 | |
|---|---|---|
| Author: | Paul Lorenz | |
| Committer: | Paul Lorenz | |
Add link metrics over gossip - replicates per-link latency metrics across controllers via gossip - narrows the metrics firehose to one controller when link latency is in gossip - validates firehose narrowing in the links-test by total ingestion - documents the metrics rollout phases
| Commit: | fc51866 | |
|---|---|---|
| Author: | Paul Lorenz | |
| Committer: | Paul Lorenz | |
Add link state replication over gossip. Fixes #3726 - replicates link state across controllers via the gossip mesh so a router can report links to a single controller instead of all - makes Link.Src atomic and repoints stale source routers at connect - adds gossip integration progress/design docs
| Commit: | bedc4ee | |
|---|---|---|
| Author: | Paul Lorenz | |
| Committer: | Paul Lorenz | |
Auto-GC stale router links. Fixes #4005 - adds a structured xlink.LinkKey ({DialerBinding, Protocol, DestId, ListenerBinding}) used by all staleness and GC checks - adds the CheckStaleLinks ctrl message and the `ziti ops verify stale-links` CLI for operator-driven, two-sided staleness verification with optional --gc - implements auto-GC via the router.link.v1 gcMode (preserve/orphaned/changed): the router walks its xlinks after each Apply that mutates listeners, dialers, or gcMode and closes one-sided-stale entries under the configured mode For #3743.
| Commit: | cd026d8 | |
|---|---|---|
| Author: | Paul Lorenz | |
| Committer: | Paul Lorenz | |
Remove unused link costTags support - removes the costTags option from router link listeners, which was parsed from config, advertised to the controller, and stored on the router model but never used for path selection or any other behavior - drops GetLinkCostTags from the xlink.Listener interface, its transport implementation, and test stubs - reserves the corresponding ctrl_pb.Listener.costTags and RouterLinks.RouterLink.linkCostTags protobuf fields and regenerates ctrl.pb.go - documents the removal in the changelog
| Commit: | 87606a1 | |
|---|---|---|
| Author: | Paul Lorenz | |
| Committer: | Paul Lorenz | |
Manage router link configuration via the controller. Fixes #4004 - adds a link ConfigHandler (router/link FactoryRegistry) that applies router.link.v1 config: Apply rebuilds the listener/dialer set wholesale, and established Xlinks survive because Listener.Close() only closes the accept loop - translates local link: YAML into router.link.v1 JSON and pushes it through the managed-config registry at startup - adds the UpdateLinkListeners ctrl message so the router republishes its listener set to the controller on change; the controller re-fans via the existing PeerStateChange path - re-evaluates dialers on link group and listener changes via RescanForDialOpportunities For #3743.
| Commit: | 7533ac0 | |
|---|---|---|
| Author: | Paul Lorenz | |
| Committer: | Paul Lorenz | |
Add leader fast-path for confirmed no-op terminator removals. For #4156 - routes V2 terminator removes to the leader (GetModelUpdateCtrlChannel), where the delete is applied anyway and the fast-path can consult current state - adds a createConfirmed flag (index-aligned with terminatorIds) to RemoveTerminatorsV2Request, set once the router receives the create acknowledgement; backwards compatible, older peers omit it and it decodes as false - on the leader, skips the raft/rate-limited delete for a terminator the router confirms was created but that no longer exists, so a storm of no-op retries can't consume the command rate limiter; unconfirmed or still-present ids take the normal ordered path - tests the fast-path filter across leader/confirmed/present/error and mixed cases
| Commit: | e463b48 | |
|---|---|---|
| Author: | Paul Lorenz | |
| Committer: | Paul Lorenz | |
Add asynchronous terminator removal (RemoveTerminatorsV2). For #4156 - adds RemoveTerminatorsV2Request/Response control-channel messages, answered asynchronously so a router no longer holds a request slot open waiting for a synchronous removal reply - adds the ControllerAsyncTerminatorRemove capability so routers use the async path only against controllers that advertise support, falling back to synchronous removal otherwise - adds a controller handler that removes the batch and replies with per-batch success/rate-limited status - tracks in-flight removal batches on the router and, when the response arrives, resolves the batch's rate-limit control and clears or requeues its terminators; lost responses self-heal via the existing rate-limiter expiry and retry scan
| Commit: | dd88d27 | |
|---|---|---|
| Author: | Andrew Martinez | |
| Committer: | Andrew Martinez | |
backport openziti/ziti#4094 to release-v2.0.x accept first-party certs issued by a separate edge signing CA - publishes FirstPartyX509CertValidation/ThirdPartyX509CertValidation usages and intermediates on router data model public keys, deprecating ClientX509CertValidation - builds the router first-party cert pool from RDM first-party keys unioned with ctrl-channel roots; TLS and VerifyClientCert paths share buildClientCertRoots with fallback to the deprecated usage for old controllers - propagates full controller signing cert chains over the mesh via SigningCertChainHeader and persists them in Controller store CertPem - sends stored public keys during router sync instead of rebuilding them; publishes controller certs leaf-only - stops router controller reconnect loops after shutdown - gives each in-process controller its own command decoder registry - adds the ha-3 three-controller harness and first-party cert integration tests - drains the cli test stdout pipe while commands run; anchors the totp token issued-at assertion to the test clock - backports the SPIFFE-capable test PKI from openziti/ziti#3947: --not-before on ziti pki create, tests/testdata/create-pki.sh/.ps1, and the generated PKI under tests/testdata/pki including the separate edge signing root and per-controller signing intermediates; existing config sets stay on the testdata/ca PKI - skips *.pem, *.cert and *.key files in codespell
| Commit: | 21f039d | |
|---|---|---|
| Author: | Andrew Martinez | |
| Committer: | Andrew Martinez | |
fixes openziti/ziti#4094 accept first-party certs issued by a separate edge signing CA - adds FirstPartyX509CertValidation and ThirdPartyX509CertValidation usages and an intermediates field to the router data model public keys, deprecating ClientX509CertValidation - publishes config CA bundle roots as first-party anchors with their intermediates and Ca store entries as third-party anchors; controller certs carry JWT validation only, since a controller identity is never a CA and anchors no client cert chains - builds router first-party and client cert trust pools from the published usages, falling back to the deprecated usage against older controllers - propagates the full signing cert chain between controllers via a new mesh SigningCertChainHeader and persists whole chains in controller records - removes the orphaned InstantStrategy.AddPublicKey, dead since public key sync moved to controller list data - gives each command dispatcher its own decoder registry so multiple in-process controllers no longer decode into the last-started controller's managers - adds a three-controller in-process HA test harness with a split signing PKI, cluster formation and first-party cert integration tests
| Commit: | e7d23ef | |
|---|---|---|
| Author: | Andrew Martinez | |
| Committer: | GitHub | |
fixes openziti/ziti#3990 push service and posture changes to subscrib… (#4057) * fixes openziti/ziti#3990 push service and posture state to subscribed SDKs - pushes indexed atomic ServiceChangeSet envelopes to subscribed SDK connections: a full snapshot on subscribe, incremental service changes per RDM scan pass, posture check definition changes as their own entries, and identity-resolved config bodies, all serialized so envelopes hit the wire in index order - pushes per-connection PostureStateChange state (monotonic seq, resync on request) for posture pass/fail, including flips caused by definition edits that mutate no posture data - registers pending RDM identity subscriptions for identities not yet synced to the router and sends an authoritative full sync plus full posture state when the identity arrives; an active push subscription pins the connection's RDM listener - advertises service subscriptions and router data model support on the control-channel capability bitmask; the controller persists each router's capabilities mask and version on the EdgeRouter entity via raft and renders them on the edge APIs, so SDKs can select capable routers before connecting - submits posture per router and corrects MFA posture semantics: pushed expiry is the earliest of timeout and pending wake/unlock grace deadlines, wake/unlock re-pass satisfies the re-prompt, api session tokens whose amr attests TOTP seed the MFA baseline from auth_time only (never iat), and token exchange carries the subject token's auth_time - sends structured denials on dial and bind refusals: posture failures carry the failing check ids, no-policy denials are access denied, unknown services are invalid service, and session token failures are invalid session; the denial's cause no longer rides the wire as an unserializable error - hard-closes accepted SDK connections on edge listener shutdown so clients observe a router going away immediately - adds integration coverage: subscription snapshots and change delivery, poll and push reconciliation as capable routers come and go, posture state and definition-change push, router views over the public SDK API, typed dial errors, MFA baseline seeding, and OIDC token-exchange auth_time preservation - removed RDM capability from SDK, router/controller only
| Commit: | a440893 | |
|---|---|---|
| Author: | Paul Lorenz | |
| Committer: | Paul Lorenz | |
Auto-GC stale router links. Fixes #4005 - adds a structured xlink.LinkKey ({DialerBinding, Protocol, DestId, ListenerBinding}) used by all staleness and GC checks - adds the CheckStaleLinks ctrl message and the `ziti ops verify stale-links` CLI for operator-driven, two-sided staleness verification with optional --gc - implements auto-GC via the router.link.v1 gcMode (preserve/orphaned/changed): the router walks its xlinks after each Apply that mutates listeners, dialers, or gcMode and closes one-sided-stale entries under the configured mode For #3743.
| Commit: | a48cf91 | |
|---|---|---|
| Author: | Paul Lorenz | |
| Committer: | Paul Lorenz | |
Manage router link configuration via the controller. Fixes #4004 - adds a link ConfigHandler (router/link FactoryRegistry) that applies router.link.v1 config: Apply rebuilds the listener/dialer set wholesale, and established Xlinks survive because Listener.Close() only closes the accept loop - translates local link: YAML into router.link.v1 JSON and pushes it through the managed-config registry at startup - adds the UpdateLinkListeners ctrl message so the router republishes its listener set to the controller on change; the controller re-fans via the existing PeerStateChange path - re-evaluates dialers on link group and listener changes via RescanForDialOpportunities For #3743.
| Commit: | 7526703 | |
|---|---|---|
| Author: | Paul Lorenz | |
| Committer: | Paul Lorenz | |
Manage router link configuration via the controller. Fixes #4004 - adds a link ConfigHandler (router/link FactoryRegistry) that applies router.link.v1 config: Apply rebuilds the listener/dialer set wholesale, and established Xlinks survive because Listener.Close() only closes the accept loop - translates local link: YAML into router.link.v1 JSON and pushes it through the managed-config registry at startup - adds the UpdateLinkListeners ctrl message so the router republishes its listener set to the controller on change; the controller re-fans via the existing PeerStateChange path - re-evaluates dialers on link group and listener changes via RescanForDialOpportunities For #3743.
| Commit: | c0ea302 | |
|---|---|---|
| Author: | Paul Lorenz | |
| Committer: | Paul Lorenz | |
Manage router link configuration via the controller. Fixes #4004 - adds a link ConfigHandler (router/link FactoryRegistry) that applies router.link.v1 config: Apply rebuilds the listener/dialer set wholesale, and established Xlinks survive because Listener.Close() only closes the accept loop - translates local link: YAML into router.link.v1 JSON and pushes it through the managed-config registry at startup - adds the UpdateLinkListeners ctrl message so the router republishes its listener set to the controller on change; the controller re-fans via the existing PeerStateChange path - re-evaluates dialers on link group and listener changes via RescanForDialOpportunities For #3743.
| Commit: | d8cedeb | |
|---|---|---|
| Author: | Paul Lorenz | |
| Committer: | Paul Lorenz | |
Persist cluster id across migration snapshot restore. For #4104 - adds a clusterId field to SyncSnapshotCommand and writes it into the database after the migration snapshot restore, so a controller bootstrapped by migrating a database ends up with a durable cluster id instead of an empty one - the snapshot restore replaces the whole FSM database with the migration source, which carries no cluster id, so without this the id written during bootstrap was silently wiped and the node came up with an empty, non-durable cluster id, defeating the mesh cluster-id validation - persists the raft index after the cluster id in RestoreSnapshot so the index remains the completion gate: a failure before it halts (SyncSnapshotCommand is a critical command) and replays/retries on restart rather than skipping the command with a blank cluster id - fails RaftRestoreFromBoltDb when the cluster id is blank after bootstrap - regenerates cmd.pb.go for the new field
| Commit: | 7c19e44 | |
|---|---|---|
| Author: | Andrew Martinez | |
| Committer: | Andrew Martinez | |
fixes openziti/ziti#3990 push service and posture state to subscribed SDKs - pushes indexed atomic ServiceChangeSet envelopes to subscribed SDK connections: a full snapshot on subscribe, incremental service changes per RDM scan pass, posture check definition changes as their own entries, and identity-resolved config bodies, all serialized so envelopes hit the wire in index order - pushes per-connection PostureStateChange state (monotonic seq, resync on request) for posture pass/fail, including flips caused by definition edits that mutate no posture data - registers pending RDM identity subscriptions for identities not yet synced to the router and sends an authoritative full sync plus full posture state when the identity arrives; an active push subscription pins the connection's RDM listener - advertises service subscriptions and router data model support on the control-channel capability bitmask; the controller persists each router's capabilities mask and version on the EdgeRouter entity via raft and renders them on the edge APIs, so SDKs can select capable routers before connecting - submits posture per router and corrects MFA posture semantics: pushed expiry is the earliest of timeout and pending wake/unlock grace deadlines, wake/unlock re-pass satisfies the re-prompt, api session tokens whose amr attests TOTP seed the MFA baseline from auth_time only (never iat), and token exchange carries the subject token's auth_time - sends structured denials on dial and bind refusals: posture failures carry the failing check ids, no-policy denials are access denied, unknown services are invalid service, and session token failures are invalid session; the denial's cause no longer rides the wire as an unserializable error - hard-closes accepted SDK connections on edge listener shutdown so clients observe a router going away immediately - adds integration coverage: subscription snapshots and change delivery, poll and push reconciliation as capable routers come and go, posture state and definition-change push, router views over the public SDK API, typed dial errors, MFA baseline seeding, and OIDC token-exchange auth_time preservation
| Commit: | 9b9e560 | |
|---|---|---|
| Author: | Andrew Martinez | |
| Committer: | Andrew Martinez | |
fixes openziti/ziti#3990 push service and posture changes to subscribed SDKs - routers push service change (ServiceChangeSet) and posture state (PostureStateChange) notifications directly to subscribed SDK connections, removing the controller from this path - adds RouterServiceSubscriptions and RouterDataModel capability bits and advertises them in the control-channel capabilities mask - persists each edge router's reported capabilities mask and version with a write-on-mismatch raft update, and renders them on the edge router APIs as a capability string list and version - normalizes a single PROCESS posture check to a one-element PROCESS_MULTI on the wire - closes accepted edge connections on router shutdown so SDKs detect a disconnect promptly rather than waiting for a heartbeat timeout - adds integration tests for capability reporting, service subscriptions, push/poll reconciliation, and posture-state push - resolves each subscribing identity's effective config set (service base configs merged with the identity's per-service overrides) and includes the config bodies on pushed service defs - adds GetIdentityServiceConfigs and refactors loadServiceConfigs to delegate to it - config edits ride the existing service-change path: a config change pushes a service-changed to every service that references it, carrying the new body - adds an integration test covering snapshot bodies, adding a config to a service, and editing a config shared by two services - asserts the resolved config body is returned by GetService, not just present on the service event, in the snapshot, config-added, and config-edited cases
| Commit: | d8ffdaf | |
|---|---|---|
| Author: | Paul Lorenz | |
Make identity revocation timestamps deterministic across controllers. Fixes #4088 - adds a mutation timestamp to the change context (both ChangeContext protos and the change.Context struct), stamped once when a command is dispatched so the replicated command carries the originating controller's time to every raft node, mirroring how raftIndex is threaded - has the identity revocation constraint derive the revocation's ExpiresAt and IssuedBefore from a cluster-consistent time instead of a per-node time.Now(): the disable transition uses the carried DisabledAt (also the correct cutoff), and delete uses the change-context timestamp; falls back to time.Now() only when no change context is present - adds change-context timestamp round-trip tests, including the zero-time mapping
| Commit: | 102db69 | |
|---|---|---|
| Author: | Paul Lorenz | |
| Committer: | Paul Lorenz | |
Auto-GC stale router links. Fixes #4005 - adds a structured xlink.LinkKey ({DialerBinding, Protocol, DestId, ListenerBinding}) used by all staleness and GC checks - adds the CheckStaleLinks ctrl message and the `ziti ops verify stale-links` CLI for operator-driven, two-sided staleness verification with optional --gc - implements auto-GC via the router.link.v1 gcMode (preserve/orphaned/changed): the router walks its xlinks after each Apply that mutates listeners, dialers, or gcMode and closes one-sided-stale entries under the configured mode For #3743.
| Commit: | af66c1f | |
|---|---|---|
| Author: | Paul Lorenz | |
| Committer: | Paul Lorenz | |
Manage router link configuration via the controller. Fixes #4004 - adds a link ConfigHandler (router/link FactoryRegistry) that applies router.link.v1 config: Apply rebuilds the listener/dialer set wholesale, and established Xlinks survive because Listener.Close() only closes the accept loop - translates local link: YAML into router.link.v1 JSON and pushes it through the managed-config registry at startup - adds the UpdateLinkListeners ctrl message so the router republishes its listener set to the controller on change; the controller re-fans via the existing PeerStateChange path - re-evaluates dialers on link group and listener changes via RescanForDialOpportunities For #3743.
| Commit: | 187aa11 | |
|---|---|---|
| Author: | Paul Lorenz | |
| Committer: | Paul Lorenz | |
Own the metrics wire format in ziti. Fixes #4036 - adds a common/servermetrics package that owns the metrics MetricsMessage wire format and the reporting/usage subsystem (message builder, usage registry, interval and usage counters), wrapping the openziti/metrics Registry for metric collection - moves the controllers metrics reporter into the router package and removes it from the shared metrics package, breaking a common -> router/env import cycle - repoints controller and router consumers to common/servermetrics; base metric collection stays on openziti/metrics - keeps the proto field numbers and the metrics content-type identical so the encoding is byte-compatible across the move, and uses a distinct proto package name so ziti's and the library's messages coexist without a global proto registry clash - adds a round-trip test asserting wire compatibility with the library's MetricsMessage - leaves openziti/metrics unchanged, so sdk-golang and the shared xgress data plane are unaffected
| Commit: | bb17872 | |
|---|---|---|
| Author: | Andrew Martinez | |
| Committer: | Andrew Martinez | |
fixes openziti/ziti#3990 push service and posture changes to subscribed SDKs - routers push service change (ServiceChangeSet) and posture state (PostureStateChange) notifications directly to subscribed SDK connections, removing the controller from this path - adds RouterServiceSubscriptions and RouterDataModel capability bits and advertises them in the control-channel capabilities mask - persists each edge router's reported capabilities mask and version with a write-on-mismatch raft update, and renders them on the edge router APIs as a capability string list and version - normalizes a single PROCESS posture check to a one-element PROCESS_MULTI on the wire - closes accepted edge connections on router shutdown so SDKs detect a disconnect promptly rather than waiting for a heartbeat timeout - adds integration tests for capability reporting, service subscriptions, push/poll reconciliation, and posture-state push - resolves each subscribing identity's effective config set (service base configs merged with the identity's per-service overrides) and includes the config bodies on pushed service defs - adds GetIdentityServiceConfigs and refactors loadServiceConfigs to delegate to it - config edits ride the existing service-change path: a config change pushes a service-changed to every service that references it, carrying the new body - adds an integration test covering snapshot bodies, adding a config to a service, and editing a config shared by two services - asserts the resolved config body is returned by GetService, not just present on the service event, in the snapshot, config-added, and config-edited cases
| Commit: | b12d197 | |
|---|---|---|
| Author: | Paul Lorenz | |
| Committer: | Paul Lorenz | |
Auto-GC stale router links. Fixes #4005 - adds a structured xlink.LinkKey ({DialerBinding, Protocol, DestId, ListenerBinding}) used by all staleness and GC checks - adds the CheckStaleLinks ctrl message and the `ziti ops verify stale-links` CLI for operator-driven, two-sided staleness verification with optional --gc - implements auto-GC via the router.link.v1 gcMode (preserve/orphaned/changed): the router walks its xlinks after each Apply that mutates listeners, dialers, or gcMode and closes one-sided-stale entries under the configured mode For #3743.
| Commit: | 381e871 | |
|---|---|---|
| Author: | Paul Lorenz | |
| Committer: | Paul Lorenz | |
Manage router link configuration via the controller. Fixes #4004 - adds a link ConfigHandler (router/link FactoryRegistry) that applies router.link.v1 config: Apply rebuilds the listener/dialer set wholesale, and established Xlinks survive because Listener.Close() only closes the accept loop - translates local link: YAML into router.link.v1 JSON and pushes it through the managed-config registry at startup - adds the UpdateLinkListeners ctrl message so the router republishes its listener set to the controller on change; the controller re-fans via the existing PeerStateChange path - re-evaluates dialers on link group and listener changes via RescanForDialOpportunities For #3743.
| Commit: | bfc8a9b | |
|---|---|---|
| Author: | Paul Lorenz | |
| Committer: | Paul Lorenz | |
Manage router link configuration via the controller. Fixes #4004 - adds a link ConfigHandler (router/link FactoryRegistry) that applies router.link.v1 config: Apply rebuilds the listener/dialer set wholesale, and established Xlinks survive because Listener.Close() only closes the accept loop - translates local link: YAML into router.link.v1 JSON and pushes it through the managed-config registry at startup - adds the UpdateLinkListeners ctrl message so the router republishes its listener set to the controller on change; the controller re-fans via the existing PeerStateChange path - re-evaluates dialers on link group and listener changes via RescanForDialOpportunities For #3743.
| Commit: | e12776b | |
|---|---|---|
| Author: | Paul Lorenz | |
| Committer: | Paul Lorenz | |
Auto-GC stale router links. Fixes #4005 - adds a structured xlink.LinkKey ({DialerBinding, Protocol, DestId, ListenerBinding}) used by all staleness and GC checks - adds the CheckStaleLinks ctrl message and the `ziti ops verify stale-links` CLI for operator-driven, two-sided staleness verification with optional --gc - implements auto-GC via the router.link.v1 gcMode (preserve/orphaned/changed): the router walks its xlinks after each Apply that mutates listeners, dialers, or gcMode and closes one-sided-stale entries under the configured mode For #3743.
| Commit: | cb68a9c | |
|---|---|---|
| Author: | Paul Lorenz | |
Add link metrics over gossip Replicates per-link latency across controllers via gossip so routing cost converges HA-wide, replacing per-controller derivation from the metrics firehose. - adds a link-metrics gossip store keyed by link id and owned by the reporting router, with a router publisher (threshold-based, force-publish on re-dial) and a controller listener (exact-iteration apply, both-end reconcile on link create) - adds a linkLatencyInGossip field to the servermetrics metrics message; the router sets it once every connected controller is gossip-capable, and the controller then stops deriving routing latency from that message while still relaying its histograms for observability - adds an inspect surface for the gossip link-metrics store
| Commit: | 84b2026 | |
|---|---|---|
| Author: | Paul Lorenz | |
Add link state replication over gossip. Fixes #3726 Replicates router-reported link state across HA controllers via a gossip store instead of per-controller fault forwarding, so every controller converges on the same view of links. - adds a versioned gossip store (Lamport clocks, deltas, acks, digests, anti-entropy, tombstones) and a generic per-state-type registration model - adds the link-state gossip type, controller listener, and router gossip client with reconcile and superseded-iteration tombstoning - adds a canary gossip type and controller/router canary handlers to detect divergence, plus inspect and `validate gossip` surfaces - advertises the ControllerLinkGossip capability and gates the router's link-state propagation on all controllers supporting it - ports the affected controller and router handlers from the channel v4 receive-handler API to v5 (AddReceiveHandlers / ContentTypeReceiver) - carries the gossip-stability work: bounded peer send timeouts, broadcast and digest on a separate I/O pool off the apply pool, digest debounce, and count-derivation, plus a per-peer broadcast message clone that fixes a concurrent-map-write crash - migrates the link subsystem to slog and adds observability (ctrl.is_leader gauge, messaging pool meters, slow-handler diagnostic) - adds a links-test fablab model and gossip chaos/validation for soak testing
| Commit: | c9e58f2 | |
|---|---|---|
| Author: | Paul Lorenz | |
Own the metrics wire format in ziti. Fixes #4036 - adds a common/servermetrics package that owns the metrics MetricsMessage wire format and the reporting/usage subsystem (message builder, usage registry, interval and usage counters), wrapping the openziti/metrics Registry for metric collection - moves the controllers metrics reporter into the router package and removes it from the shared metrics package, breaking a common -> router/env import cycle - repoints controller and router consumers to common/servermetrics; base metric collection stays on openziti/metrics - keeps the proto field numbers and the metrics content-type identical so the encoding is byte-compatible across the move, and uses a distinct proto package name so ziti's and the library's messages coexist without a global proto registry clash - adds a round-trip test asserting wire compatibility with the library's MetricsMessage - leaves openziti/metrics unchanged, so sdk-golang and the shared xgress data plane are unaffected
| Commit: | 30256b9 | |
|---|---|---|
| Author: | Paul Lorenz | |
| Committer: | Paul Lorenz | |
Enforce api-session and identity revocations on the router. Fixes #3929 Backport to release-v2.0.x of #3927 (main-line fix in PR #3930). The router's CheckConnections reaper enforced only JWT expiry, so a revoked OIDC api-session, or a disabled/deleted identity, kept its live circuits and hosted terminators until the access token expired. The router now enforces the RouterDataModel revocations directly, tightening access-loss propagation to the reaper interval. - adds a Type field to DataState_Revocation and the raft Revocation command proto, mirroring rest_model.RevocationTypeEnum (API_SESSION/IDENTITY/JTI) so the management API, OIDC producers, sync, and router enforcement share one vocabulary; the common.RevocationType* constants are compile-time bound to the enum to prevent drift - adds an IssuedBefore cutoff so an identity revocation invalidates only sessions issued before it; a session re-authenticated after the cutoff survives the still-lingering revocation. Persists IssuedBefore on the db and model Revocation and carries it (plus the Type) through the single and batched raft marshalling - adds RouterDataModel.IsApiSessionRevoked and IsIdentityRevoked and enforces both in CheckConnections, closing a revoked session's connections - revokes a deleted or disabled identity's live OIDC sessions via an IdentityRevocationConstraint in the db package, run as a store pre-commit constraint so the revocation is written in the same transaction as the identity change and cannot be skipped (self-contained OIDC JWTs aren't otherwise reachable). NewIdentityManager installs it with the revocation type and lifetime - has the OIDC end-session (TerminateSessionFromRequest) revoke the specific api-session named by the z_asid claim, with an identity-scoped fallback; sets IssuedBefore on the identity fallback and the management revocation API; adds RevocationManager.CreateOrReplace, routed through by both the OIDC paths and the management revocation API, so a repeat logout/termination/revocation refreshes the cutoff rather than colliding on the reused id. Expiry derives from the longest configured token duration via a shared common.MaxTokenDuration helper - adds tests/revocation_enforcement_oidc_test.go covering api-session revocation (and a fresh session staying unaffected), identity disable and delete, and the identity cutoff (a post-cutoff session surviving the lingering revocation) - regenerates edge_cmd.pb.go (protoc-gen-go v1.31.0) and edge_ctrl.pb.go (v1.36.11) against the release-v2.0.x base, matching each file's existing generator version
| Commit: | fabd691 | |
|---|---|---|
| Author: | Andrew Martinez | |
| Committer: | Andrew Martinez | |
fixes openziti/ziti#3990 push service and posture changes to subscribed SDKs - routers push service change (ServiceChangeSet) and posture state (PostureStateChange) notifications directly to subscribed SDK connections, removing the controller from this path - adds RouterServiceSubscriptions and RouterDataModel capability bits and advertises them in the control-channel capabilities mask - persists each edge router's reported capabilities mask and version with a write-on-mismatch raft update, and renders them on the edge router APIs as a capability string list and version - normalizes a single PROCESS posture check to a one-element PROCESS_MULTI on the wire - closes accepted edge connections on router shutdown so SDKs detect a disconnect promptly rather than waiting for a heartbeat timeout - adds integration tests for capability reporting, service subscriptions, push/poll reconciliation, and posture-state push
| Commit: | 2fd62f9 | |
|---|---|---|
| Author: | Paul Lorenz | |
| Committer: | Paul Lorenz | |
Auto-GC stale router links. Fixes #4005 - adds a structured xlink.LinkKey ({DialerBinding, Protocol, DestId, ListenerBinding}) used by all staleness and GC checks - adds the CheckStaleLinks ctrl message and the `ziti ops verify stale-links` CLI for operator-driven, two-sided staleness verification with optional --gc - implements auto-GC via the router.link.v1 gcMode (preserve/orphaned/changed): the router walks its xlinks after each Apply that mutates listeners, dialers, or gcMode and closes one-sided-stale entries under the configured mode For #3743.
| Commit: | bb236d1 | |
|---|---|---|
| Author: | Paul Lorenz | |
| Committer: | Paul Lorenz | |
Manage router link configuration via the controller. Fixes #4004 - adds a link ConfigHandler (router/link FactoryRegistry) that applies router.link.v1 config: Apply rebuilds the listener/dialer set wholesale, and established Xlinks survive because Listener.Close() only closes the accept loop - translates local link: YAML into router.link.v1 JSON and pushes it through the managed-config registry at startup - adds the UpdateLinkListeners ctrl message so the router republishes its listener set to the controller on change; the controller re-fans via the existing PeerStateChange path - re-evaluates dialers on link group and listener changes via RescanForDialOpportunities For #3743.
| Commit: | 12d94b7 | |
|---|---|---|
| Author: | Paul Lorenz | |
| Committer: | Paul Lorenz | |
Auto-GC stale router links. Fixes #4005 - adds a structured xlink.LinkKey ({DialerBinding, Protocol, DestId, ListenerBinding}) used by all staleness and GC checks - adds the CheckStaleLinks ctrl message and the `ziti ops verify stale-links` CLI for operator-driven, two-sided staleness verification with optional --gc - implements auto-GC via the router.link.v1 gcMode (preserve/orphaned/changed): the router walks its xlinks after each Apply that mutates listeners, dialers, or gcMode and closes one-sided-stale entries under the configured mode For #3743.
| Commit: | 16ffe0b | |
|---|---|---|
| Author: | Paul Lorenz | |
| Committer: | Paul Lorenz | |
Manage router link configuration via the controller. Fixes #4004 - adds a link ConfigHandler (router/link FactoryRegistry) that applies router.link.v1 config: Apply rebuilds the listener/dialer set wholesale, and established Xlinks survive because Listener.Close() only closes the accept loop - translates local link: YAML into router.link.v1 JSON and pushes it through the managed-config registry at startup - adds the UpdateLinkListeners ctrl message so the router republishes its listener set to the controller on change; the controller re-fans via the existing PeerStateChange path - re-evaluates dialers on link group and listener changes via RescanForDialOpportunities For #3743.
| Commit: | 5b9ba46 | |
|---|---|---|
| Author: | Paul Lorenz | |
Use gossip to propagate link data between routers and controllers. Fixes #3726 Replaces the legacy link fault/refresh protocol with a gossip-based anti-entropy store so routers and controllers converge on link state. - adds the gossip store, anti-entropy, and link/canary gossip handlers on both controller and router - splits router connects into a separate pool from router gossip/canary events (routerEventsPool, peerEventsPool) so connect bursts can't starve the gossip hot path - tidies canary sender close handling - refactors gossip store data to be organized by owner - adds store metrics, host metrics, and drops router metrics on router delete - adds a per-owner FNV-64a hash digest to anti-entropy so matching owners short-circuit per-entry comparison, plus diagnostic instrumentation (slow-handler dumps, disk gauges, stale-send and iteration-mismatch logging) - fixes a cmap re-entry deadlock in the digest handler by collecting (owner, ownerData) pairs under iteration and hashing outside it - defers link registry side-effect events outside the registry lock (applyLinkLocked) to avoid piling dial/accept workers behind a back-pressured event channel
| Commit: | 1081898 | |
|---|---|---|
| Author: | Paul Lorenz | |
| Committer: | Paul Lorenz | |
Implement stale links checking
| Commit: | 90e2848 | |
|---|---|---|
| Author: | Paul Lorenz | |
| Committer: | Paul Lorenz | |
Handle changes to link groups and listeners by publishing listener changes and re-evaluating dialers
| Commit: | 0ee288e | |
|---|---|---|
| Author: | Paul Lorenz | |
| Committer: | GitHub | |
Distribute routers and router-target configs through the RDM. Fixes #3976 (#3977) - adds a `DataState.Router` event variant (id, name, fingerprint, configs) to the RDM protobuf and the matching ConfigType.Target field flows - loads routers into `RouterDataModelSender` at startup and registers entity-change listeners so router create/update/delete and config reassignment emit RDM events - filters `Config` events per-router at `RouterSender`: each router sees the full `Router` set but only its own router-target Configs - receiver-side: parses and stores `Router` entities; GCs router-target Configs locally when they drop off the router's `Configs` list, so no synthetic remove events are needed on the wire - extends the `router-data-model-test` fablab model with router-config distribution scenarios (assignment, reassignment, controller restart, RDM cache miss) - updates `validate-router-data-model` to recognize the new event shape - refreshes the design doc to describe per-router filtering and the change-notification flow
| Commit: | 5fc769c | |
|---|---|---|
| Author: | Paul Lorenz | |
Enforce api-session and identity revocations on the router. Fixes #3929 Backport to release-v2.0.x of #3927 (main-line fix in PR #3930). The router's CheckConnections reaper enforced only JWT expiry, so a revoked OIDC api-session, or a disabled/deleted identity, kept its live circuits and hosted terminators until the access token expired. The router now enforces the RouterDataModel revocations directly, tightening access-loss propagation to the reaper interval. - adds a Type field to DataState_Revocation and the raft Revocation command proto, mirroring rest_model.RevocationTypeEnum (API_SESSION/IDENTITY/JTI) so the management API, OIDC producers, sync, and router enforcement share one vocabulary; the common.RevocationType* constants are compile-time bound to the enum to prevent drift - adds an IssuedBefore cutoff so an identity revocation invalidates only sessions issued before it; a session re-authenticated after the cutoff survives the still-lingering revocation. Persists IssuedBefore on the db and model Revocation and carries it (plus the Type) through the single and batched raft marshalling - adds RouterDataModel.IsApiSessionRevoked and IsIdentityRevoked and enforces both in CheckConnections, closing a revoked session's connections - revokes a deleted or disabled identity's live OIDC sessions via an IdentityRevocationConstraint in the db package, run as a store pre-commit constraint so the revocation is written in the same transaction as the identity change and cannot be skipped (self-contained OIDC JWTs aren't otherwise reachable). NewIdentityManager installs it with the revocation type and lifetime - has the OIDC end-session (TerminateSessionFromRequest) revoke the specific api-session named by the z_asid claim, with an identity-scoped fallback; sets IssuedBefore on the identity fallback and the management revocation API; adds RevocationManager.CreateOrReplace, routed through by both the OIDC paths and the management revocation API, so a repeat logout/termination/revocation refreshes the cutoff rather than colliding on the reused id. Expiry derives from the longest configured token duration via a shared common.MaxTokenDuration helper - adds tests/revocation_enforcement_oidc_test.go covering api-session revocation (and a fresh session staying unaffected), identity disable and delete, and the identity cutoff (a post-cutoff session surviving the lingering revocation) - regenerates edge_cmd.pb.go (protoc-gen-go v1.31.0) and edge_ctrl.pb.go (v1.36.11) against the release-v2.0.x base, matching each file's existing generator version
| Commit: | ff61927 | |
|---|---|---|
| Author: | Paul Lorenz | |
| Committer: | Paul Lorenz | |
Enforce api-session and identity revocations on the router. Fixes #3927 The router's CheckConnections reaper enforced only JWT expiry, so a revoked OIDC api-session, or a disabled/deleted identity, kept its live circuits and hosted terminators until the access token expired. The router now enforces the RouterDataModel revocations directly, tightening access-loss propagation to the reaper interval. - adds a Type field to DataState_Revocation and the raft Revocation command proto, mirroring rest_model.RevocationTypeEnum (API_SESSION/IDENTITY/JTI) so the management API, OIDC producers, sync, and router enforcement share one vocabulary; the common.RevocationType* constants are compile-time bound to the enum to prevent drift - adds an IssuedBefore cutoff so an identity revocation invalidates only sessions issued before it; a session re-authenticated after the cutoff survives the still-lingering revocation. Persists IssuedBefore on the db and model Revocation and carries it (plus the Type) through the single and batched raft marshalling - adds RouterDataModel.IsApiSessionRevoked and IsIdentityRevoked and enforces both in CheckConnections, closing a revoked session's connections - revokes a deleted or disabled identity's live OIDC sessions via an IdentityRevocationConstraint in the db package, run as a store pre-commit constraint so the revocation is written in the same transaction as the identity change and cannot be skipped (self-contained OIDC JWTs aren't otherwise reachable). NewIdentityManager installs it with the revocation type and lifetime - has the OIDC end-session (TerminateSessionFromRequest) revoke the specific api-session named by the z_asid claim, with an identity-scoped fallback; sets IssuedBefore on the identity fallback and the management revocation API; adds RevocationManager.CreateOrReplace, routed through by both the OIDC paths and the management revocation API, so a repeat logout/termination/revocation refreshes the cutoff rather than colliding on the reused id. Expiry derives from the longest configured token duration via a shared common.MaxTokenDuration helper - adds tests/revocation_enforcement_oidc_test.go covering api-session revocation (and a fresh session staying unaffected), identity disable and delete, and the identity cutoff (a post-cutoff session surviving the lingering revocation)
| Commit: | cdc5d89 | |
|---|---|---|
| Author: | Paul Lorenz | |
| Committer: | Paul Lorenz | |
Add configs field to routers. Fixes #3780
| Commit: | 2bf9808 | |
|---|---|---|
| Author: | Paul Lorenz | |
| Committer: | Paul Lorenz | |
Add target field to config types. Fixes #3744
| Commit: | 09db468 | |
|---|---|---|
| Author: | Paul Lorenz | |
| Committer: | Paul Lorenz | |
Update circuit-perf-diff test to report more metrics. Allow deploying ziti-traffic-test from git ref - adds stageziti.StageZitiTrafficTest / StageZitiTrafficTestOnce, which clone openziti/ziti, check out the requested ref (tag, branch, or commit SHA), and build zititest/ziti-traffic-test from source for linux/amd64; output is cached in the kit bin dir as ziti-traffic-test-<version> - extends StageZiti to build the ziti multi-tool from source via the same git-clone path when version is not a release tag (commit hash, branch name, "main"); semver-tagged versions still download the prebuilt GitHub release - introduces gitBuildSpec / buildFromGit in stageziti so the git-build pipeline is shared by both binaries; supports overriding the source repo via ZITI_TRAFFIC_TEST_REPO_URL (a local filesystem path works, letting developers build from a working tree) - adds Loop4SimType.Version: when set, the fablab run builds ziti-traffic-test from that ref before staging; empty preserves the legacy local-path behavior - relaxes canonicalizeGoAppVersion so bare commit hashes and branch names pass through unchanged and can flow into the git-build path, while bare semver still gets the leading "v" prefix; covered by a new unit test - shifts loop4.proto ContentType values from 1-3 to 1000-1002 to avoid collision with channel/v4 reserved types (Ping/Result/Latency/etc.), matching the convention used by ctrl_pb / edge_client_pb - collects xgress-level meters (retransmissions, retx failures, dup acks, dropped payloads, dup payloads, blocked-by-local/remote-window) per data-plane host and emits a "Network health" block in the per-pair report so flow-control regressions are visible alongside service throughput/latency - tags loop-host-xg with sim-services-client and adds a metrics connector block to its config so the listener can push its SDK xgress metrics into fablab via the metrics service - adds tc_actions.go registering netem-based scenarios (tcClear, tcJitter, tcLowLoss, tcHighLatency, tcHighRtt, ...) that apply tc qdiscs to the default interface of every data-plane host; auto- detects the interface so it works across AWS AMI variants without severing ssh on a mgmt NIC - hardens runFullComparison with consecutive-failure tracking: a single failed pair is logged and skipped after a best-effort stopDataPlane recovery, but maxConsecutivePairFailures (5) in a row aborts the run; protects 8-hour comparisons from a transient ssh blip on a single pair - replaces the old GOPATH-bin copy in stageBaselineTrafficTest with a StageZitiTrafficTestOnce call against the new git-build path, so baseline binaries are produced reproducibly from a pinned commit rather than whatever happens to be in GOPATH/bin
| Commit: | 6515e36 | |
|---|---|---|
| Author: | Andrew Martinez | |
| Committer: | GitHub | |
fixes #3680 add revocation management API, CLI, and enforcement (#3789) * fixes #3680 add revocation management API, CLI, and enforcement - adds Management API endpoints for revocations (POST, GET, LIST) with type-aware validation (JTI/API_SESSION require UUID, IDENTITY requires existing identity) - adds CLI commands: ziti edge create revocation identity|api-session|jti - adds revocation checks to resolveOidcSession in security_ctx.go so the REST API returns 401 for revoked OIDC tokens. Previously only ValidateAccessToken (router ctrl channel path) checked revocations, so revoked tokens still received 200 OK from the management and client HTTP APIs - adds api-session revocation check to ValidateAccessToken, which only checked JTI and identity revocations - adds Type field to Revocation model, store, and protobuf message - adds integration tests covering CRUD, input validation, and token enforcement for all three revocation types - use release edge-api@v0.28.1
| Commit: | 949adaf | |
|---|---|---|
| Author: | Paul Lorenz | |
| Committer: | Paul Lorenz | |
Coalesce OIDC JWT revocations to reduce controller write pressure. Fixes #3764 - adds DeleteRevocationsBatchCommand so expired-revocation cleanup goes through raft as a single log entry per batch - adds CreateRevocationsBatchCommand for batched revocation creation through raft - moves refresh-token revocations from synchronous inline creation to a background batcher that flushes on a configurable interval, removing the database and raft as a bottleneck on token refreshes - skips revocation creation for tokens expiring within a configurable threshold (revocationMinTokenLifetime), since they become invalid on their own - validates that revocationMinTokenLifetime is less than 50% of the configured refresh token lifetime - makes the revocation enforcer frequency configurable and restricts it to run only on the raft leader - adds tests for multi-batch delete, batched create with router RDM propagation, and skip-threshold behavior - adds new configuration tunables under edge.oidc: revocationBucketInterval, revocationMinTokenLifetime, revocationBucketMaxSize, revocationMaxQueued, revocationEnforcerFrequency
| Commit: | 3a945e4 | |
|---|---|---|
| Author: | Paul Lorenz | |
Add initial POC work on federated routers
| Commit: | 7b4ae12 | |
|---|---|---|
| Author: | Paul Lorenz | |
| Committer: | Paul Lorenz | |
Add CreateCircuitV3 for RDM-authorized circuit creation. Fixes #3721 - adds CreateCircuitV3 message type and handler for routers that have already authorized dials locally via RDM, bypassing service session tokens in favor of identity ID, service ID, and pre-assigned circuit ID - renames CreateCircuitRequest/Response to CreateCircuitV2Request/V2Response for clarity now that V3 exists - adds CircuitManager.Reserve to atomically claim circuit IDs before routing, preventing collisions on pre-assigned IDs - extends CreateCircuitParams with GetCircuitId so V3 can supply a pre-assigned circuit ID (falls back to UUID generation when empty) - fixes IsDialableByIdentity which was incorrectly calling IsBindableByIdentity - extracts V2 handler into its own file create_circuit_v2.go - adds CreateCircuitV3RequestType/ResponseType (20222/20223) to edge_ctrl protobuf - registers V3 handler in controller server
| Commit: | 62e2916 | |
|---|---|---|
| Author: | Paul Lorenz | |
| Committer: | Paul Lorenz | |
Coalesce OIDC JWT revocations to reduce controller write pressure. Fixes #3681 - adds DeleteRevocationsBatchCommand so expired-revocation cleanup goes through raft as a single log entry per batch - adds CreateRevocationsBatchCommand for batched revocation creation through raft - moves refresh-token revocations from synchronous inline creation to a background batcher that flushes on a configurable interval, removing the database and raft as a bottleneck on token refreshes - skips revocation creation for tokens expiring within a configurable threshold (revocationMinTokenLifetime), since they become invalid on their own - validates that revocationMinTokenLifetime is less than 50% of the configured refresh token lifetime - makes the revocation enforcer frequency configurable and restricts it to run only on the raft leader - adds tests for multi-batch delete, batched create with router RDM propagation, and skip-threshold behavior - adds new configuration tunables under edge.oidc: revocationBucketInterval, revocationMinTokenLifetime, revocationBucketMaxSize, revocationMaxQueued, revocationEnforcerFrequency
| Commit: | e163817 | |
|---|---|---|
| Author: | Paul Lorenz | |
| Committer: | Paul Lorenz | |
Fixes for SDK terminator management. Add support for ziti sdk inspection. Fixes #3609 - removes legacy v1 terminator code path; all terminators now use v2 flow - refactors edgeTerminator.close() to decouple SDK notification from control plane notification - adds pending SDK close notification queue with retry when channel is busy - adds post-create inspect mechanism that verifies SDK still holds the bind after terminator creation - queues second post-create inspect when establishment takes >30s to catch SDK timeout races - detects and discards stale reordered binds on the same connection by comparing connIds - re-establishes replacement terminators when a delete/create race is detected - eliminates IsEntityPresent pre-filter in removeTerminatorsHandler to prevent raft ordering races - fixes ValidateTerminators to query identities from the correct manager with the correct filter field - adds postCreate flag to ValidateTerminatorsV2Request so routers skip redundant SDK inspect - returns retry-later (nil result) from router validation when inspect is temporarily unavailable - blocks SyncAllSubscribers until completion and guards RouterDataModel replacement with in-progress flag - fixes InheritLocalData to enable service access tracking for all subscribed identities - adds Services.Has check in GetServiceAccessPolicies to prevent false policy grants - validates policy-to-identity associations in ValidateServicePolicies - adds `ziti agent tunnel dump-sdk` command for SDK context inspection via IPC agent - adds `ziti fabric inspect sdk` command to query SDK context through routers - fixes --expected-per-host CLI flag binding in validate terminators command - changes bind-access-lost retry hint from NotRetriable to RetryStartOver - moves trace route response and xgress close handling off channel handler goroutine - fixes listTerminators test helper to URL-encode filter parameter - improves sdk-hosting-test validation resilience with login and query retries - adds terminator_create_flow.md documenting the full lifecycle across SDK, router, and controller - adds detailed logging for data model sync, service access tracking, and subscriber change detection
| Commit: | 6b869ce | |
|---|---|---|
| Author: | Paul Lorenz | |
| Committer: | Paul Lorenz | |
Add support for ctrlChanListener on router to the model. Fixes #3635
| Commit: | d1300de | |
|---|---|---|
| Author: | Paul Lorenz | |
| Committer: | Paul Lorenz | |
Add preferredLeader flag to raft configuration. Fixes #3600
| Commit: | 6e50523 | |
|---|---|---|
| Author: | Paul Lorenz | |
| Committer: | Paul Lorenz | |
Expand on controller details send for controller endpoint updates
| Commit: | abaed96 | |
|---|---|---|
| Author: | Paul Lorenz | |
| Committer: | Paul Lorenz | |
Add groups to ctrlChanListeners. Add router listener.
| Commit: | 3b345b8 | |
|---|---|---|
| Author: | Paul Lorenz | |
| Committer: | Paul Lorenz | |
Add support for ctrlChanListener on router to the model
| Commit: | 8f84dad | |
|---|---|---|
| Author: | Paul Lorenz | |
| Committer: | Paul Lorenz | |
Add gap detectino and handling to router data model. Fixes #3599
| Commit: | ab51fa8 | |
|---|---|---|
| Author: | Paul Lorenz | |
| Committer: | Paul Lorenz | |
Fix the control channel header conflicts with channel headers. Use single strategy for sharing capabilities with bit mask.
| Commit: | 45bf527 | |
|---|---|---|
| Author: | Paul Lorenz | |
| Committer: | Paul Lorenz | |
Provide more error context to SDKs for terminator errors. Fixes #3524
| Commit: | 2ffc6e1 | |
|---|---|---|
| Author: | Paul Lorenz | |
| Committer: | Paul Lorenz | |
Update ziti version to 2.0
| Commit: | 284a08b | |
|---|---|---|
| Author: | Paul Lorenz | |
| Committer: | Paul Lorenz | |
Remove support for create terminator v1. Fixes #3516
| Commit: | 08359a9 | |
|---|---|---|
| Author: | Paul Lorenz | |
| Committer: | Paul Lorenz | |
Remove legacy link management code from the controller. Fixes #3512
| Commit: | c18a594 | |
|---|---|---|
| Author: | Paul Lorenz | |
| Committer: | Paul Lorenz | |
Clustering coordination fixes * Allow routers to request current cluster membership information. Fixes #3503 * Get cluster membership information from raft directly, rather than trying to cache it in the DB. Fixes #3501 * Set a router data model timeline when initializing a new HA setup, rather than letting it stay blank. Fixes #3500 * Reduce router data model full state updates. Fixes #3504
| Commit: | 3cfc8b5 | |
|---|---|---|
| Author: | Paul Lorenz | |
| Committer: | Paul Lorenz | |
Controller should clear links not in full link sync from router. Fixes #3492
| Commit: | ae83069 | |
|---|---|---|
| Author: | Paul Lorenz | |
| Committer: | Paul Lorenz | |
Add permissions list to identity. Fixes #3430. Add read-only permission. Fixes #2109. Add CRUD permissions by entity type. Fixes #3435
| Commit: | ffe6e16 | |
|---|---|---|
| Author: | Paul Lorenz | |
| Committer: | Paul Lorenz | |
Separate raft command submission limiting from in-flight limiting. Fix some create terminator error handling. Fixes #3318
| Commit: | d37c721 | |
|---|---|---|
| Author: | Paul Lorenz | |
| Committer: | Paul Lorenz | |
Optimize router data model subscription code. Add additional events. Fixes #3359
| Commit: | ce83c0f | |
|---|---|---|
| Author: | Andrew Martinez | |
| Committer: | GitHub | |
fixes openziti/ziti#2324 add token based enrollment (#3342) * fixes openziti/ziti#2324 add token based enrollment - allows enrollment to certificate auth - allows enrollment to ext jwt token auth - alters ext jwt claimsProperty (maps identity id) to support JSON pointers, defaults to `/sub` - adds ext jwt enrollToCert, enrollToToken to controller valid enrollment end-authenticator state - adds ext jwt enrollAuthPolicyId to map end identity auth policy to, defaults to `default` - adds ext jwt enrollAttributeSelector, supports single field name or JSON pointer to point to a single string or array of string attributes to give the identity, defaults to no selector - adds ext jwt enrollNameSelector, supports single field name or JSON pointer to a string field to use as the name, defaults to `/sub` - add enrollment errors to determine if enrollment has occurred - adds CLI support for ext jwt signer enroll flags
| Commit: | 928c5ec | |
|---|---|---|
| Author: | Andrew Martinez | |
| Committer: | Andrew Martinez | |
fixes openziti/ziti#2324 add token based enrollment - allows enrollment to certificate auth - allows enrollment to ext jwt token auth - alters ext jwt claimsProperty (maps identity id) to support JSON pointers, defaults to `/sub` - adds ext jwt enrollToCert, enrollToToken to controller valid enrollment end-authenticator state - adds ext jwt enrollAuthPolicyId to map end identity auth policy to, defaults to `default` - adds ext jwt enrollAttributeSelector, supports single field name or JSON pointer to point to a single string or array of string attributes to give the identity, defaults to no selector - adds ext jwt enrollNameSelector, supports single field name or JSON pointer to a string field to use as the name, defaults to `/sub` - add enrollment errors to determine if enrollment has occurred - adds CLI support for ext jwt signer enroll flags
| Commit: | 6530637 | |
|---|---|---|
| Author: | Paul Lorenz | |
| Committer: | Paul Lorenz | |
Add alert events. Fixes #3264
The documentation is generated from this commit.
| Commit: | a8220b4 | |
|---|---|---|
| Author: | Paul Lorenz | |
| Committer: | Paul Lorenz | |
Add alert events. Fixes #3264
The documentation is generated from this commit.
| Commit: | 19883df | |
|---|---|---|
| Author: | Paul Lorenz | |
| Committer: | Paul Lorenz | |
Add alert events. Fixes #3264
The documentation is generated from this commit.
| Commit: | e78595e | |
|---|---|---|
| Author: | Paul Lorenz | |
| Committer: | Paul Lorenz | |
Add alert events. Fixes #3264
The documentation is generated from this commit.
| Commit: | be49f22 | |
|---|---|---|
| Author: | Paul Lorenz | |
| Committer: | Paul Lorenz | |
Add alert events. Fixes #3264
The documentation is generated from this commit.
| Commit: | e2e18b0 | |
|---|---|---|
| Author: | Paul Lorenz | |
| Committer: | Paul Lorenz | |
Add alert events. Fixes #3264
The documentation is generated from this commit.