Proto commits in zllovesuki/specter

These 70 commits are when the Protocol Buffers files have changed:

2025-11-30

pki: introduce certificate renewal

The documentation is generated from this commit.

2025-05-10

spec: move BuildVersion and BuildTime into spec overlay: include spec.BuildVersion as part of negotiation tun/server: show connected clients version if available

2025-05-07

tun: add keyless tls support

2024-03-29

rpc: use in-memory buffered stream instead of synchronous net.Pipe

2024-03-27

(wip) client cert renewal TODO: implement renewal logic in client

2023-10-11

go.mod: update module path to be under go.miragespace.co

2023-06-03

chord: added ListKeys to query the network for keys with prefix

wip

2023-05-10

tun: allow the internal endpoint query tunnels on client this commit introduces the ability to list the connected clients on /clients internal endpoint, and make reverse RPC query to the client to list the configured tunnels. other notable changes in this commit: spec/transport: introduce ListConnected() chord: merge /stats and /ring into a single router

2023-04-25

acme: add support for custom domain and DNS This commit introduces major change in functionality, where specter will now self-host the ACME DNS. Self-hosting DNS also allows for allowing client to register custom domain to be used for the tunnel, with automatic TLS issuance.

2023-04-13

[wip] external kv

2023-04-08

pki: use client certificate for authentication This commit refactors the client transport and authentication flow, where client certificate is used instead of ClientID + Token. This is a breaking change that will require configs from previous versions to be migrated.

2023-03-02

spec/protocol: add UnpublishTunnel; tun/server: implement UnpublishTunnel and ReleaseTunnel; tun/client: expose UnpublishTunnel and ReleaseTunnel to Phantom

spec/protocol: add RegisteredHostnames and ReleaseTunnel to TunnelService - also restructure the tunnel.proto to not move Link message structure too much tun/client: make some APIs private, and expose GetRegisteredHostnames for Phantom

2023-02-24

use typos-cli to fix all typos

2023-02-04

Support for multiple virtual nodes on a physical node

2023-02-03

spec/protocol: rename tunnel related definitions to Tunnel/Destination semantic

2023-02-01

tun/gateway: add support for forwarding internal endpoint from one node to another

2023-01-30

rpc: use twirp over software defined transport for rpc This commit refactor the entire RPC system and use twirp as the framework instead of handrolled rpc. The benefit of codegen and the full http ecosystem makes extending easier. Other changes in this commit: Makefile: add codegen for twirp and apply vtprotobuf patch where applicable spec/mocks: fix PipeTransport() sending delegation to the wrong end chord: stats handler now shows RPC stats

2023-01-28

somehow there's another version of the paper

2023-01-27

attempting to fix lookup error and connection reuse once and for all

2023-01-25

wip: refactor transport reuse with negotiation

2023-01-24

wip: refactor to use new stream per rpc call

2022-09-04

spec/chord: add context.Context to KV operations This commit includes major refactor of chord.KV to include context.Context in order to be able to pass request context. This is most useful when we are implementing replication to next N successors for KV operations. Actual implementation is left as an exercise for later.

2022-08-18

spec/tun: use proto to send status code; tun: provide tunnel feedback when connecting to remote node

kv/aof: store checksum and verify checksum on read

2022-08-16

kv/aof: implement LogEntry and version This commit allows for versioning LogEntry instead of storing raw Mutations to the AOF log. This allows for later implementations to, for example, compress log entries before storing to disk.

kv/aof: implement append only persistence This commit introduces a new chord.KVProvider implementation that will persist most kv mutations to disk with append-only logs, and restoring the in-memory state on start-up by replaying the logs. Lease operations are not persisted as they are deemed volatile, and the new KV implementation is meant for persisting SimpleKV and PrefixKV which they are much more useful in case the node crashes.

2022-08-15

tun/server: report to client how many servers were the tunnel published to

2022-08-14

cmd/client: support for connecting TCP target This commit introduces support for one-off or listener mode to connect to tunneled target from the specter client. Some changes to the gateway had to be made because of quic's EarlyConnection where the stream is not established despite using .OpenStreamSync(). Therefore, the client needs to poke the gateway. Other changes in this commit: util: move GetOutboundIP to util package tun/client: call .Sync() before closing the config file

2022-08-13

tun/client: self healing tunnels tun client will now establish multiple connections to specter server and automatically request more nodes to be connected to in order to maintain at most 3 links. tunnel will now live in its own command space Other changes in this commit: cmd/specter: move App definition to this package spec/overlay: remove unused interfaces integrations: attempting to create integration tests CI: add integration job

tun/client: multiple and persistent tunnels tun/client now supports persisting some states to disk, and publish new tunnels automatically when the config file changes. fix #6 and #5 TODO: auto-healing broken connection with specter server TODO: context cancellation in http acceptor Other changes in this commit: spec/protocol: consolidate tunnel RPC into ClientRequest/ClientResponse cmd/client: update to use only tun/client exported APIs

chord/kv: add PrefixContains interface

spec/proto: reuse keys for children prefix

2022-08-10

chord/kv: implement atomic leave; This commit completes the second part of the algorithm proposed by Ali Ghodsi of request lock from the successor before leaving. More tests are needed to verify the mechanism. Changes in this commit: chord/kv_membership: remove the lock on predecessor when joining; break out finish operation into advisory stablize and release; implement leave locking as per algorithm README: update chord KV status

chord/kv: partial atomic ring maintenance During the process of implementing certmagic.Storage over specter KV, and validating the v1 implementation of key consistency, it was discovered that concurrent joins can lead to temporary loss of key consistency, which means certmagic will "lose" the writes it just issued mere milliseconds ago. The atomic ring maintenance algorithm proposed by Ali Ghodsi is shown to be correct and can maintain key consistency during membership change, at the expense of availability (because of asynchronous network). With some minor modifications, such algorithm is partially implemented (albeit deviated greatly), and shown to be maintaining key consistency just enough for our limited purpose. Changes in this commit: acme: implement certmagic.Storage using specter KV backend allowing for self bootstrapping chord/kv: implement atomic ring maintenance with the assumption that the joining will not fail during the process spec/chord: added extended interfaces to allow for atomic join spec/protocol: added corresponding RPCs for extended interfaces cmd/server: use specter KV backed storage for certmagic cmd/client: allow specifying tunnel target for test client Dockerfile: added validator build target, added test client build target README: added reference to Ali Ghodsi's paper and dissertation

2022-08-08

spec/chord_kv: separate operations into 3 categories; kv: move MemoryMap into kv/memory; spec/protocol: rename KV field names to reflect category change

2022-08-07

spec/chord_kv: introduces Lease Acquire/Renew/Release This commit introduces new APIs to chord.KV interface, and allowing a caller to hold a lease for a specific lease name, with time-to-live auto expiring leases. TODO: grab a monitonic clock from the time package somehow

2022-08-06

spec/chord_kv: implement prefix kv This commit introduces prefix/children KV interfaces, allowing the caller to track an array of keys under a prefix. This is useful for tracking key hierarchy (as the property of DHT does not allow for a simple prefix search).

2022-08-04

tun/gateway: fix http3 multiplex h3 and specter-tcp/1 cipher: generate dynamic *tls.Config for http3; spec/protocol: rename TCP proto; tun/gateway: add quic logo for http3 test in browser tun/gateway: add http3 related tests

2022-08-03

spec/chord_kv: export protocol.KVTransfer to support future complex KV operations

spec/chord_kv: rename interfaces to be more accurate

2022-08-01

spec/chord: shrink RPC KV definitions to only DirectPuts (renamed from LocalPuts) chord/kv_test: use 400 keys in transferIn

2022-07-31

kv: implement atomic MakeKey

2022-07-28

tun/gateway: enable http2 support on gateway

2022-07-26

go.mod: use kon.nect.sh vanity url

2022-05-24

Makefile: add android to release target; chord: use KV interfaces on LocalNode instead of directly from KV in key transfer; cmd: added colored level in logger; overlay: rearrange fields for alignment; rpc: use const size

2022-05-19

spec: move proto definitions to separate folder; Makefile: force test with -count=1

2022-05-17

for the last time, fix the key transfer procedure by the successor to the predecessor notifying

spec/protocol: add kv operation to request keys to transfer; node: fix key transfer in procedure by only requesting keys from successor upon joining

tun: multi connect in the working, however chord KV key transfer is broken?

go.mod: rename module path to github.com/zllovesuki/specter

2022-05-16

spec/tun: include helper functions to generate identities keys; spec/protocol: include tun.IdentitiesPair for forward and backward resolution

spec/transport: add support for optional datagram; overlay: implement datagram support for application

2022-05-15

go.mod: upgrade indirect depenencies; sprc/tun: decorate alpn with actual names and figuring out how to access those descriptors; tun: use buffer pool

[WIP] tun: implementing server rpc handler

[WIP] tun: starting implementing the tunnel part of the project; spec: added tun specific protobuf; transport: added Identity() for tunnel server

overlay: exploit the fact that remote will always come to us first to reuse quic connection bidirectionally; node: return error upon leaving, and notify after key transfer; spec: add new Connection proto for negotiation

chord/kv: implement local KV operations; node: implement key transfer to succcessor upon leaving; rpc: move spec into spec/rpc; rpc: change size buffer length into constant

2022-05-14

overlay: use net.Conn instead of custom type, and wrap quic.Stream for net.Conn; rpc: refactor rpc to return response, and completely redo rpc protocol buffer definitions

chord/kv: implement FindKeys for key transfer [WIP full implementation]; node: skip certain tasks when not running in ring

rpc: now use context.Context; spec: move transport and rpc interface for abstraction; chord/kv: start working on key transfer

overlay: transport should have one 1 type of rpc

chord: use extended successors list for better tolerance; transport: use reaper to remove dead quic connections; transport: use quic datagram to actively check for liveness

2022-05-13

chord: new node/leaves now correctly fix the ring; TODO: connections are cached, need to evict

chord: RemoteNode sort of works via RPC, but it is extremely incorrect; [WIP] fix chord implementation

[WIP] overlay: start implementing RPC via QUIC

2022-05-12

chord: add kv interface; kv: start working on kv implementation; node: move chord VNode implementations to separate module

spec: update rpc spec

spec: start to define RPC specs