Proto commits in danrusei/danube

These 71 commits are when the Protocol Buffers files have changed:

2026-06-14

replace the single-slot PendingDelivery in the Exclusive and Shared r… (#235) * replace the single-slot PendingDelivery in the Exclusive and Shared reliable dispatchers with a pipelined in-flight window * dispatch config section has been added for max_unacked_messages * dispatch config section has been added for max_unacked_messages * validate the dispatch config

The documentation is generated from this commit.

2026-05-26

retrieve all security bindings for danube admin ui

refactor admin UI cluster page and create new page for rbac admin in UI (#225) * add topic subscription details * improve cluster page, add raft consensus and load balancing * add security services within UI

2026-05-24

danube-admin serve schema registry details (#224)

2026-05-10

Edge register implementation (#220) * add edge registry with heartbeat * add tests for edge registry * use topics defined on the config instead autogenerated

2026-05-05

create the danube-edge capability (#217) * create the danube-edge capability * adapted the auth model * danube-edge server side refactor * use danube-client in danube-edge for resilience * adding edge broker workflow and tests * token is not required for auth none * implement create and delete topic on edge mode * consumer on cloud created after the edge creates the topic on remote cluster * add coordination between edge broker to cluster while creating / deleting topics * start edge service only to cluster mode * resolve wal missing broadcast in append batch * remove the edged secured tests for now * use StreamMessage instead of trimed messsage * First batch resolves via StorageFactory, subsequent batches hit the local HashMap

2026-04-20

Impl key shared subs (#213) * key shared subscription implementation in progress * expose the public api for key shared

2026-04-05

client auth cleanup and eager topic creation (#212) * implement with_token_supplier in order to dynamically rotate the token * create JwtValidationCache in order to cache 30 sec the authn * resolved the bug broker couldn't serve topics assigned to it during convergence window

2026-04-04

implement RBAC and authz (#211) * implemented authentication context propagation and authorization model and metadata-backed authorizer * create roles and bindings though the admin cli * jwt-first authentication implementation for Danube * consolidate the security files within 3 main files * implement cache for authz calls * ensure secure connect between brokers raft traffic * added super admin * add authz metrics * created the unit tests * create the secure end to end tests * resolve a bug of clusters running with seed nodes * add correct path to the certs * resolved the bug on how security data was retrieved from raft * creating the topic for tests

2026-03-28

implement Nack and dead letter queue (#210) * created the SubscriptionFailurePolicy with the defaults * wire nack * wire the pendingdelivery struct in reliable dispatchers * policy-based delay/backoff wiring * implemented retry exhaustion only * implemented subscription dlq * add reliable nack , redelivery metrics * adding dispatch failure handling * create replicatoor for dlq * replicator may need tls to connects to the peers * revert publish message async logic * engine holds failure policy * improve the structure of the dispatcher

2026-03-02

use autodiscovery of existing cluster, for scalling up with helm (#202)

2026-03-01

add raft cluster commands to admin cli and mcp (#201) * add mcp tools for raft cluster * remove etcd references from danube-persistent-storage

2026-02-28

Migrate from etcd to openraft/redb for metadata storage (#199) * replaced ETCD with embeded raft consensus layer * improve the danube-raft tests * second phase of the implementation of the raft service * updating config files * initializing raft cluster within danube * using serde to serialize instead of bincode * increase the raft timers for grpc transport * small changes to ensure data consistency * document the danube-raft crate * remove etcd from docker-compose files * removed local cache, use instead the raft state machine * new node should use --join commands * resolve the admin https config * remove broker A gracefully from the cluster

2026-02-15

Proxy connect (#195) * introduced connect_url + broker_url to work with proxy in case of k8s * ensure connect_url contain https scheme if secure * use the x-danube header also for heath-check * delete old files

2026-02-08

Improve /simplify danube client (#190) * danube client builders returns Result instead of panicking * centralize retry/backoff logic * state machines for TopicConsumer/TopicProducer * schema registry ergonomics * removed the references for ErrorType or ErrorMessage

2026-02-01

resolve rebalance issue, allow to be used from admin , even if autorebalance is off (#189)

2026-01-26

danube load manager auto rebalance (#182) resolve #51 * implemented danube automated rebalancing * add balance and rebalance admin cli commands * based on the user define assignment strategy, calculate rankings * include topic based metrics in the ranking calculation * rebalancing moved within its own module * create danube rebalance test workflow * wire up config to load manager * expeted imbalance on phase 1, but load manager redistrubute the topics balanced * updated the topic list command in the workflow * update all topics count operation * broker id to number parse bug * use polling instead of sleep * the code was using the ranking score instead of the topic count * extended automatic rebalancing polling * added load_report_interval_seconds field configurable * ensures each new topic assignment sees updated load state * use strategy fair in tests, as the topics have no traffic * adaptive threshold based on CV severity * ensure topics are not counted twice * ensures true round-robin for initial topic assignments * min load for small loads * resolve the rebalancing storm, allow only one move per cycle * validate inbalance after broker activate

2026-01-06

Schema registry improvements (#168) * improve schema registry, implement get_schema_by_id * update the danube clis with the latest changes to the schema * resolve multiple schema versions working for a topic * resolve one admin test issue * schema_id is gloabl unique now

2026-01-03

Schema registry implementation (#165) * implement schema registry * adding the examples * add schema register compatibility check * add schema register compatibility check * doc the implementation * created the TopicSchemaContext and move it's implementation to topic_schema * create schema registry tests * remove the println from tests * implemented the schema registry for admin cli * create a test for broker-side schema registry integration for the Danube admin CLI * updated the tests with the new StreamMessage structure * adapt clis to use the newly implemented schema registry * minor bug on describe topic * add danube cli tests * run broker without tls for testing purpose * remove old comments

2025-11-22

get topic lists from admin grpc instead scrapping from prom (#157) * get topics to broker mapping from brokers via grpc * added delivery strategy on listing topics * add logs for cluster/topic actions * resolve the topic listing errors

2025-11-15

added cluster/topic actions for danube-admin-gateway (#156) * added cluster/topic actions for danube-admin-gateway * small readme update

2025-11-10

refactored danube-admin-gateway, additional broker details added on registering (#153) * add admin and admin-clean targets in Makefile * on broker registration add additional fields * refactoring * identify the broker id for the broker hosting the topic

2025-10-31

deactivate / activate an broker for maintenance (#142)

Unload topic from the broker, moving to another available broker in the cluster (#141) * implementation plan * updated the implemetation plan * implemented unload topic

2025-10-10

removed segment reference from StreamMessage (#123)

2025-10-09

Reliable topic options removed from producer creation (#121) * topic reliable creationg refactor doc * removed topic realiable dispatch options from producer creation

2025-09-22

removed unused proto file (#103) Co-authored-by: Dan Rusei <dan.rusei@booking.com>

2025-08-24

admin-cli: add E2E workflow; add JSON outputs and topic describe (#86) * admin-cli: add E2E workflow; add JSON outputs and topic describe; add base64 dep; add isolated CLI integration tests for namespaces/topics (partitioned + non-partitioned); stabilize tests by commenting describe; improve partition handling in tests * resolve the topics describe

2025-02-20

Resolve remote (#81) * connect when is needed to grpc storage backend * use streaming to send grpc data to backend storage * update Storagebackends only with closed segments

2025-02-08

Permanent storage (#77) * create persistent storage crate * move disk storage code * use PersistentStorage instead of DiskStorage * configured managed storage * remove storage crates * resolve the disk storage with /namespace/topic

2025-02-02

implement TLS transport and JWT authentication (#73) * garbage * add tls and jwt support * create the AuthService * create the script to generate the certificates * adding rustls required for grpc with tls * add auth to danube-client * implement Auth for HealthCheck and Schema service * add auth service to lookup service and consumers / producers * copy the certificates * add certs to github secrets * add ca volume on each broker * make integration tests to use tls * change the certificate generation * make the tests to work with tls * update the tests * simplify the tests * simplify client configuration * add schema type * resolve a bug with overlapping segments for distinct topics * ensure the response form lookup is with https if secure * updated the certificates

2025-01-26

use proto with no version (#71)

ensure message uniqueness, by adding offset to the Segment and adapting the MessageID (#70)

2025-01-10

add proto files in danube-core crate

2025-01-09

Chg storage (#57) * move StorageConfig from client to server configuration * add storage config in topic creation * renameStorageBackend to MetadataStorage for clarity and instantiate message storage in Broker

2024-12-28

use client defined Reliable Options (#35) * allow users to provide reliable dispatch details * use danube_client reliable_options and config_dispatch on all crates * implement on broker From<TopicDispatchStrategy> for ConfigDispatchStrategy * improve github workflows, ensure the unit tests are run for libs as well * set proper segment size as are provided in MB

2024-12-27

Impl storagebackend (#34) * create storage backends * implement InMemory storage * resolve reliable dispatch test errors * resolve the topic_storage tests * enable user to select backend storage type * implement disk and aws permanent storage

2024-12-24

Create the reliable dispatch library (#30) * new reliable delivery crate in progress * move reliable_dispatch to the other crate * change reliable_delivery to reliable_dispatch * move consumer dispatcher logic to dispatchers * complete the move of reliable dispatch to new lib * change from TopicDeliveryStrategy to TopicDispatchStrategy

2024-12-23

simplify consumer dispatch code (#29) * create AckMessage struct and remove subscription_name from MsgID * refactor the consumer dispatch code to be cleaner

2024-12-20

Segment size (#28) * moving from segment number of messsages to segment size * resolve consumer.receive contention * resolve the consumer ack bug

2024-12-17

Impl msg ack (#27) * change within document from retention to delivery strategy * work in progress define the message * use StreamMessage for both receiving from producer as well as sending to consumer * enable consumer to send ack * implement server side message acknowledge * refactor the consumer dispatch to send messages in strict order * improving the consumer dispatch methods

2024-12-09

Delivery strategy (#26) * create the topic storage struct with the main functions * creating new reliable dispatchers * create TopicStore lifecycle management policy * implement dispatcher reliable single consumer * add few comments * share the topic_storage across all subscription dispatchers * add segments index to have the ordered list of segments * creating the dispatcher reliable multiple consumers * change from reliable to persistent verb * allow client to define retention strategy * minor bug, on matching the str * rename from persitent to reliable delivery

2024-08-13

implement the consumers for partitioned topics

2024-08-07

update admin cmds

remove unsused admin cmds

2024-08-04

adding metadata message and custom attributes

2024-08-01

fix add / remove topics from danube-admin

2024-07-08

add broker_info

2024-07-07

introduce the danube admin cli crate

2024-07-06

implement health check service on the client side

implemented client service healthcheck on the server side

2024-06-20

implement client lookup service

get_topic return Result instead of tuple

2024-06-19

send actionable errros to client using grpc Status

2024-05-28

consumer struct moved to Arc Mutex as it has to be modified by several functions

2024-05-27

proto file refactoring

2024-05-25

server side subscribe - add consumer to dispatchers

2024-05-24

subscribe server side - work in progress

implemented subscribe on the client side

create the consumer client interface

2024-05-23

work in progress for the send_message server side

2024-05-22

change the namings in proto file to reduce confusion

working to send messages with producer

defining the proto for producer send

create schema on the client side

2024-05-21

client create producer

2024-05-19

using ErrorDetails to respond to grpc calls

2024-05-18

minor changes to producer proto file

2024-04-23

client to broker connection via grpc

creating the initial proto file

2024-04-22

setting up infrastructure