Proto commits in consensus-shipyard/mir

These commits are when the Protocol Buffers files have changed: (only the last 100 relevant commits are shown)

2024-01-04

Refactor factory module implementation Signed-off-by: Matej Pavlovic <matopavlovic@gmail.com>

Replace protobuf timer events by stdevents Signed-off-by: Matej Pavlovic <matopavlovic@gmail.com>

Replace legacy Init event by stdevents.Init Signed-off-by: Matej Pavlovic <matopavlovic@gmail.com>

The documentation is generated from this commit.

Remove legacy proto-related code Signed-off-by: Matej Pavlovic <matopavlovic@gmail.com>

2023-12-25

Add a pingpong sample with custom events Signed-off-by: Matej Pavlovic <matopavlovic@gmail.com>

Move Trantor types to stdtypes Signed-off-by: Matej Pavlovic <matopavlovic@gmail.com>

2023-11-07

Remove follow-up events (#522) * Remove follow-up event stripping functions * Remove Next field from Event protobuf * Remove usages of follow-up events * Re-introduce simulation code handling follow-ups Signed-off-by: Matej Pavlovic <matopavlovic@gmail.com>

2023-08-02

Use big.Int encoded as string for vote weights (#506) * Use big.Int encoded as string for vote weights * Check validity of externally received memberships * Use large node weights in Trantor tests Signed-off-by: Matej Pavlovic <matopavlovic@gmail.com>

2023-07-26

Garbage-collect batch DB (#502) * Remove deduplication from fake batch db * Simplify fake batch db events * Re-generate protobufs * Remove forgotten ts store from fake batch db * Add garbage-collection to fake batch db * Re-generate protobufs * Clean up respondIfReady signature in msc * Make ISS aware of batch db module ID * Garbage-collect batch db state from ISS * Explain behavior of BatchStored in a comment Signed-off-by: Matej Pavlovic <matopavlovic@gmail.com>

Make mempool not drop transactions (#501) * Rename deliveredrequests to deliveredtxs * Add methods for querying client progress * Update mempool-related protobuf definitions * Re-generate protobufs * Clean up and add imports for future code * Adapt mempool transaction lookup to new protobufs * Make mempool aware of the current epoch This is required for not emitting the same transactions multiple times in the same epoch, but still re-emitting transactions that have not been delivered in a previous epoch. The main changes to the logic of the mempool itself are not yet included in this commit to simplify review. They will follow in the next commit. * Make mempool not drop transactions The explanation of how this works is written in the comments in the beginning of the updated formbatches.go file. Signed-off-by: Matej Pavlovic <matopavlovic@gmail.com>

2023-07-24

Externalize Preprepare Validity checker (#494)

2023-07-18

Externalize Checkpoint validation to dedicated module (#493)

2023-07-07

Add timeouts and batch size to mempool The mempool itself can now wait for a transaction batch to fill. After a request for a new batch, the mempool waits until one of: - it has a full batch with the configured number of transactions - it has a full batch with the configured maximum total payload size - a configurable timeout Signed-off-by: Matej Pavlovic <matopavlovic@gmail.com>

2023-07-06

Proper DSL implementation of the Pingpong example * Add proper DSL implementation of pingpong sample * Clean up DSL and low-level pingpong implementation * Generate pingpong protobufs, types, and functions Signed-off-by: Matej Pavlovic <matopavlovic@gmail.com>

2023-07-03

Parametrize availability layer sub-certificate limit The limit on the number of sub-certificates is not hard-coded to 5 any more, but is a proper parameter of the Trantor system. Some tests were designed before the code was even generalized to support more than one sub-certificate (the limit effectively being 1) and stopped working after the code supported 5 sub-certificates by default. Tests that should have failed were passing due to this. This test makes this limit a proper parameter that can be higher by default, but still set to 1 in tests. Details: * Remove obsolete Certs field from MSC params * Use parameter template for availability module * Remove unused field in dynamic availability params * Re-generate protobufs * Set default subcert limit to 5, but test with 1 Signed-off-by: Matej Pavlovic <matopavlovic@gmail.com>

2023-06-16

Sanitize messages and add handler for modules that do not tolerate transport events (#460) * Add handler for modules that do not tolerate transport events * Sanitize all messages

2023-05-26

Weighted voting (#459) * Introduce VoteWeight type * Add serialization to VoteWeight type * Generalize quorums on weighted membership * Use generalized quorum in checkpoint protocol * Work around strange rxgo behavior with empty sums * Set default weight in all memberships to 1 * Use weighted voting in PBFT * Update comments for blacklist leader policy * Use generalized quorums in leader selection policy * Use generalized quorums in multisig collector Signed-off-by: Matej Pavlovic <matopavlovic@gmail.com>

2023-05-14

Get rid of unused TxIDString and BatchIDString types (#448)

2023-05-08

Orderers DSL cleanup (#434)

2023-05-04

Update Orderers module to use DSL (#432)

2023-05-03

Update appmodule to use DSL (#428)

2023-04-26

Commonpb cleanup, Rename req to tx (#422) If there is still any transaction referred to as "request", it's a mistake and should be fixed. Signed-off-by: Matej Pavlovic <matopavlovic@gmail.com>

2023-04-24

Clean up empty certificates (#417) * Simplify certs * Add `empty` field to DeliverCert event * Remove special case for empty batch ID * Update (generalize) PBFT view change comments Now the orderer does not necessarily order availability certificates. The comments now reflect that. * Fix handling of malformed proposals The `aborted` flag must be set to true rather than taken over from the proposal. * Clean up producing and handling empty certificates Instead of encoding an empty certificate through an ad-hoc special value, use a dedicated flag to tell the batch fetcher that it should not even look at the certificate. * Fix rebasing artifacts Signed-off-by: Matej Pavlovic <matopavlovic@gmail.com>

Codegen: fix `repeated bytes` issue (#419) - Distinguish between `bytes` and repeated types. - Use DSL types in mempoolpb. - Skip linting for generated code (since some generated files now contain type conversions that are technically not needed). Signed-off-by: Matej Pavlovic <matopavlovic@gmail.com> Co-authored-by: Matej Pavlovic <matopavlovic@gmail.com>

2023-04-21

Use unified Membership abstraction (#415) The membership is now defined as a protobuf and the corresponding generated type is used everywhere where membership information is required. No more `map[t.NodeID]t.NodeAddress`. Signed-off-by: Matej Pavlovic <matopavlovic@gmail.com>

Types cleanup (#413) Move module-specific type definitions from the general types package to more appropriate places. Clean up serialization package. Signed-off-by: Matej Pavlovic <matopavlovic@gmail.com>

2023-04-20

Clean up Protobufs (#406) Complete refactoring of the protocol buffers definitions and usage. Most of the obsolete definitions have been removed and the existing definitions refactored to the same standard. Roughly, each module defines its own protobufs.

2023-04-12

Cleanup ISS module and update to DSL * Update ISS to use DSL * Cleanup unused files and proto messages * Changed zero value from nil (see #401) * Check checkpoint membership (with warning if cannot be checked)

2023-03-08

Add documentation for the factory module (#369) Co-authored-by: Matej Pavlovic <matopavlovic@gmail.com>

2023-03-01

Batchfetcher and mempool modules update to DSL codegen. (#348) * Use DSL's codegen on batchfetcher and mempool * Fix #353 * Add support for map types to DSL's codegen

2023-02-27

Update availability module This is a squash of multiple @ranchalp's commits: - Reimplement the availability module using codegen - Have orderer's module verify preprepare certificates - Have av. module compute certificate batch without waiting for request - Clean unused internal/dsl/ availability code Signed-off-by: Matej Pavlovic <matopavlovic@gmail.com>

2023-02-20

Implement orderer validity checker selection Signed-off-by: Matej Pavlovic <matopavlovic@gmail.com>

Rename CertData to Data in PBFT Preprepare message Signed-off-by: Matej Pavlovic <matopavlovic@gmail.com>

Improve Segment abstraction, add proposals to it Signed-off-by: Matej Pavlovic <matopavlovic@gmail.com>

Use full membership information in segment struct Signed-off-by: Matej Pavlovic <matopavlovic@gmail.com>

Store segment seqNrs and proposals in a map Signed-off-by: Matej Pavlovic <matopavlovic@gmail.com>

Add instance ID to PBFT deliver events Signed-off-by: Matej Pavlovic <matopavlovic@gmail.com>

2023-01-20

Refactor checkpoint instantiation and epoch data The epoch configuration is now passed directly to the checkpoint protocol at creation, rather than in a separate event. The relevant code is refactored to remove redundancy. The epoch data is augmented by the membership of the epoch immediately preceding the epoch to which the checkpoint is associated. It is this membership that created the checkpoint (the checkpoint of epoch e is created by nodes of epoch e-1) The preceding membership needs to be used to verify checkpoint certificates. This is not the case yet and must be addressed in a future commit. Signed-off-by: Matej Pavlovic <matopavlovic@gmail.com>

2023-01-16

Include leader policy in checkpoints Signed-off-by: Matej Pavlovic <matopavlovic@gmail.com>

2023-01-13

Add stateful LeaderSelectionPolicy to EpochData for snapshots

2023-01-06

Remove dedicated NewLogFile event and use NewEpoch The function EventNewEpochLogger is meant to serve as an example. Signed-off-by: Matej Pavlovic <matopavlovic@gmail.com>

2022-12-28

Add function to split logging on specific event tracked

2022-12-23

Generate code for dsl request-response syntax

Remove an unused codegen extension

Add origin fields to the model

Handle string fields annotated with [(mir.type) = error]

Add code generation for handling net messages

Code generation prototype core

2022-12-22

Delete unused proto imports Signed-off-by: Matej Pavlovic <matopavlovic@gmail.com>

Add epoch to new orderer params Signed-off-by: Matej Pavlovic <matopavlovic@gmail.com>

2022-12-08

Create PBFT orderers outside of ISS This commit implements the new Orderers factory module that creates PBFT orderers (SB instances). PBFT orderers are no longer contained within the iss package, removing an odd handling of events by the sbeventservice (now removed). This commit also implements the new orderers and ordererspb packages, and the issutil package that contains functions called by both iss and orderers. Also, isspbftpb is now ordererspbftpb. Signed-off-by: Matej Pavlovic <matopavlovic@gmail.com> Co-authored-by: Matej Pavlovic <matopavlovic@gmail.com>

2022-11-24

Fix bug when catching up during epoch transition Signed-off-by: Matej Pavlovic <matopavlovic@gmail.com>

2022-11-01

Add more metadata to EpochConfig protobuf Signed-off-by: Matej Pavlovic <matopavlovic@gmail.com>

Implement client watermarks Signed-off-by: Matej Pavlovic <matopavlovic@gmail.com>

Add maputil Trnasform, small cosmetic changes Signed-off-by: Matej Pavlovic <matopavlovic@gmail.com>

2022-10-31

Pass whole StableCheckpoint to SMR RestoreState Signed-off-by: Matej Pavlovic <matopavlovic@gmail.com>

2022-10-25

Add EpochProgress event to checkpoint protos Signed-off-by: Matej Pavlovic <matopavlovic@gmail.com>

2022-10-07

Take checkpointing out of ISS The checkpoint protocol now runs separately from ISS. It has its own factory module where each checkpoint is created using a separate instance of the protocol. The application now also receives the stable checkpoints. Signed-off-by: Matej Pavlovic <matopavlovic@gmail.com>

Separate checkpoint protobufs in their own package This is only to prepare the ground for the whole checkpointing protocol to be put in a separate package. Signed-off-by: Matej Pavlovic <matopavlovic@gmail.com>

2022-10-03

Add sample pingpong protocol (used in video)

2022-09-27

Implement threshold criptography

threshcrypto: Add DSL module

Format code

api,dummy: Validate signature share origin in VerifyShare

Move threshcrypto messages to separate package

2022-08-31

Rename Deliver event to DeliverCert Signed-off-by: Matej Pavlovic <matopavlovic@gmail.com>

Use separate event for batch delivery, PR comments Signed-off-by: Matej Pavlovic <matopavlovic@gmail.com>

2022-08-25

Add batchdb interface to proto files

2022-08-23

Consolidate comments and delete dead code Signed-off-by: Matej Pavlovic <matopavlovic@gmail.com>

Slightly reorganize code based on review comments Signed-off-by: Matej Pavlovic <matopavlovic@gmail.com>

Implement factory module Signed-off-by: Matej Pavlovic <matopavlovic@gmail.com>

Adapt ISS to use availability module Signed-off-by: Matej Pavlovic <matopavlovic@gmail.com>

2022-08-12

Keep track of memberships internally Signed-off-by: Matej Pavlovic <matopavlovic@gmail.com>

Chat app reconfiguration Signed-off-by: Matej Pavlovic <matopavlovic@gmail.com>

Refactor state stanpshot and add membership to it WARNING! Stable checkpoint verification has been disabled because it is buggy. Signed-off-by: Matej Pavlovic <matopavlovic@gmail.com>

2022-08-04

Rename bcbpb.Request to bcbpb.BroadcastRequest

2022-08-03

Use requestpb.Request instead of []byte to represent transactions

2022-08-02

Use bytes instead of string in protobufs for hashes Unlike in go, where a string represents an arbitrary sequence of bytes, in protobufs, it must be possible to encode a string in UTF-8.

2022-08-01

add type field to request (#165)

2022-07-28

Add NewConfig event (#159)

2022-07-16

Fix grpctransport protobuf generation The generated files had been moved, but the proto file hadn't. Moved the proto file accordingly.

2022-07-15

protos: Remove unused WALLoadAll event Signed-off-by: Sergey Fedorov <sergey.fedorov@protocol.ai>

2022-07-12

Move ContextStoreOrigin and DslOrigin to separate proto files It helps to avoid cyclic dependencies of proto files.

Add events and boilerplate for the availability layer Similarly to the interface between the availability layer and the mempool, the interface between the consensus layer and the availability layer is pull-oriented: the consensus layer requests (pulls) availability certificates from the availability layer.

Add boilerplate and documentation for mempool events

Add event types for mempool The interface between the availability layer and the mempool is pull-based. Push-based architecture was also considered, but it would basically require replicating the mempool structure in the availability layer module.

Add Batch and Transaction IDs to the mempool interface The idea is to encapsulate batch and transaction IDs in the mempool module to allow for more flexibility (e.g., the ID of a batch can be a simple hash or a Merkle tree root) and avoid potential subtle bugs (e.g., different modules could accidentally use different hash functions). It also helps to make the code more expressive.

Add multisig collector prototype The prototype implements the desired interface and is supposed to work properly in the crash-stop model. However, it does not use a real persistent storage (and, hence, does not support node recovery). The prototype implements a Byzantine fault-tolerant protocol, but it is susceptible to DOS attacks from Byzantine nodes. Finally, at the moment, the prototype assumes reliable delivery of messages between correct nodes.

2022-07-10

Verify stable checkpoint messages Signed-off-by: Matej Pavlovic <matopavlovic@gmail.com>

2022-07-07

Add protoc plugin to generate code to simplify dsl implementation The plugin generates more generic-friendly enums for oneof fields marked with a special annotation mir.event_type. In future iterations, the plugin can be extended to automatically generate most of the boilerplate code associated with programming in mir.

Periodic catching up, configurable epoch retention Sending stable checkpoints to nodes that are suspected of having fallen behind now happens periodically. The number of old epochs to keep is now configurable. Only epochs that are older than the latest stable checkpoint minus a configuration parameter are garbage-collected. Signed-off-by: Matej Pavlovic <matopavlovic@gmail.com>

Remove unused functions and parameters Signed-off-by: Matej Pavlovic <matopavlovic@gmail.com>

Implement end-of-segment checkpoints in PBFT These checkpoints are necessary for liveness. Each SB instance must be live independently of the other instances. In particular, even instances that delivered all slots in a segment must keep running until a global stable checkpoint is reached. Otherwise the system might get stuck. Signed-off-by: Matej Pavlovic <matopavlovic@gmail.com>

2022-06-27

Implement DSL modules The high-level goal is to make it possible to implement distributed protocols in mir with a syntax that is very close to the standard pseudocode notations as well as to separate the protocol logic from the boilerplate. The way it is done is basically by creating a domain-specific language inside go. The motivating example can be found in pkg/bcb/bcbmodule.go. The core implementation is in pkg/dsl/dslmodule.go. The rest is mostly boilerplate and auxiliary functions.

2022-06-23

Fix naming of the contextStore field in proto files

Small refactoring of ContextStore

Add constructors for EventList

2022-06-20

Remove ClientTracker, ReqStore, and request refs Since the availability layer will not be handled by ISS, related modules are now removed and the ISS protocol itself significantly simplified (e.g. no dealing with request references only or waiting for requests to become available). Signed-off-by: Matej Pavlovic <matopavlovic@gmail.com>

Add contextstore implementation The goal of contextstore is to simplify context recovery when working with modules like Crypto and Hasher. Currently, using these modules requires creating a new protobuf type each time the user wants to verify a signature or compute a hash. This is associated with a lot of boilerplate code. ContextStore allows saving the context locally in the module and only sending an automatically generated id in a standard ContextStoreOrigin protobuf.

2022-06-16

Generalize WAL module Signed-off-by: Matej Pavlovic <matopavlovic@gmail.com>