Proto commits in streamnative/oxia

These 70 commits are when the Protocol Buffers files have changed:

2025-04-11

step commit

step commit

step commit

2024-10-24

fix: fix assignment lost when restart node

2024-10-08

Allow to pass the same secondary index name multiple times on the same record (#546)

The documentation is generated from this commit.

2024-10-03

Protobuf changes for secondary indexes (#541)

2024-09-25

On delete-range send 1 single notification to clients (#532) When there is a delete-range request, we are currently sending 1 `deleted` notification for each of the keys that were deleted in the range. The problem is that the number of keys in the range is unbounded and can lead to a huge batch in the DB and also to a big notification batch to send out to client. It's not easy to split the notification batch, since it needs to be written atomically with the db update itself. Instead, we add a new notification type that will simply tell the client that a range of keys was deleted.

2024-09-23

Allow to disable notifications on a namespace (#526) This is part-1 of the changes. It adds the option to configure a namespace with notifications disabled. If a use case only needs key-value semantics and doesn't use notifications, we can disable them to avoid the overhead. (Part-2 will use the configuration to implement the notifications skipping).

2024-09-13

Rename ShardId field to Shard (#511) Renaming the protobuf field from `ShardId` to `Shard`. The motivation is to have consistent `shard: 123` in the logs, such that filtering them based on shard becomes easier. All logs context are already using the `shard` label, though when we print Protobuf struct directly, it would today have `shardId`.

2024-07-28

Added server side support for WriteStream operation (#499)

2024-05-18

Added range-scan support in oxia db (#479)

2024-05-16

Support sequential write operation on server dbs (#472)

2024-05-15

Added partition-key to override shard routing (#470)

2024-05-14

Server side handling of floor/ceiling get requests (#467)

2024-01-05

*: Update license

2023-07-22

Use VT proto for serialization (#366) Using https://github.com/planetscale/vtprotobuf for adding extra methods to the generated Protobuf code. The extra marshallVT/unmarshallVT methods are generated as "rolled-out" code instead of using reflection so they're using much less CPU and memory allocations. It's also capable of generating hooks for getting objects from pool.

2023-04-12

Include head & commit offset in GetStatus (#326)

2023-03-30

Added server side support for deleting shards (#299)

2023-03-23

Changed shard id type to int64 (#296) With namespaces, we'll be able to create and delete shards. It will be safer to use `int64` rather than `uint32` to avoid having to deal with any rollover.

2023-03-22

Passing namespace to server components for metrics and logs (#292)

2023-03-21

Added namespace into assignments dispatcher (#291)

2023-03-20

Added support in coordinator for multiple namespaces (#289) Added concept of "default" namespace. `ReplicationFactor` and `InitialShardsCount` are now moved at the namespace level.

2023-02-24

Make KeepAlives non-streaming (#267)

2023-02-08

Streaming read (#257)

2023-02-01

Remove batched list (#250) stacked on #245 diff: https://github.com/nahguam/oxia/compare/add-list-rpc...nahguam:oxia:remove-batched-list?expand=1

Split List out to separate streaming rpc (#245)

2023-01-28

Added client identity (#249) Added concept of client identity in the API: 1. A client can pass an optional `Identity` option (string). If none is passed, a UUID is internally generated 2. A session is tied to a particular client's identity 3. The client identity is the same across all the shards 4. The identity is exposed in the `Version` object (for ephemeral records) 5. Applications can make several uses of client identity: 1. Specify the client identity as in "node-port" of a particular service 2. Debug to know who created an ephemeral record 3. Allow creating more complex abstractions by identifying "was this record created by me?" 4. Maintain the identity across restarts of the same applications (e.g.: restart a pod that comes back up with the same name and will use the same client identity). There will be no effort from Oxia to disallow conflicting identities, though they should be discouraged.

2023-01-25

Add batching in the leader, before writing to WAL (#237)

2023-01-24

Renamed Payload -> Value (#239) Co-authored-by: Dave Maughan <dave.maughan@streamnative.io>

2023-01-20

Renamed Stat -> Version (#233) 1. Renaming the concept of `Stat` to `Version` object 2. Changed `Version` field into `VersionId` which represents the offset when the record was modified 3. Added `ModificationsCount` field to let users know how many changes since the record was last created Co-authored-by: Dave Maughan <dave.maughan@streamnative.io>

Renamed Fence -> NewTerm (#232) Co-authored-by: Dave Maughan <dave.maughan@streamnative.io>

proto cleanup (#230) * Fix import path (remove 'proto/' prefix) * Apply make proto_format * Apply protolint fixes * enums CAPITALS_WITH_UNDERSCORES * sessionId -> session_id * import order Co-authored-by: Matteo Merli <mmerli@apache.org>

2023-01-19

Break Snapshot chunks into small gRPC requests (#218)

Rename epoch -> term (#229) Using `term` as it's generally more suited to indicate different leader election periods (Stacked on #225, #226, #227 and #228) Co-authored-by: Dave Maughan <dave.maughan@streamnative.io>

Renamed AddEntries to Replicate (#228) (Stacked on top of #225, #226 and #227) Co-authored-by: Dave Maughan <dave.maughan@streamnative.io>

Renamed "index" to "offset" (#227) (Stacked on top of #225 & #226 ) Co-authored-by: Dave Maughan <dave.maughan@streamnative.io>

Renamed ShardsAssignmentsResponse -> ShardsAssignments (#226) (stacked on top of #225) Co-authored-by: Dave Maughan <dave.maughan@streamnative.io>

Renamed OxiaControl proto service to OxiaCoordination (#225) Renamed proto file `coordiationation.proto` -> `replication.proto` and the Service to `OxiaCoordination` Co-authored-by: Dave Maughan <dave.maughan@streamnative.io>

2023-01-15

Adding license and license headers (#210) Added license, headers and tool to add and verify in CI. * `make license-check` * `make license-format`

2023-01-06

Session Manager (#120) https://www.notion.so/streamnativeio/Design-for-session-management-b3be2beca3a941bf8c2313a0ddeaf8b3

2023-01-05

Leader sends snapshot to followers when needed (#176) The leader will send a snapshot to followers when either: * The follower is getting bootstrapped * The follower is tailing at an entry that was already trimmed in the leader wal (Stacked on top of #173) Co-authored-by: Dave Maughan <davidamaughan@gmail.com>

2023-01-04

Added client implementation of notifications (#166) Stacked on top of #164 & #165 Client API and implementation for notifications Co-authored-by: Andras Beni <andras.beni@streamnative.io> Co-authored-by: Dave Maughan <dave.maughan@streamnative.io>

2023-01-02

When the coordinator restarts, check leader status (#163) When the coordinator restarts, we need to double-check if the leader is up & running and in the expected state, otherwise, trigger a new leader election. Added a new `GetStatus()` RPC for the coordinator to get info about storage nodes.

2022-12-29

Notification support in DB (#164) First part, added support in DB to handle the log of notifications. As described in https://www.notion.so/streamnativeio/Notifications-4f1a9c5e09da4346a05cf27f6cf0f3b0 Co-authored-by: Andras Beni <andras.beni@streamnative.io>

2022-12-20

Shard upperbound inclusive (#145)

2022-12-19

Ensure entries timestamp are assigned only once (#139) We are currently assigning the timestamps on entries at the time when they get applied on the db layer. The problem is that this happens in each node (leader and followers) and the result is that the timestamp will change after a new leader election. We need a stable timestamp that is assigned only once in the leader and gets applied in all the followers. Added end-to-end test that verify the "stat" is the same after a new leader is elected.

Fixed truncate (#141) Fixed the logic of truncate on 2 aspects: 1. We need to be able to truncate a follower after a restart, as long as it's on the same epoch 2. The leader needs to compare the head index of a late follower with its own head index "at the moment of the leader election" and it should never ask to truncate to an offset which is past the last offset of the follower

2022-12-16

Follower controller handling of snapshot (#132) * Follower controller handling of snapshot * Update server/internal_rpc_server.go Co-authored-by: Andras Beni <andras.beni@streamnative.io> Co-authored-by: Andras Beni <andras.beni@streamnative.io>

2022-12-15

Remove epoch from RPC responses (#123) * Coordinator tests * Fixed lint * Perform fencing in parallel * Fixed race conditions * Fixed lint * Fixed lint * Retry fencing failed followers * Fixed lint * Fixed race condition in test * Improved testing assertion * Fixed duplicated CLI options * Remove epoch from RPC responses * Fixes * Use individual response types * Fixed merge issue * fixed compile

Retry fencing failed followers (#122) * Coordinator tests * Fixed lint * Perform fencing in parallel * Fixed race conditions * Fixed lint * Fixed lint * Retry fencing failed followers * Fixed lint * Fixed race condition in test * Improved testing assertion * Fixed duplicated CLI options

2022-12-08

[Improvement] GetRange → List (#109) * GetRange → List * GetRange → List Co-authored-by: Dave Maughan <dave.maughan@streamnative.io>

2022-12-03

Improved logic for epoch checking in follower controller (#92)

2022-11-30

Assignment dispatcher (#62) * Assignment dispatcher * Almost tests * Roll back string replace * Revert mock refactoring * Fix standalone server's leader port * Add tests * Review comments * typo * Remove unused field

2022-11-29

Refactoring to use Offset instead of EntryId where possible (#74) When referring to a position in the log, it's not necessary to always carry the EntryId(Epoch, Offset) around. While the entry id is needed in Fence & Truncate calls, once we have a leader or follower, they can refer to positions with just the offset, playing from the commit index to head index. Modifications * For a given shards: * valid offsets are going from 0 to N * -1 the InvalidOffset * When the WAL is empty, the last offset is -1 * Tidwall/wal is based on that the first entry has index 1 -> Therefore we are translating from 0..N to 1..N+1. * In the long run it would probably make sense to fork the wal code inside our codebase and make some adjustments. eg: the idx starting at 0 and the serialization format (which we don't need since we already have one). * Changed the wal implementations to avoid tracking the wal readers, since it was not needed and was causing deadlocks given the different order of locks between the wal and the readers. * Refactoring to use Offset instead of EntryId where possible * Avoid atomic.Int64 type since it's only in Go 1.19 * Fixed typo

2022-11-28

Refactor leader changes (#58) * Follower cursor implementation * Leader controller refactoring * Update server/follower_controller.go Co-authored-by: Andras Beni <andras.beni@streamnative.io> * Removed unused message * Fixed issue of sending fence requests to both leader and follower * Fixes after merge with main * Fixed test Co-authored-by: Andras Beni <andras.beni@streamnative.io> Co-authored-by: Dave Maughan <dave.maughan@streamnative.io>

2022-11-21

Remove reconfiguration from proto & go code (#51)

[Doc] Comment typos (#53) * Fix typos * Typo * Note sort order * Typo^2

Fix typos (#52)

[Improvement] Rename Status.BAD_VERSION (#49) * Rename Status.BAD_VERSION * Further instances

2022-11-16

Client side shard management (#29) * Client side shard management * CR changes * lint Co-authored-by: Matteo Merli <mmerli@apache.org>

2022-11-15

Split client batch to separate writes and reads (#33)

2022-11-14

Use shard-id field in all internal RPC requests (#27) * Use shard-id field in all internal RPC requests * Fixed writing stream response back * Fixed compile error

2022-11-09

Added option to have multiple hashing/sharding schemes (#22)

[Cleanup] Proto cleanup (#21)

2022-11-08

Added DB interface and impl (#19) * Indexed database batch * Added DB interface and impl * Merged with client proto changes and broke down into methods

[Feature] Client proto service (#20) * [Feature] Client proto service First cut of the client protobuf service for Milestone 1 * rename range requests and responses * Merge ShardAssignment and Shard

2022-11-07

Shard manager implementation (#15) * Shard manager implementation * Remove diff files Co-authored-by: Matteo Merli <mmerli@apache.org>

2021-12-28

Linked basic put operation

2021-12-23

Added the logic to open/close leader and follower controllers

2021-12-21

Initial import