package meta

Get desktop application:
View/edit binary Protocol Buffers messages

• rpc Export (Empty, stream ExportedChunk)

  meta.proto:385

  Export all meta data. Including raft hard state, logs and state machine. The exported data is a list of json strings in form of `(tree_name, sub_tree_prefix, key, value)`.

• rpc ExportV1 (ExportRequest, stream ExportedChunk)

  meta.proto:397

  Export all meta data. Including raft hard state, logs and state machine. The exported data is a list of json strings in form of: `(tree_name, {"<key_space>":{"key":<key>,"value":<value>}})`. ```text ["raft_log",{"Logs":{"key":83,"value":{"log_id":{"leader_id":{"term":1,"node_id":0},"index":83},"payload":{"Normal":{"txid":null,"time_ms":1667290974891,"cmd":{"UpsertKV":{"key":"__fd_clusters/test_tenant/test_cluster/databend_query/KMZ4VvqDFVExlZFThKDzZ1","seq":{"GE":1},"value":"AsIs","value_meta":{"expire_at":1667291034}}}}}}}}] ["state_machine/0",{"Nodes":{"key":1,"value":{"name":"1","endpoint":{"addr":"localhost","port":28103},"grpc_api_advertise_address":"0.0.0.0:9191"}}}] ```

  message ExportRequest

  meta.proto:54

  Request meta-service to export all data in a stream.

  • optional uint64 chunk_size = 10

    The number of lines of json string contained in a stream item. Note that too many lines in a stream item may cause "gRPC message too large" error. The default chunk_size is 32.

• rpc GetClientInfo (Empty, ClientInfo)

  meta.proto:416

  Respond with the information about the client. Since: 2022-09-09 0.8.30

  message ClientInfo

  meta.proto:298

  • string client_addr = 10

    The address of the connected in form of "<ip>:<port>"

  • optional uint64 server_time = 20

    The timestamp when the meta-server received the request in milliseconds since 1970 . It is created with `SeqV::now_ms()` To convert it back to time with: `SystemTime::UNIX_EPOCH + Duration::from_millis(server_time)`

• rpc GetClusterStatus (Empty, ClusterStatus)

  meta.proto:412

  Get cluster status Since: 2023-10-19

  message ClusterStatus

  meta.proto:264

  • uint64 id = 1

  • string binary_version = 2

  • string data_version = 3

  • string endpoint = 4

  • uint64 raft_log_size = 5

  • string state = 6

  • bool is_leader = 7

  • uint64 current_term = 8

  • uint64 last_log_index = 9

  • string last_applied = 10

  • optional string snapshot_last_log_id = 11

  • optional string purged = 12

  • optional string leader = 13

  • map<uint64, string> replication = 14

  • repeated string voters = 15

  • repeated string non_voters = 16

  • uint64 last_seq = 17

  • uint64 snapshot_key_count = 18

  • optional RaftLogStatus raft_log_status = 19

• rpc Handshake (stream HandshakeRequest, stream HandshakeResponse)

  meta.proto:361

  handshake

  message HandshakeRequest

  meta.proto:43

  • uint64 protocol_version = 1

  • bytes payload = 2

  message HandshakeResponse

  meta.proto:48

  • uint64 protocol_version = 1

  • bytes payload = 2

• rpc KvApi (RaftRequest, RaftReply)

  meta.proto:367

  General KV API for get, mget, list, and upsert; It is introduced to replace `WriteMsg` and `ReadMsg`. 2022-09-14: since: 0.8.35

• rpc KvReadV1 (RaftRequest, stream StreamItem)

  meta.proto:378

  Handle application read request. This API is not exposed to client directly, but is only used for internal request forwarding. The request will be forwarded to leader if current node is not leader. It returns a stream of `StreamItem`. - For single-reply request, the stream contains only one item, e.g. `Get`. - For multi-reply request, the stream contains multiple items, e.g. `MGet` and `List`. 2023-10-17: since 1.2.163

• rpc MemberList (MemberListRequest, MemberListReply)

  meta.proto:408

  Get MetaSrv member list endpoints

  message MemberListRequest

  meta.proto:39

  • string data = 1

  message MemberListReply

  meta.proto:41

  • repeated string data = 1

• rpc Transaction (TxnRequest, TxnReply)

  meta.proto:405

  message TxnRequest

  meta.proto:223

  A Transaction request sent to the databend-meta service. To provide backward compatibility, the `TxnRequest` is processed in the following order: - Loop and evaluate the condition in the `operations`, and execute the corresponding operation, if the condition is met. This will stop and return once a condition is met and one of the corresponding operations is executed. - If none of the conditions are met, the `condition` as the **last** condition will be evaluated. And the `if_then` will be executed and return. - If none operation are executed, `else_then` will be executed.

  • repeated ConditionalOperation operations = 4

    Series of conditional operations to execute. It will stop once a condition is met and one of the corresponding operations is executed

  • repeated TxnCondition condition = 1

    `condition` is a list of predicates. If all of them success, the `if_then` will be executed, otherwise `else_then` op will be executed.

  • repeated TxnOp if_then = 2

    `if_then` is a list of operations will be executed when all condition evaluates to true.

  • repeated TxnOp else_then = 3

    `else_then` is a list of operations will be executed when not all condition evaluates to true.

  message TxnReply

  meta.proto:251

  TransactionResponse represents the result of executing a transaction. `execution_path` identifies condition that is executed and is identified as: - "operation:{index}": executed operation at index - "then": `condition` field were met, and the `if_then` operation is executed. - "else": neither operations nor condition are met and `else_then` is executed. `success` is set to false only when `else_then` is executed.

  • bool success = 1

  • repeated TxnOpResponse responses = 2

  • string execution_path = 4

    Identifies which execution path was taken

• rpc Watch (WatchRequest, stream WatchResponse)

  meta.proto:403

  Add watch key stream. Whenever the watch key data updated, client will be notified across the stream.

  message WatchRequest

  meta.proto:66

  • string key = 1

    key is the key to register for watching.

  • optional string key_end = 2

    `key_end`` is the end of the range [key, key_end) to watch. If key_end is None, then watch only key. If want to watch prefix of key, use `get_start_and_end_of_prefix` to generate [key, key_end).

  • WatchRequest.FilterType filter_type = 3

  • bool initial_flush = 4

    Whether to send current values of all watched keys when watch starts Useful for initializing client-side caches. The client: - first get a full copy of the key-values, - then update every time a key-value is changed.

  message WatchResponse

  meta.proto:104

  Response in a watch stream from server to the client.

  • optional Event event = 1

    The event containing key-value change information or a value emitted during initial-flush. This includes the key, current value (if any), and previous value (if any).

  • bool is_initialization = 2

    Indicates whether this event is part of the initialization. When true: - The event represents an existing key-value record at the time the watch was established - These events are sent when initial_flush=true was specified in WatchRequest When false: - The event represents a real-time change that occurred after the watch was established - A special event with no key-value data may be sent to indicate the completion of the initial flush phase

• rpc AppendEntries (RaftRequest, RaftReply)

  meta.proto:352

• rpc Forward (RaftRequest, RaftReply)

  meta.proto:344

  Forward a request to another node.

• rpc InstallSnapshotV003 (stream SnapshotChunkRequestV003, SnapshotResponseV003)

  meta.proto:354

  Added in 1.2.547, 2024-06-27

  message SnapshotChunkRequestV003

  meta.proto:323

  The item of snapshot chunk stream. The first item contains `rpc_meta`, including the application defined format of this snapshot data, the leader vote and snapshot-meta. Since the second item, the `rpc_meta` should be empty and will be ignored by the receiving end.

  • optional string rpc_meta = 10

    json serialized meta data, including vote and snapshot_meta. ```text (SnapshotFormat, Vote, SnapshotMeta) ```

  • bytes chunk = 20

    Snapshot data chunk

  message SnapshotResponseV003

  meta.proto:336

  V003 snapshot response

  • string vote = 10

    json serialized remote vote

• rpc KvReadV1 (RaftRequest, stream StreamItem)

  meta.proto:348

  Handling internally redirected KvReadV1 request. Without checking token.

• rpc TransferLeader (TransferLeaderRequest, Empty)

  meta.proto:356

  message TransferLeaderRequest

  meta.proto:33

  • optional Vote from = 1

  • uint64 to = 2

  • optional LogId last_log_id = 3

• rpc Vote (RaftRequest, RaftReply)

  meta.proto:355

BooleanExpression represents a tree of transaction conditions combined with logical operators. It enables complex condition checking by allowing both simple conditions and nested expressions.

Used in: ConditionalOperation

• BooleanExpression.CombiningOperator operator = 1

  Operator determining how child expressions and conditions are combined

• repeated BooleanExpression sub_expressions = 2

  Nested boolean expressions, allowing tree-like structure Example: (A AND B) OR (C AND D) where A,B,C,D are conditions

• repeated TxnCondition conditions = 3

  Leaf-level transaction conditions These are the actual checks performed against the state.

Logical operator to combine multiple conditions, including sub compound conditions or simple conditions.

Used in: BooleanExpression

• AND = 0

• OR = 1

Represents a set of operations that execute only when specified conditions are met.

Used in: TxnRequest

• optional BooleanExpression predicate = 1

  Tree of conditions that must be satisfied for operations to execute

• repeated TxnOp operations = 2

  Operations to execute when condition_tree evaluates to true These operations are executed in sequence

Used as request type in: MetaService.Export, MetaService.GetClientInfo, MetaService.GetClusterStatus

Used as response type in: RaftService.TransferLeader

(message has no fields)

Used in: WatchResponse

• string key = 1

• optional SeqV current = 2

  current hold current value of key(if any)

• optional SeqV prev = 3

  prev value of key(if any)

Data chunk for export/import meta data

Used as response type in: MetaService.Export, MetaService.ExportV1

• repeated string data = 10

Return the value by key, then add the delta to the key. This operation assume the value bytes is a json encoded `uint64`, e.g. `1025` in bytes is `b"1025"`. If the result is negative, it will be set to zero.

Used in: TxnOp

• string key = 1

  The key to fetch and add the delta.

• optional uint64 match_seq = 3

  Assert the seq number of the record before update. - If it does not match, no update will be made and the record seq number won't change. - If it is None, the update will always be made.

• int64 delta = 2

  The delta to add to the value.

Response for FetchAddU64, contains the value before and after `add`

Used in: TxnOpResponse

• string key = 1

• uint64 before_seq = 2

  The seq number of the record before update. `0` if the key does not exist.

• uint64 before = 3

  The value before update.

• uint64 after_seq = 4

  The seq number of the record after update. It should always be greater than `before_seq`.

• uint64 after = 5

  The value after update.

Used in: SeqV

• optional uint64 expire_at = 1

  Expiration time in **seconds or milliseconds** since Unix epoch (1970-01-01). The interpretation depends on the magnitude of the value: - Values > `1_000_000_000_000`: treated as milliseconds since epoch - Values ≤ `1_000_000_000_000`: treated as seconds since epoch

A quorum granted or not granted Leader.

• uint64 term = 1

• uint64 node_id = 2

The log id

Used in: TransferLeaderRequest

• uint64 term = 1

• uint64 node_id = 2

• uint64 index = 3

Request to put a value with an auto-generated sequential key. This operation appends a sequence number to the given prefix to create a unique key, then stores the value under that key. The sequence number is maintained using a separate sequence key and is guaranteed to be monotonically increasing.

Used in: TxnOp

• string prefix = 1

  Key prefix for sequential put operation. A sequence number will be appended to the prefix to form the final key. The sequence number is u64 encoded as a 21-digit zero-padded string with underscores every 3 digits for readability(u64 max is 20 digits). The total length of the encoded sequence number is 21 digit + 6 `_` = 27 chars. Examples: - `/foo/bar/` → `/foo/bar/000_000_000_000_001_200_123` - `/foo/bar_` → `/foo/bar_000_000_000_000_001_200_123`

• string sequence_key = 2

  Key used to generate sequence numbers. Example: `/__seq` The sequence value must be a JSON-encoded uint64 (e.g., `1025` as `b"1025"`), compatible with FetchAddU64 operation. Because internally, the sequence number is built with `FetchAddU64(1)`. User responsibility: ensure sequence key validity. If removed and re-added, the sequence number will reset.

• bytes value = 3

  The value to put.

• optional uint64 expires_at_ms = 4

  Absolute expire time in milliseconds since epoch, specified by the client

• optional uint64 ttl_ms = 5

  Time to live in milliseconds (relative expire time since raft-log proposal, the time on raft-leader). If set, `expires_at_ms` is ignored.

Status about local raft-log storage

Used in: ClusterStatus

• uint64 cache_items = 1

• uint64 cache_used_size = 2

• uint64 wal_total_size = 3

• uint64 wal_open_chunk_size = 4

• uint64 wal_offset = 5

• uint64 wal_closed_chunk_count = 6

• uint64 wal_closed_chunk_total_size = 7

• map<string, uint64> wal_closed_chunk_sizes = 8

Used as response type in: MetaService.KvApi, RaftService.AppendEntries, RaftService.Forward, RaftService.Vote

• string data = 1

• string error = 2

Used as request type in: MetaService.KvApi, MetaService.KvReadV1, RaftService.AppendEntries, RaftService.Forward, RaftService.KvReadV1, RaftService.Vote

• string data = 1

Used in: Event, StreamItem, TxnDeleteResponse, TxnGetResponse, TxnPutResponse

• uint64 seq = 1

• bytes data = 2

• optional KVMeta meta = 3

Item for a Streaming read reply, e.g., for `Mget` and `List`.

Used as response type in: MetaService.KvReadV1, RaftService.KvReadV1

• string key = 1

• optional SeqV value = 2

messages for txn

Used in: BooleanExpression, TxnRequest

• string key = 1

• oneof target

  • bytes value = 2

    Compare the stored value of `key` against the given value.

  • uint64 seq = 3

    Compare the stored seq of `key` against the given seq.

  • uint64 keys_with_prefix = 5

    Compare the count of keys having the prefix `key` against the given value. Usually when using this option, append a slash `/` to the end of the prefix `key`. For example, if you want to count the keys with prefix `foo`, you should use `foo/` as the `key`.

• TxnCondition.ConditionResult expected = 4

  the expected result of condition, if `expected` match the condition result, then the `if_then` op will be executed, otherwise `else_then` op will be executed.

condition result

Used in: TxnCondition

• EQ = 0

• GT = 1

• GE = 2

• LT = 3

• LE = 4

• NE = 5

Delete by prefix request and response

Used in: TxnOp

• string prefix = 1

Used in: TxnOpResponse

• string prefix = 1

• uint32 count = 2

Delete request and response

Used in: TxnOp

• string key = 1

• bool prev_value = 2

  if or not return the prev value Not used anymore

• optional uint64 match_seq = 3

  Delete only if the `seq` matches the specified value. Such a condition skips changed record and does not result in a transaction failure. A key use case for this feature is to support `truncate-table`: the deletion of only non-modified files from a snapshot view of copied files. A table contains associated `copied-files`. Suppose Process 1 updates the table by modifying `f-3`, removing `f-4`, and adding `f-7`. In this case, Process 2 can delete only the non-modified files: ``` Process 1 Process 2 // Add copied files // Attempt to delete copied files ------------------------------------------------------------- // 1. Initial Table-A(seq:1): f-2(seq:2) f-3(seq:3) f-4(seq:4) // 2. Acquire a snapshot view Table-A(seq:1): f-2(seq:2) f-3(seq:3) f-4(seq:4) // 3. Modify table and files Table-A(seq:7): f-2(seq:2) f-3(seq:5) // updated // removed f-4 f-7(seq:6) // added // 4. Sequence of table changed from 1 to 7 // Fetch table again Table-A(seq:7): // 5. Remove only unmodified file Table-A(seq:8): // f-2 is the only file deleted f-3(seq:5) // sequence number updated f-7(seq:6) // sequence number updated ```

Used in: TxnOpResponse

• string key = 1

• bool success = 2

• optional SeqV prev_value = 3

Request to get the seq and value by key

Used in: TxnOp

• string key = 1

Used in: TxnOpResponse

• string key = 1

• optional SeqV value = 2

Used in: ConditionalOperation, TxnRequest

• oneof request

  • TxnGetRequest get = 1

    TODO add Echo to get a response

  • TxnPutRequest put = 2

  • TxnDeleteRequest delete = 3

  • TxnDeleteByPrefixRequest delete_by_prefix = 4

  • FetchAddU64 fetch_add_u64 = 5

  • PutSequential put_sequential = 6

Used in: TxnReply

• oneof response

  • TxnGetResponse get = 1

  • TxnPutResponse put = 2

  • TxnDeleteResponse delete = 3

  • TxnDeleteByPrefixResponse delete_by_prefix = 4

  • FetchAddU64Response fetch_add_u64 = 5

Put request and response

Used in: TxnOp

• string key = 1

• bytes value = 2

• bool prev_value = 3

  Whether or not to return the prev value Not used anymore

• optional uint64 expire_at = 4

  Absolute expire time in seconds or milliseconds since Unix epoch (1970-01-01).

• optional uint64 ttl_ms = 5

  Time to last in milliseconds. TTL is the relative expire time, since the raft-log proposed. If `ttl_ms` is set, `expire_at` is ignored.

Used in: TxnOpResponse

• string key = 1

• optional SeqV prev_value = 2

  The value before put

• optional SeqV current = 3

  The value after put

The vote for RequestVote etc.

Used in: TransferLeaderRequest

• uint64 term = 1

• uint64 node_id = 2

• bool committed = 3

Used in: WatchRequest

• ALL = 0

  recv all kind update event.

• UPDATE = 1

  filter only update event.

• DELETE = 2

  filter only delete event.