package meta

Get desktop application:
View/edit binary Protocol Buffers messages

• rpc Handshake (stream HandshakeRequest, stream HandshakeResponse)

  meta.proto:373

  handshake

  message HandshakeRequest

  meta.proto:43

  • uint64 protocol_version = 1

  • bytes payload = 2

  message HandshakeResponse

  meta.proto:48

  • uint64 protocol_version = 1

  • bytes payload = 2

• rpc KvApi (RaftRequest, RaftReply)

  meta.proto:379

  General KV API for get, mget, list, and upsert; It is introduced to replace `WriteMsg` and `ReadMsg`. 2022-09-14: since: 0.8.35

• rpc KvReadV1 (RaftRequest, stream StreamItem)

  meta.proto:390

  Handle application read request. This API is not exposed to client directly, but is only used for internal request forwarding. The request will be forwarded to leader if current node is not leader. It returns a stream of `StreamItem`. - For single-reply request, the stream contains only one item, e.g. `Get`. - For multi-reply request, the stream contains multiple items, e.g. `MGet` and `List`. 2023-10-17: since 1.2.163

• rpc Export (Empty, stream ExportedChunk)

  meta.proto:397

  Export all meta data. Including raft hard state, logs and state machine. The exported data is a list of json strings in form of `(tree_name, sub_tree_prefix, key, value)`.

• rpc ExportV1 (ExportRequest, stream ExportedChunk)

  meta.proto:409

  Export all meta data. Including raft hard state, logs and state machine. The exported data is a list of json strings in form of: `(tree_name, {"<key_space>":{"key":<key>,"value":<value>}})`. ```text ["raft_log",{"Logs":{"key":83,"value":{"log_id":{"leader_id":{"term":1,"node_id":0},"index":83},"payload":{"Normal":{"txid":null,"time_ms":1667290974891,"cmd":{"UpsertKV":{"key":"__fd_clusters/test_tenant/test_cluster/databend_query/KMZ4VvqDFVExlZFThKDzZ1","seq":{"GE":1},"value":"AsIs","value_meta":{"expire_at":1667291034}}}}}}}}] ["state_machine/0",{"Nodes":{"key":1,"value":{"name":"1","endpoint":{"addr":"localhost","port":28103},"grpc_api_advertise_address":"0.0.0.0:9191"}}}] ```

  message ExportRequest

  meta.proto:54

  Request meta-service to export all data in a stream.

  • optional uint64 chunk_size = 10

    The number of lines of json string contained in a stream item. Note that too many lines in a stream item may cause "gRPC message too large" error. The default chunk_size is 32.

• rpc Watch (WatchRequest, stream WatchResponse)

  meta.proto:415

  Add watch key stream. Whenever the watch key data updated, client will be notified across the stream.

  message WatchRequest

  meta.proto:66

  • string key = 1

    key is the key to register for watching.

  • optional string key_end = 2

    `key_end`` is the end of the range [key, key_end) to watch. If key_end is None, then watch only key. If want to watch prefix of key, use `get_start_and_end_of_prefix` to generate [key, key_end).

  • WatchRequest.FilterType filter_type = 3

  • bool initial_flush = 4

    Whether to send current values of all watched keys when watch starts Useful for initializing client-side caches. The client: - first get a full copy of the key-values, - then update every time a key-value is changed.

  message WatchResponse

  meta.proto:104

  Response in a watch stream from server to the client.

  • optional Event event = 1

    The event containing key-value change information or a value emitted during initial-flush. This includes the key, current value (if any), and previous value (if any).

  • bool is_initialization = 2

    Indicates whether this event is part of the initialization. When true: - The event represents an existing key-value record at the time the watch was established - These events are sent when initial_flush=true was specified in WatchRequest When false: - The event represents a real-time change that occurred after the watch was established - A special event with no key-value data may be sent to indicate the completion of the initial flush phase

• rpc Transaction (TxnRequest, TxnReply)

  meta.proto:417

  message TxnRequest

  meta.proto:218

  A Transaction request sent to the databend-meta service. To provide backward compatibility, the `TxnRequest` is processed in the following order: - Loop and evaluate the condition in the `operations`, and execute the corresponding operation, if the condition is met. This will stop and return once a condition is met and one of the corresponding operations is executed. - If none of the conditions are met, the `condition` as the **last** condition will be evaluated. And the `if_then` will be executed and return. - If none operation are executed, `else_then` will be executed.

  • repeated ConditionalOperation operations = 4

    Series of conditional operations to execute. It will stop once a condition is met and one of the corresponding operations is executed

  • repeated TxnCondition condition = 1

    `condition` is a list of predicates. If all of them success, the `if_then` will be executed, otherwise `else_then` op will be executed.

  • repeated TxnOp if_then = 2

    `if_then` is a list of operations will be executed when all condition evaluates to true.

  • repeated TxnOp else_then = 3

    `else_then` is a list of operations will be executed when not all condition evaluates to true.

  message TxnReply

  meta.proto:246

  TransactionResponse represents the result of executing a transaction. `execution_path` identifies condition that is executed and is identified as: - "operation:{index}": executed operation at index - "then": `condition` field were met, and the `if_then` operation is executed. - "else": neither operations nor condition are met and `else_then` is executed. `success` is set to false only when `else_then` is executed.

  • bool success = 1

  • repeated TxnOpResponse responses = 2

  • string error = 3

    Not used

  • string execution_path = 4

    Identifies which execution path was taken

• rpc MemberList (MemberListRequest, MemberListReply)

  meta.proto:420

  Get MetaSrv member list endpoints

  message MemberListRequest

  meta.proto:39

  • string data = 1

  message MemberListReply

  meta.proto:41

  • repeated string data = 1

• rpc GetClusterStatus (Empty, ClusterStatus)

  meta.proto:424

  Get cluster status Since: 2023-10-19

  message ClusterStatus

  meta.proto:258

  • uint64 id = 1

  • string binary_version = 2

  • string data_version = 3

  • string endpoint = 4

  • uint64 raft_log_size = 5

  • string state = 6

  • bool is_leader = 7

  • uint64 current_term = 8

  • uint64 last_log_index = 9

  • string last_applied = 10

  • optional string snapshot_last_log_id = 11

  • optional string purged = 12

  • optional string leader = 13

  • map<uint64, string> replication = 14

  • repeated string voters = 15

  • repeated string non_voters = 16

  • uint64 last_seq = 17

  • uint64 snapshot_key_count = 18

  • optional RaftLogStatus raft_log_status = 19

• rpc GetClientInfo (Empty, ClientInfo)

  meta.proto:428

  Respond with the information about the client. Since: 2022-09-09 0.8.30

  message ClientInfo

  meta.proto:292

  • string client_addr = 10

    The address of the connected in form of "<ip>:<port>"

  • optional uint64 server_time = 20

    The timestamp when the meta-server received the request in milliseconds since 1970 . It is created with `SeqV::now_ms()` To convert it back to time with: `SystemTime::UNIX_EPOCH + Duration::from_millis(server_time)`

• rpc Forward (RaftRequest, RaftReply)

  meta.proto:356

  Forward a request to another node.

• rpc KvReadV1 (RaftRequest, stream StreamItem)

  meta.proto:360

  Handling internally redirected KvReadV1 request. Without checking token.

• rpc AppendEntries (RaftRequest, RaftReply)

  meta.proto:364

• rpc InstallSnapshotV1 (SnapshotChunkRequest, RaftReply)

  meta.proto:365

  message SnapshotChunkRequest

  meta.proto:310

  • uint64 ver = 100

  • string rpc_meta = 10

    json serialized meta data, including vote and snapshot_meta. ```text (Vote, SnapshotMeta) ```

  • optional SnapshotChunkV1 chunk = 20

• rpc InstallSnapshotV003 (stream SnapshotChunkRequestV003, SnapshotResponseV003)

  meta.proto:366

  message SnapshotChunkRequestV003

  meta.proto:335

  The item of snapshot chunk stream. The first item contains `rpc_meta`, including the application defined format of this snapshot data, the leader vote and snapshot-meta. Since the second item, the `rpc_meta` should be empty and will be ignored by the receiving end.

  • optional string rpc_meta = 10

    json serialized meta data, including vote and snapshot_meta. ```text (SnapshotFormat, Vote, SnapshotMeta) ```

  • bytes chunk = 20

    Snapshot data chunk

  message SnapshotResponseV003

  meta.proto:348

  V003 snapshot response

  • string vote = 10

    json serialized remote vote

• rpc Vote (RaftRequest, RaftReply)

  meta.proto:367

• rpc TransferLeader (TransferLeaderRequest, Empty)

  meta.proto:368

  message TransferLeaderRequest

  meta.proto:33

  • optional Vote from = 1

  • uint64 to = 2

  • optional LogId last_log_id = 3

BooleanExpression represents a tree of transaction conditions combined with logical operators. It enables complex condition checking by allowing both simple conditions and nested expressions.

Used in: ConditionalOperation

• BooleanExpression.CombiningOperator operator = 1

  Operator determining how child expressions and conditions are combined

• repeated BooleanExpression sub_expressions = 2

  Nested boolean expressions, allowing tree-like structure Example: (A AND B) OR (C AND D) where A,B,C,D are conditions

• repeated TxnCondition conditions = 3

  Leaf-level transaction conditions These are the actual checks performed against the state.

Logical operator to combine multiple conditions, including sub compound conditions or simple conditions.

Used in: BooleanExpression

• AND = 0

• OR = 1

Represents a set of operations that execute only when specified conditions are met.

Used in: TxnRequest

• optional BooleanExpression predicate = 1

  Tree of conditions that must be satisfied for operations to execute

• repeated TxnOp operations = 2

  Operations to execute when condition_tree evaluates to true These operations are executed in sequence

Used as request type in: MetaService.Export, MetaService.GetClientInfo, MetaService.GetClusterStatus

Used as response type in: RaftService.TransferLeader

(message has no fields)

Used in: WatchResponse

• string key = 1

• optional SeqV current = 2

  current hold current value of key(if any)

• optional SeqV prev = 3

  prev value of key(if any)

Data chunk for export/import meta data

Used as response type in: MetaService.Export, MetaService.ExportV1

• repeated string data = 10

Used in: SeqV

• optional uint64 expire_at = 1

A quorum granted or not granted Leader.

• uint64 term = 1

• uint64 node_id = 2

The log id

Used in: TransferLeaderRequest

• uint64 term = 1

• uint64 node_id = 2

• uint64 index = 3

Status about local raft-log storage

Used in: ClusterStatus

• uint64 cache_items = 1

• uint64 cache_used_size = 2

• uint64 wal_total_size = 3

• uint64 wal_open_chunk_size = 4

• uint64 wal_offset = 5

• uint64 wal_closed_chunk_count = 6

• uint64 wal_closed_chunk_total_size = 7

• map<string, uint64> wal_closed_chunk_sizes = 8

Used as response type in: MetaService.KvApi, RaftService.AppendEntries, RaftService.Forward, RaftService.InstallSnapshotV1, RaftService.Vote

• string data = 1

• string error = 2

Used as request type in: MetaService.KvApi, MetaService.KvReadV1, RaftService.AppendEntries, RaftService.Forward, RaftService.KvReadV1, RaftService.Vote

• string data = 1

Used in: Event, StreamItem, TxnDeleteResponse, TxnGetResponse, TxnPutResponse

• uint64 seq = 1

• bytes data = 2

• optional KVMeta meta = 3

Used in: SnapshotChunkRequest

• uint64 offset = 10

• bool done = 11

• bytes data = 12

Item for a Streaming read reply, e.g., for `Mget` and `List`.

Used as response type in: MetaService.KvReadV1, RaftService.KvReadV1

• string key = 1

• optional SeqV value = 2

messages for txn

Used in: BooleanExpression, TxnRequest

• string key = 1

• oneof target

  • bytes value = 2

    Compare the stored value of `key` against the given value.

  • uint64 seq = 3

    Compare the stored seq of `key` against the given seq.

  • uint64 keys_with_prefix = 5

    Compare the count of keys having the prefix `key` against the given value. Usually when using this option, append a slash `/` to the end of the prefix `key`. For example, if you want to count the keys with prefix `foo`, you should use `foo/` as the `key`.

• TxnCondition.ConditionResult expected = 4

  the expected result of condition, if `expected` match the condition result, then the `if_then` op will be executed, otherwise `else_then` op will be executed.

condition result

Used in: TxnCondition

• EQ = 0

• GT = 1

• GE = 2

• LT = 3

• LE = 4

• NE = 5

Delete by prefix request and response

Used in: TxnOp

• string prefix = 1

Used in: TxnOpResponse

• string prefix = 1

• uint32 count = 2

Delete request and response

Used in: TxnOp

• string key = 1

• bool prev_value = 2

  if or not return the prev value Not used anymore

• optional uint64 match_seq = 3

  Delete only if the `seq` matches the specified value. Such a condition skips changed record and does not result in a transaction failure. A key use case for this feature is to support `truncate-table`: the deletion of only non-modified files from a snapshot view of copied files. A table contains associated `copied-files`. Suppose Process 1 updates the table by modifying `f-3`, removing `f-4`, and adding `f-7`. In this case, Process 2 can delete only the non-modified files: ``` Process 1 Process 2 // Add copied files // Attempt to delete copied files ------------------------------------------------------------- // 1. Initial Table-A(seq:1): f-2(seq:2) f-3(seq:3) f-4(seq:4) // 2. Acquire a snapshot view Table-A(seq:1): f-2(seq:2) f-3(seq:3) f-4(seq:4) // 3. Modify table and files Table-A(seq:7): f-2(seq:2) f-3(seq:5) // updated // removed f-4 f-7(seq:6) // added // 4. Sequence of table changed from 1 to 7 // Fetch table again Table-A(seq:7): // 5. Remove only unmodified file Table-A(seq:8): // f-2 is the only file deleted f-3(seq:5) // sequence number updated f-7(seq:6) // sequence number updated ```

Used in: TxnOpResponse

• string key = 1

• bool success = 2

• optional SeqV prev_value = 3

Request to get the seq and value by key

Used in: TxnOp

• string key = 1

Used in: TxnOpResponse

• string key = 1

• optional SeqV value = 2

Used in: ConditionalOperation, TxnRequest

• oneof request

  • TxnGetRequest get = 1

    TODO add Echo to get a response

  • TxnPutRequest put = 2

  • TxnDeleteRequest delete = 3

  • TxnDeleteByPrefixRequest delete_by_prefix = 4

Used in: TxnReply

• oneof response

  • TxnGetResponse get = 1

  • TxnPutResponse put = 2

  • TxnDeleteResponse delete = 3

  • TxnDeleteByPrefixResponse delete_by_prefix = 4

Put request and response

Used in: TxnOp

• string key = 1

• bytes value = 2

• bool prev_value = 3

  Whether or not to return the prev value Not used anymore

• optional uint64 expire_at = 4

  Absolute expire time

• optional uint64 ttl_ms = 5

  Time to last in milliseconds. TTL is the relative expire time, since the raft-log applied. If `ttl_ms` is set, `expire_at` is ignored.

Used in: TxnOpResponse

• string key = 1

• optional SeqV prev_value = 2

The vote for RequestVote etc.

Used in: TransferLeaderRequest

• uint64 term = 1

• uint64 node_id = 2

• bool committed = 3

Used in: WatchRequest

• ALL = 0

  recv all kind update event.

• UPDATE = 1

  filter only update event.

• DELETE = 2

  filter only delete event.