package import_sstpb

Get desktop application:
View/edit binary Protocol Buffers messages

ImportSST provides a service to import a generated SST file to a region in TiKV. In order to import an SST file to a region, the user should: 1. Retrieve the meta of the region according to the SST file's range. 2. Upload the SST file to the servers where the region's peers locate in. 3. Issue an ingest request to the region's leader with the SST file's metadata. It's the user's responsibility to make sure that the SST file is uploaded to the servers where the region's peers locate in, before issue the ingest request to the region's leader. However, the region can be scheduled (so the location of the region's peers will be changed) or split/merged (so the range of the region will be changed), after the SST file is uploaded, but before the SST file is ingested. So, the region's epoch is provided in the SST file's metadata, to guarantee that the region's epoch must be the same between the SST file is uploaded and ingested later.

• rpc SwitchMode (SwitchModeRequest, SwitchModeResponse)

  import_sstpb.proto:41

  Switch to normal/import mode.

  message SwitchModeResponse

  import_sstpb.proto:104

  (message has no fields)

• rpc GetMode (GetModeRequest, GetModeResponse)

  import_sstpb.proto:43

  Get import mode(normal/import).

  message GetModeRequest

  import_sstpb.proto:107

  (message has no fields)

  message GetModeResponse

  import_sstpb.proto:110

  • SwitchMode mode = 1

• rpc Upload (stream UploadRequest, UploadResponse)

  import_sstpb.proto:45

  Upload an SST file to a server.

  message UploadRequest

  import_sstpb.proto:151

  • oneof chunk

    • SSTMeta meta = 1

    • bytes data = 2

  message UploadResponse

  import_sstpb.proto:158

  (message has no fields)

• rpc Ingest (IngestRequest, IngestResponse)

  import_sstpb.proto:47

  Ingest an uploaded SST file to a region.

  message IngestRequest

  import_sstpb.proto:161

  • optional kvrpcpb.Context context = 1

  • optional SSTMeta sst = 2

• rpc Compact (CompactRequest, CompactResponse)

  import_sstpb.proto:49

  Compact the specific range for better performance.

  message CompactResponse

  import_sstpb.proto:184

  (message has no fields)

• rpc SetDownloadSpeedLimit (SetDownloadSpeedLimitRequest, SetDownloadSpeedLimitResponse)

  import_sstpb.proto:51

  message SetDownloadSpeedLimitRequest

  import_sstpb.proto:276

  • uint64 speed_limit = 1

    The download speed limit (bytes/second). Set to 0 for unlimited speed.

  message SetDownloadSpeedLimitResponse

  import_sstpb.proto:281

  (message has no fields)

• rpc Download (DownloadRequest, DownloadResponse)

  import_sstpb.proto:54

  Download an SST file from an external storage, and performs key-rewrite after downloading.

  message DownloadRequest

  import_sstpb.proto:187

  • map<string, SSTMeta> ssts = 1

    Map represents the map of <name, SSTMeta>. We'll generate all SSTMeta into one SST File.

  • uint64 resolved_ts = 3

    resolved_ts is used to merge related SST Files.

  • optional SSTMeta sst = 2

    The SST meta used to identify the downloaded file. Must be the same among all nodes in the same Raft group. Note: the "crc32" and "cf_name" fields are ignored in this request, and the "range" field represents the closed key range after rewrite (as origin keys in encoded representation).

  • string name = 9

    The file name of the SST file.

  • optional RewriteRule rewrite_rule = 13

    Performs a key prefix rewrite after downloading the SST file. All keys in the SST will be rewritten as: new_key = new_key_prefix + old_key[len(old_key_prefix)..] When used for TiDB, rewriting the prefix changes the table ID. Please note that key-rewrite is applied on the origin keys in encoded representation (the SST itself should still use data keys in encoded representation). You need to ensure that the keys before and after rewriting are in the same order, otherwise the RPC request will fail.

  • repeated RewriteRule sorted_rewrite_rules = 20

  • optional backup.StorageBackend storage_backend = 14

  • string storage_cache_id = 17

    The identity for the stroage backend. When this field presents, the storage would be cached. If there is a cached storage, TiKV would use it driectly.

  • bool is_raw_kv = 15

  • optional backup.CipherInfo cipher_info = 16

    cipher_info is used to decrypt sst when download sst

  • DownloadRequestType request_type = 18

    The type of the download request.

  • optional kvrpcpb.Context context = 19

  message DownloadResponse

  import_sstpb.proto:253

  • optional Range range = 1

    The actual key range (after rewrite) of the downloaded SST. The range is inclusive in both ends.

  • bool is_empty = 2

    Whether the SST is empty. An empty SST is prohibited in TiKV, do not ingest if this field is true. (Deprecated, should be replaced by checking `length == 0` in the future)

  • optional Error error = 3

  • uint32 crc32 = 4

    The CRC32 checksum of the rewritten SST file (implementation can return zero, indicating the CRC32 was not calculated).

  • uint64 length = 5

    The actual length of the rewritten SST file.

  • repeated SSTMeta ssts = 6

    This field only return when file-copy backup enabled. Because it will merge many SST files in a download request.

• rpc Write (stream WriteRequest, WriteResponse)

  import_sstpb.proto:57

  Open a write stream to generate sst files

  message WriteRequest

  import_sstpb.proto:299

  • oneof chunk

    • SSTMeta meta = 1

    • WriteBatch batch = 2

  • optional kvrpcpb.Context context = 3

  message WriteResponse

  import_sstpb.proto:307

  • optional Error error = 1

  • repeated SSTMeta metas = 2

• rpc RawWrite (stream RawWriteRequest, RawWriteResponse)

  import_sstpb.proto:58

  message RawWriteRequest

  import_sstpb.proto:324

  • oneof chunk

    • SSTMeta meta = 1

    • RawWriteBatch batch = 2

  • optional kvrpcpb.Context context = 3

  message RawWriteResponse

  import_sstpb.proto:332

  • optional Error error = 1

  • repeated SSTMeta metas = 2

• rpc MultiIngest (MultiIngestRequest, IngestResponse)

  import_sstpb.proto:61

  Ingest Multiple files in one request

  message MultiIngestRequest

  import_sstpb.proto:166

  • optional kvrpcpb.Context context = 1

  • repeated SSTMeta ssts = 2

• rpc DuplicateDetect (DuplicateDetectRequest, stream DuplicateDetectResponse)

  import_sstpb.proto:64

  Collect duplicate data from TiKV.

  message DuplicateDetectRequest

  import_sstpb.proto:337

  • optional kvrpcpb.Context context = 1

  • bytes start_key = 2

  • bytes end_key = 3

  • bool key_only = 4

    Return only the keys found by scanning, not their values.

  • uint64 min_commit_ts = 5

    We only check the data whose timestamp is larger than `min_commit_ts`. `min_commit_ts` is exclueded.

  message DuplicateDetectResponse

  import_sstpb.proto:353

  • optional errorpb.Error region_error = 1

  • optional Error key_error = 2

  • repeated KvPair pairs = 3

    The these keys will be in asc order (but commit time is in desc order), and the content is just like following: [ {key: "key1", value: "value11", commit_ts: 1005}, {key: "key1", value: "value12", commit_ts: 1004}, {key: "key1", value: "value13", commit_ts: 1001}, {key: "key2", value: "value21", commit_ts: 1004}, {key: "key2", value: "value22", commit_ts: 1002}, ... ]

• rpc Apply (ApplyRequest, ApplyResponse)

  import_sstpb.proto:67

  Apply download & apply increment kv files to TiKV.

  message ApplyRequest

  import_sstpb.proto:414

  • optional KVMeta meta = 1

    The meta of the KV file.

  • repeated KVMeta metas = 12

  • optional RewriteRule rewrite_rule = 2

    Performs a key prefix rewrite after downloading the file. All keys in the files will be rewritten as: new_key = new_key_prefix + old_key[len(old_key_prefix)..] When used for TiDB, rewriting the prefix changes the table ID. Please note that key-rewrite is applied on the origin keys in encoded representation. You need to ensure that the keys before and after rewriting are in the same order, otherwise the RPC request will fail.

  • repeated RewriteRule rewrite_rules = 13

  • string storage_cache_id = 5

    The identity for the stroage backend. When this field presents, the storage would be cached. If there is a cached storage, TiKV would use it driectly.

  • optional backup.StorageBackend storage_backend = 3

  • optional kvrpcpb.Context context = 4

    context represents region info and it used to build raft commands.

  • optional backup.CipherInfo cipher_info = 11

    plaintext data key to decrypt kv file if configured during log backup.

  • repeated encryptionpb.MasterKey master_keys = 14

    master keys config used to decrypt data keys in restore if configured during log backup.

  message ApplyResponse

  import_sstpb.proto:449

  • optional Range range = 1

    The actual key range (after rewrite) of the downloaded file. The range is inclusive in both ends.

  • optional Error error = 2

• rpc ClearFiles (ClearRequest, ClearResponse)

  import_sstpb.proto:70

  ClearFiles clear applied file after restore succeed.

  message ClearRequest

  import_sstpb.proto:457

  • string prefix = 1

    clear files in import directory with given prefix.

  message ClearResponse

  import_sstpb.proto:462

  • optional Error error = 1

• rpc SuspendImportRPC (SuspendImportRPCRequest, SuspendImportRPCResponse)

  import_sstpb.proto:73

  Suspend ingest for data listeners don't support catching import data.

  message SuspendImportRPCRequest

  import_sstpb.proto:77

  • bool should_suspend_imports = 1

    whether to suspend new imports.

  • uint64 duration_in_secs = 2

    the duration of import service suspension when should_deny_imports is false, this won't take effect.

  • string caller = 3

    The identifier for the caller.

  message SuspendImportRPCResponse

  import_sstpb.proto:89

  • bool already_suspended = 1

    The last state before this RPC.

Used as request type in: ImportSST.Compact

Used as field type in: import_kvpb.CompactClusterRequest

• optional Range range = 1

  Compact files in the range and above the output level. Compact all files if the range is not specified. Compact all files to the bottommost level if the output level is -1.

• int32 output_level = 2

• optional kvrpcpb.Context context = 3

Used in: DownloadRequest

• Legacy = 0

  For the compatibility with old version of TiDBs

• Keyspace = 1

  For the TiDBs with newer versions that support keyspace feature.

For now it is just used for distinguishing the error of the request with the error of gRPC, add more concrete types if it is necessary later.

Used in: ApplyResponse, ClearResponse, DownloadResponse, DuplicateDetectResponse, RawWriteResponse, WriteResponse

• string message = 1

• optional errorpb.Error store_error = 2

  We meet some internal errors of the store.

Used as response type in: ImportSST.Ingest, ImportSST.MultiIngest

• optional errorpb.Error error = 1

Used in: ApplyRequest

• string name = 1

  The file name of the KV file.

• uint64 range_offset = 11

  file offset, sometimes only need to get a part of data from the merged file

• uint64 length = 2

  file length for check.

• uint64 range_length = 12

  range length of the merged file, if it exists.

• string cf = 3

  tell us which cf should apply. WRITE_CF or DEFAULT_CF e.g.

• bool is_delete = 4

  is_delete represents whether we should delete the kv in tikv. it may not be too much delete file. only rollBack operation will generate delete kv file.

• uint64 start_ts = 10

  the key ts space being smaller than start_ts can be filter.

• uint64 restore_ts = 5

  the key ts space large than restore_ts can be filter.

• bytes start_key = 6

• bytes end_key = 7

• bytes sha256 = 8

  used for checksum when download kv file.

• uint64 start_snapshot_ts = 9

  the key ts space less than start_snapshot_ts can be filter. Deprecated: this field 'start_snapshot_ts' is replaced by the field 'start_ts'.

• backup.CompressionType compression_type = 13

  the compression type for the file.

• optional encryptionpb.FileEncryptionInfo file_encryption_info = 14

  encryption information of the kv file, not set if encryption is not enabled.

Used in: DuplicateDetectResponse

• bytes key = 1

• bytes value = 2

• uint64 commit_ts = 3

Used in: RawWriteBatch, WriteBatch

• bytes key = 1

• bytes value = 2

• Pair.OP op = 3

Used in: Pair

• Put = 0

• Delete = 1

Used in: ApplyResponse, CompactRequest, DownloadResponse, SSTMeta, SwitchModeRequest

• bytes start = 1

• bytes end = 2

Used in: RawWriteRequest

• uint64 ttl = 1

• repeated Pair pairs = 2

• uint64 ts = 3

  To be compatible with the key encoding of API V2. This field should be generated from the client instead of the server, since the message will be send to all the replicas of a region. Otherwise, the underlying data generated by the server would be inconsistent which is hard to scale for other features like MVCC over RawKV.

A rewrite rule is applied on the *encoded* keys (the internal storage representation).

Used in: ApplyRequest, DownloadRequest

• bytes old_key_prefix = 1

• bytes new_key_prefix = 2

• uint64 new_timestamp = 3

  (Optional) Rewrite all keys in the range to use this timestamp.

• uint64 ignore_after_timestamp = 4

  (Optional) Skip keys with timestamps greater than this during download, useful for compacted SST backups.

• uint64 ignore_before_timestamp = 5

  (Optional) Skip write CF keys with timestamps less than this during download. Default CF keys are preserved.

Used in: DownloadRequest, DownloadResponse, IngestRequest, MultiIngestRequest, RawWriteRequest, RawWriteResponse, UploadRequest, WriteRequest, WriteResponse, raft_cmdpb.IngestSSTRequest

• bytes uuid = 1

• optional Range range = 2

• uint32 crc32 = 3

• uint64 length = 4

• string cf_name = 5

• uint64 region_id = 6

• optional metapb.RegionEpoch region_epoch = 7

• bool end_key_exclusive = 8

• uint64 total_kvs = 9

  total_kvs and total_bytes is equivalent to PD's approximate_keys and approximate_size set these values can save time from tikv upload keys and size to PD through Heartbeat.

• uint64 total_bytes = 10

• kvrpcpb.APIVersion api_version = 11

  API version implies the encode of the key and value.

• bytes cipher_iv = 12

  cipher_iv is used to encrypt/decrypt sst

Used in: GetModeResponse, SwitchModeRequest

• Normal = 0

• Import = 1

Used as request type in: ImportSST.SwitchMode

Used as field type in: import_kvpb.SwitchModeRequest

• SwitchMode mode = 1

• repeated Range ranges = 2

Used in: WriteRequest

• uint64 commit_ts = 1

• repeated Pair pairs = 2