package rpc

Get desktop application:
View/edit binary Protocol Buffers messages

PeerDataNode service defines the service for data fetching from peer data node A data fetching process will proceed with the following sequence: 1. StartSession to start data fetching session. (as long as any ongoing session is alive, the peer data node will not purge its raw data during archiving/backfill/snapshot process until all sessions exprired or closed) 2. KeepAlive in sub routine to keep session alive througout the data copying process. close the stream on client side will close the session. 3. FetchTableShardMetaData to fetch table shard metadata with a range [startBatchID, endBatchID] of batche needed the response will return the list of metadata for batches, including available column ids at the peer node 4. FetchVectorPartyRawData to fetch vector party from peer data node

• rpc Health (HealthCheckRequest, HealthCheckResponse)

  peer_streaming.proto:148

  message HealthCheckRequest

  peer_streaming.proto:124

  • string service = 1

  message HealthCheckResponse

  peer_streaming.proto:128

  • HealthCheckResponse.ServingStatus status = 1

• rpc StartSession (StartSessionRequest, Session)

  peer_streaming.proto:150

  StartSession starts a session for data streaming

  message StartSessionRequest

  peer_streaming.proto:101

  • string table = 1

  • uint32 shard = 2

  • int64 ttl = 3

  • string nodeID = 4

    caller node id

• rpc KeepAlive (stream Session, stream KeepAliveResponse)

  peer_streaming.proto:153

  KeepAlive sends keep alive message to the peer data service to renew session and get new ttl for the session

  message KeepAliveResponse

  peer_streaming.proto:113

  • int64 ID = 1

  • int64 ttl = 2

• rpc FetchTableShardMetaData (TableShardMetaDataRequest, TableShardMetaData)

  peer_streaming.proto:155

  FetchTableShardMetaData fetches metadata for given table shard

  message TableShardMetaDataRequest

  peer_streaming.proto:72

  • string table = 1

    table name

  • int32 incarnation = 2

    table incarnation

  • uint32 shard = 3

    shard id

  • int32 startBatchID = 4

    first batch id to fetch, fact only

  • int32 endBatchID = 5

    last batch id to fetch, fact only

  • int64 sessionID = 6

    established session id

  • string nodeID = 7

    caller node id

  message TableShardMetaData

  peer_streaming.proto:60

  • string table = 1

    table name

  • int32 incarnation = 2

    table incarnation

  • uint32 shard = 3

    shard id

  • optional KafkaOffset kafkaOffset = 4

  • oneof meta

    • FactTableShardMetaData factMeta = 5

    • DimensionTableShardMetaData dimensionMeta = 6

  • repeated BatchMetaData batches = 7

    batches

• rpc FetchVectorPartyRawData (VectorPartyRawDataRequest, stream VectorPartyRawData)

  peer_streaming.proto:157

  FetchVectorPartyRawData fetches raw data for specified vector party

  message VectorPartyRawDataRequest

  peer_streaming.proto:83

  • string table = 1

  • int32 incarnation = 2

  • uint32 shard = 3

  • int32 batchID = 4

  • oneof version

    • ArchiveVersion archiveVersion = 5

    • SnapshotVersion snapshotVersion = 6

  • uint32 columnID = 7

  • int64 sessionID = 8

    established session id

  • string nodeID = 9

    caller node id

• rpc BenchmarkFileTransfer (BenchmarkRequest, stream VectorPartyRawData)

  peer_streaming.proto:159

  benchmark function to test performance using different config for file transfer

  message BenchmarkRequest

  peer_streaming.proto:118

  • string file = 1

  • int32 chunkSize = 2

  • int32 bufferSize = 3

Used in: BatchMetaData, VectorPartyRawDataRequest

• uint32 archiveVersion = 1

  archive version, fact table only

• uint32 backfillSeq = 2

  backfillSeq, fact table only

Used in: FactTableShardMetaData

• int64 redoFileID = 1

  checkpoint redolog file id, dimension table only

• uint32 redoFileOffset = 2

  checkpoint redolog file offset, dimension table only

Used in: TableShardMetaData

• int32 batchID = 1

  batch id

• uint32 size = 2

  size of batch

• optional ArchiveVersion archiveVersion = 3

  archive version, archive only

• repeated VectorPartyMetaData vps = 4

Used in: TableShardMetaData

• optional SnapshotVersion snapshotVersion = 1

• int32 lastBatchID = 2

  last batch id for snapshot, snapshot only

• int32 lastBatchSize = 3

  last batch size for snapshot

Used in: TableShardMetaData

• uint32 highWatermark = 1

  high watermark, fact only

• optional BackfillCheckpoint backfillCheckpoint = 2

Used in: HealthCheckResponse

• UNKNOWN = 0

• SERVING = 1

• NOT_SERVING = 2

Used in: TableShardMetaData

• int64 checkPointOffset = 1

  archiving/backfill/snapshot checkpoint kafka offset

• int64 commitOffset = 2

  ingestion committed kafka offset

Used as request type in: PeerDataNode.KeepAlive

Used as response type in: PeerDataNode.StartSession

• int64 ID = 1

  established session id

• string nodeID = 2

  caller node id

Used in: DimensionTableShardMetaData, VectorPartyRawDataRequest

• int64 redoFileID = 1

  checkpoint redolog file id, dimension table only

• uint32 redoFileOffset = 2

  checkpoint redolog file offset, dimension table only

Used in: BatchMetaData

• uint32 columnID = 1

  column id

Used as response type in: PeerDataNode.BenchmarkFileTransfer, PeerDataNode.FetchVectorPartyRawData

• bytes chunk = 1