package source.v1
Get desktop application:
View/edit binary Protocol Buffers messages
service Source
source.proto:11rpc AckFn (stream , stream )
source.proto:25AckFn acknowledges a stream of datum offsets. When AckFn is called, it implicitly indicates that the datum stream has been processed by the source vertex. The caller (numa) expects the AckFn to be successful, and it does not expect any errors. If there are some irrecoverable errors when the callee (UDSource) is processing the AckFn request, then it is best to crash because there are no other retry mechanisms possible. Clients sends n requests and expects n responses.
message
source.proto:120AckRequest is the request for acknowledging datum. It takes a list of offsets to be acknowledged.
optional request = 1
Required field holding the request. The list will be ordered and will have the same order as the original Read response.
optional handshake = 2
message
source.proto:140AckResponse is the response for acknowledging datum. It contains one empty field confirming the batch of offsets that have been successfully acknowledged. The contract between client and server is that the server will only return the AckResponse if the ack request is successful. If the server hangs during the ack request, the client can decide to timeout and error out the data forwarder. The reason why we define such contract is that we always expect the server to be able to process the ack request. Client is expected to send the AckRequest to the server with offsets that are strictly corresponding to the previously read batch. If the client sends the AckRequest with offsets that are not, it is considered as a client error and the server will not return the AckResponse.
optional result = 1
Required field holding the result.
optional handshake = 2
Handshake message between client and server to indicate the start of transmission.
rpc IsReady (, )
source.proto:38IsReady is the heartbeat endpoint for user defined source gRPC.
message
source.proto:172ReadyResponse is the health check result for user defined source.
bool ready = 1
Required field holding the health check result.
rpc NackFn (, )
source.proto:29NackFn negatively acknowledges a batch of offsets. Invoked during a critical error in the mono vertex or pipeline. Unlike AckFn its not a streaming rpc because this is only invoked when there is a critical error (error path).
message
source.proto:151optional request = 1
Required field holding the request. The list will be ordered and will have the same order as the original Read response.
message
source.proto:160optional result = 1
Required field holding the result.
rpc PartitionsFn (, )
source.proto:35PartitionsFn returns the list of partitions for the user defined source.
message
source.proto:193PartitionsResponse is the response for the partitions request.
optional result = 1
Required field holding the result.
rpc PendingFn (, )
source.proto:32PendingFn returns the number of pending records at the user defined source.
message
source.proto:180PendingResponse is the response for the pending request.
optional result = 1
Required field holding the result.
rpc ReadFn (stream , stream )
source.proto:17Read returns a stream of datum responses. The size of the returned responses is less than or equal to the num_records specified in each ReadRequest. If the request timeout is reached on the server side, the returned responses will contain all the datum that have been read (which could be an empty list). The server will continue to read and respond to subsequent ReadRequests until the client closes the stream. Once it has sent all the datum, the server will send a ReadResponse with the end of transmission flag set to true.
message
source.proto:52ReadRequest is the request for reading datum stream from user defined source.
optional request = 1
Required field indicating the request.
optional handshake = 2
message
source.proto:69ReadResponse is the response for reading datum stream from user defined source.
optional result = 1
Required field holding the result.
optional status = 2
Status of the response. Holds the end of transmission flag and the status code.
optional handshake = 3
Handshake message between client and server to indicate the start of transmission.
message
source.proto:121Used in:
repeated offsets = 1
Required field holding the offset to be acked
message
source.proto:141Used in:
optional success = 1
Required field indicating the ack request is successful.
message
source.proto:44Handshake message between client and server to indicate the start of transmission.
Used in: , , ,
bool sot = 1
Required field indicating the start of transmission.
message
source.proto:152Used in:
repeated offsets = 1
Required field holding the offset to be nacked
message
source.proto:161Used in:
optional success = 1
Required field indicating the nack request is successful.
message
source.proto:207Offset is the offset of the datum.
Used in: , ,
bytes offset = 1
offset is the offset of the datum. This field is required. We define Offset as a byte array because different input data sources can have different representations for Offset. The only way to generalize it is to define it as a byte array, Such that we can let the UDSource to de-serialize the offset using its own interpretation logics.
int32 partition_id = 2
Optional partition_id indicates which partition of the source the datum belongs to. It is useful for sources that have multiple partitions. e.g. Kafka. If the partition_id is not specified, it is assumed that the source has a single partition.
message
source.proto:194Used in:
repeated int32 partitions = 1
Required field holding the list of active partitions.
optional int32 total_partitions = 2
Total number of partitions in the source.
message
source.proto:181Used in:
int64 count = 1
Required field holding the number of pending records at the user defined source. A negative count indicates that the pending information is not available.
message
source.proto:53Used in:
uint64 num_records = 1
Required field indicating the number of records to read.
uint32 timeout_in_ms = 2
Required field indicating the request timeout in milliseconds. uint32 can represent 2^32 milliseconds, which is about 49 days. We don't use uint64 because time.Duration takes int64 as nano seconds. Using uint64 for milli will cause overflow.
message
source.proto:70Used in:
bytes payload = 1
Required field holding the payload of the datum.
optional offset = 2
Required field indicating the offset information of the datum.
optional event_time = 3
Required field representing the time associated with each datum. It is used for watermarking.
repeated string keys = 4
Optional list of keys associated with the datum. Key is the "key" attribute in (key,value) as in the map-reduce paradigm. We add this optional field to support the use case where the user defined source can provide keys for the datum. e.g. Kafka and Redis Stream message usually include information about the keys.
map<string, string> headers = 5
Optional list of headers associated with the datum. Headers are the metadata associated with the datum. e.g. Kafka and Redis Stream message usually include information about the headers.
optional metadata = 6
Metadata is the metadata of the message
message
source.proto:89Used in:
bool eot = 1
End of transmission flag.
Status.Code code = 2
optional Status.Error error = 3
optional string msg = 4
Code to indicate the status of the response.
Used in:
SUCCESS = 0
FAILURE = 1
Error to indicate the error type. If the code is FAILURE, then the error field will be populated.
Used in:
UNACKED = 0
OTHER = 1