package common

Get desktop application:
View/edit binary Protocol Buffers messages

Some requests from client needs server responds ack asynchronously, like: - Heartbeat - Checkpoint - Metrics

Used as request type in: coordinator.CoordinatorApi.ReceiveAck, taskmanager.TaskManagerApi.ReceiveAck

• oneof request_id

  the id which sent by the request needs to ack. it may points to multiple semantics: - for heartbeat, it represents heartbeat id - for checkpoint, it represents checkpoint id - for metrics, it represents metric id

  • uint64 heartbeat_id = 1

• optional google.protobuf.Timestamp timestamp = 2

  The timestamp when the ack response sent

• Ack.AckType ack_type = 3

  the ack type

• NodeType node_type = 4

  the node type of ack server

• optional SubDataflowId execution_id = 6

  the execution id

Ack type, like heartbeat, checkpoint

Used in: Ack

• HEARTBEAT = 0

* Constant operator

• bytes value = 1

  value of constant, format: [<flag byte>, <data bytes>]

• uint32 operator_id = 2

  operator id

Enum of Data Type. each one corresponds to a primitive type in JavaScript

Used in: Entry, KafkaDesc

• DATA_TYPE_ENUM_UNSPECIFIED = 0

  undefined

• DATA_TYPE_ENUM_BIGINT = 1

  bigint

• DATA_TYPE_ENUM_NUMBER = 2

  number

• DATA_TYPE_ENUM_NULL = 3

  null

• DATA_TYPE_ENUM_STRING = 4

  string

• DATA_TYPE_ENUM_BOOLEAN = 5

  boolean

• DATA_TYPE_ENUM_OBJECT = 6

  object

• DATA_TYPE_ENUM_ARRAY = 7

  array

An union linked-list structure of the description of Dataflow. Dataflow can be shared between API, Coordinator and TaskManager. However, they may check the Dataflow by distinct validators. Each part's concern is different and they must be sure it's a legal Dataflow to them.

Used as request type in: coordinator.CoordinatorApi.CreateDataflow

Used as field type in: apiserver.CreateDataflowOptions, DataflowStates, taskmanager.CreateSubDataflowRequest

• optional ResourceId job_id = 1

  job id, for now it is which table the stream graph output will sink

• repeated DataflowMeta meta = 2

  graph structure

• map<uint32, OperatorInfo> nodes = 3

  details of nodes

• optional SubDataflowId execution_id = 4

  execution id, optional for API, mandatory for TaskManager

* StreamGraph metadata, it stores the structural information of a stream graph

Used in: Dataflow

• uint32 center = 1

  center node id

• repeated uint32 neighbors = 2

  center's neighbors

Used as response type in: coordinator.CoordinatorApi.GetDataflow

• optional Dataflow graph = 1

  the structure of the dataflow

• repeated SubdataflowInfo subdataflow_infos = 2

  task infos of all subdataflow

• DataflowStatus status = 3

  dataflow status

* Stream Graph Status. It shows which status a stream job is now.

Used in: DataflowStates, taskmanager.CreateSubDataflowResponse

• INITIALIZED = 0

• RUNNING = 1

• CLOSING = 2

• CLOSED = 3

Used in: Sink

• DELIVERY_GUARENTEE_NONE = 0

• DELIVERY_AT_LEAST_ONCE = 1

• DELIVERY_EXACTLY_ONCE = 2

Entry that represents a structure of Typed Value For improving performance, when being compiled by tonic, the type of value after compilation is recommended to set as bytes::Bytes so that Entry's value can be a zero-copy bytes slice.

Used in: KeyedDataEvent

• DataTypeEnum data_type = 1

• bytes value = 2

  entry value

Some common rpc error code

• ERROR_CODE_UNSPECIFIED = 0

• ERROR_CODE_RESOURCE_NOT_FOUND = 1

• ERROR_CODE_RPC_UNAUTHORIZED = 2

• ERROR_CODE_RPC_INVALID_ARGUMENT = 3

• ERROR_CODE_RPC_PERMISSION_DENIED = 4

• ERROR_CODE_INTERNAL_ERROR = 5

• ERROR_CODE_DATAFLOW_OPERATOR_INFO_MISSING = 6

• ERROR_CODE_CYCLIC_DATAFLOW = 7

• ERROR_CODE_DATAFLOW_CONFIGURATION_MISSING = 8

Basic information of executor

Used in: SubdataflowInfo

• uint32 executor_id = 1

• ExecutorStatus status = 2

status of executor

Used in: ExecutorInfo

• EXECUTOR_STATUS_INITIALIZED = 0

• EXECUTOR_STATUS_RUNNING = 1

• EXECUTOR_STATUS_TERMINATING = 2

• EXECUTOR_STATUS_TERMINATED = 3

Used in: OperatorInfo

• oneof value

  • Func func = 1

Used in: OperatorInfo

• oneof value

  • Func func = 1

Used in: Filter, FlatMap, Join.StreamJoin, KeyBy, Mapper, RedisDesc, Reducer

• string function = 1

structure of heartbeat

Used as request type in: coordinator.CoordinatorApi.ReceiveHeartbeat, taskmanager.TaskManagerApi.ReceiveHeartbeat

• uint64 heartbeat_id = 1

  heartbeat id which increases monotonically

• optional google.protobuf.Timestamp timestamp = 2

  The timestamp when the heartbeat sent

• NodeType node_type = 3

  The client node type

• optional SubDataflowId subdataflow_id = 4

  Execution Id of sub-dataflow

• uint32 task_id = 5

  id of task executor

The common structure of remote host address in Lightflus

Used in: OperatorInfo

• string host = 1

• uint32 port = 2

• oneof value

  • Join.StreamJoin stream_join = 1

Used in: Join

• uint32 operator_id = 1

• optional Func func = 2

Used in: Sink, Source

• repeated string brokers = 1

• string topic = 2

• optional KafkaDesc.KafkaOptions opts = 3

• DataTypeEnum data_type = 4

Used in: KafkaDesc

• optional string group = 1

• optional uint32 partition = 2

Used in: OperatorInfo

• oneof value

  • Func func = 1

Event that keyed transferred between operators KeyedDataEvent can be traced in a distributed system with event id

Used as request type in: taskmanager.TaskManagerApi.SendEventToOperator

Used as field type in: KeyedEventSet

• optional ResourceId job_id = 1

• optional Entry key = 2

  key of data event

• uint32 to_operator_id = 3

  operator_id this event will be sent

• repeated Entry data = 5

  mandatory

• int64 event_time = 6

  event time

• uint32 from_operator_id = 7

  operator_id this event where be sent

• optional KeyedDataEvent.Window window = 8

  the window of this event

• int64 event_id = 9

  event id, generated by source

Used in: KeyedDataEvent

• int64 start_time = 1

• int64 end_time = 2

Used as request type in: taskmanager.TaskManagerApi.BatchSendEventsToOperator

• repeated KeyedDataEvent events = 1

• optional ResourceId job_id = 2

• uint32 to_operator_id = 3

  operator_id this event will be sent

• uint32 from_operator_id = 4

  operator_id this event where be sent

Used in: OperatorInfo

• oneof value

  • Func func = 1

Used in: Sink

• optional MysqlDesc.ConnectionOpts connection_opts = 1

• optional MysqlDesc.Statement statement = 2

Used in: MysqlDesc

• string host = 1

• string username = 2

• string password = 3

• string database = 4

Used in: MysqlDesc

• string statement = 1

• repeated Statement.Extractor extractors = 2

Used in: Statement

• uint32 index = 1

• string extractor = 2

The type of node

Used in: Ack, Heartbeat

• JOB_MANAGER = 0

  Job manager

• TASK_WORKER = 1

  Task worker

* OperatorInfo, stores detail information of an operator

Used in: Dataflow

• uint32 operator_id = 1

• optional HostAddr host_addr = 2

  host addr configs

• repeated uint32 upstreams = 3

  upstreams operator_id

• oneof details

  optional for different operator type

  • Source source = 5

    for source

  • Sink sink = 6

    for sink

  • Mapper mapper = 7

  • Filter filter = 8

  • KeyBy key_by = 9

  • Reducer reducer = 10

  • FlatMap flat_map = 11

  • Window window = 12

    Join join = 11;

• OPERATOR_RUNNING = 0

• OPERATOR_TERMINATED = 1

Used in: Sink

• optional RedisDesc.ConnectionOpts connection_opts = 1

• optional Func key_extractor = 2

• optional Func value_extractor = 3

Used in: RedisDesc

• string host = 1

• string username = 2

• string password = 3

• int64 database = 4

• bool tls = 5

Used in: OperatorInfo

• oneof value

  • Func func = 1

* JobId, represents a stream job.

Used as request type in: coordinator.CoordinatorApi.TerminateDataflow, taskmanager.TaskManagerApi.GetSubDataflow, taskmanager.TaskManagerApi.StopDataflow

Used as field type in: apiserver.CreateResourceResponse, apiserver.DeleteResourceRequest, apiserver.GetResourceRequest, apiserver.Resource, Dataflow, KeyedDataEvent, KeyedEventSet, SubDataflowId, coordinator.GetDataflowRequest, taskmanager.CreateSubDataflowRequest

• string resource_id = 1

• string namespace_id = 2

common Rpc Response

Used as response type in: coordinator.CoordinatorApi.CreateDataflow, coordinator.CoordinatorApi.ReceiveAck, coordinator.CoordinatorApi.ReceiveHeartbeat, coordinator.CoordinatorApi.TerminateDataflow, taskmanager.TaskManagerApi.ReceiveAck, taskmanager.TaskManagerApi.ReceiveHeartbeat

Used as field type in: taskmanager.StopDataflowResponse

• string status = 1

• string err_msg = 2

Used in: OperatorInfo

• oneof desc

  • KafkaDesc kafka = 1

  • MysqlDesc mysql = 2

  • RedisDesc redis = 3

• DeliveryGuarentee delivery_guarentee = 4

Used in: OperatorInfo

• oneof desc

  • KafkaDesc kafka = 3

Id of sub-dataflow execution

Used in: Ack, Dataflow, Heartbeat, SubdataflowInfo

• optional ResourceId job_id = 1

  Job Id

• uint32 sub_id = 2

  Sub Dataflow id

Used as response type in: taskmanager.TaskManagerApi.GetSubDataflow

• optional SubdataflowInfo subdataflow_infos = 1

Basic information of task

Used in: DataflowStates, SubDataflowStates

• optional SubDataflowId execution_id = 1

  execution id of task

• map<uint32, ExecutorInfo> executors_info = 2

  information of executors

The common structure of Timestamp in Lightflus

Used in: Trigger.Watermark, Window.FixedWindow, Window.SessionWindow, Window.SlidingWindow

• uint64 millis = 1

• uint64 seconds = 2

• uint32 minutes = 3

• uint32 hours = 4

Used in: Window

• oneof value

  • Trigger.Watermark watermark = 1

Used in: Trigger

• optional Time trigger_time = 1

Used in: OperatorInfo

• oneof value

  • Window.FixedWindow fixed = 1

  • Window.SlidingWindow slide = 2

  • Window.SessionWindow session = 3

• optional Trigger trigger = 4

Used in: Window

• optional Time size = 1

  Only for sliding & fixed window

Used in: Window

• optional Time timeout = 1

  Only for Session Window

Used in: Window

• optional Time size = 1

• optional Time period = 2