package ballista.protobuf

Get desktop application:
View/edit binary Protocol Buffers messages

• optional LogicalPlanNode query = 1

  interactive query

• optional Task task = 2

  Execute query and store resulting shuffle partition in memory

• optional ShuffleId fetch_shuffle = 3

  Fetch a shuffle partition from an executor

• repeated KeyValuePair settings = 100

  configuration settings

Used in: LogicalExprNode

• AggregateFunction aggr_function = 1

• optional LogicalExprNode expr = 2

Used in: AggregateExprNode

• MIN = 0

• MAX = 1

• SUM = 2

• AVG = 3

• COUNT = 4

• COUNT_DISTINCT = 5

Used in: HashAggregateExecNode

• PARTIAL = 0

• FINAL = 1

• COMPLETE = 2

Used in: LogicalPlanNode

• repeated LogicalExprNode group_expr = 1

• repeated LogicalExprNode aggr_expr = 2

Used in: LogicalExprNode

• optional LogicalExprNode expr = 1

• string alias = 2

copied from GandivaType from Apache Arrow project

Used in: Field

• NONE = 0

  arrow::Type::NA

• BOOL = 1

  arrow::Type::BOOL

• UINT8 = 2

  arrow::Type::UINT8

• INT8 = 3

  arrow::Type::INT8

• UINT16 = 4

  represents arrow::Type fields in src/arrow/type.h

• INT16 = 5

• UINT32 = 6

• INT32 = 7

• UINT64 = 8

• INT64 = 9

• HALF_FLOAT = 10

• FLOAT = 11

• DOUBLE = 12

• UTF8 = 13

• BINARY = 14

• FIXED_SIZE_BINARY = 15

• DATE32 = 16

• DATE64 = 17

• TIMESTAMP = 18

• TIME32 = 19

• TIME64 = 20

• INTERVAL = 21

• DECIMAL = 22

• LIST = 23

• STRUCT = 24

• UNION = 25

• DICTIONARY = 26

• MAP = 27

Used in: LogicalExprNode

• optional LogicalExprNode l = 1

• optional LogicalExprNode r = 2

• string op = 3

Used in: LogicalPlanNode

• string name = 1

• string location = 2

• FileType file_type = 3

• bool has_header = 4

• optional Schema schema = 5

Used in: LogicalPlanNode

• string table_name = 1

• string path = 2

• bool has_header = 3

• string delimiter = 4

• string file_extension = 5

• optional ProjectionColumns projection = 6

• optional Schema schema = 7

• repeated LogicalExprNode filters = 8

• string qualifier = 2

• optional Field field = 1

Used in: LogicalPlanNode

• bool produce_one_row = 1

Used in: LogicalPlanNode

• bool verbose = 1

Used in: DfField, Schema

• string name = 1

  name of the field

• ArrowType arrow_type = 2

• bool nullable = 3

• repeated Field children = 4

  for complex data types like structs, unions

Used in: CreateExternalTableNode

• NdJson = 0

• Parquet = 1

• CSV = 2

Used in: PhysicalPlanNode

• uint32 limit = 1

Used in: PhysicalPlanNode

• repeated LogicalExprNode group_expr = 1

• repeated LogicalExprNode aggr_expr = 2

• AggregateMode mode = 3

Used in: RepartitionNode

• repeated LogicalExprNode hash_expr = 2

• uint64 batch_size = 1

Used in: LogicalExprNode

• optional LogicalExprNode expr = 1

Used in: LogicalExprNode

• optional LogicalExprNode expr = 1

Used in: LogicalPlanNode

• optional LogicalPlanNode left = 1

• optional LogicalPlanNode right = 2

• JoinType join_type = 3

• repeated string left_join_column = 4

• repeated string right_join_column = 5

Used in: JoinNode

• INNER = 0

• LEFT = 1

• RIGHT = 2

Used in: Action

• string key = 1

• string value = 2

Used in: LogicalPlanNode

• uint32 limit = 1

Used in: PhysicalPlanNode

• uint32 limit = 1

logical expressions

Used in: AggregateExprNode, AggregateNode, AliasNode, BinaryExprNode, CsvTableScanNode, HashAggregateExecNode, HashRepartition, IsNotNull, IsNull, Not, ParquetTableScanNode, ProjectionExecNode, ProjectionNode, SelectionExecNode, SelectionNode, SortNode

• oneof ExprType

  • string column_name = 10

    column references

  • AliasNode alias = 14

    alias

  • string literal_string = 20

    literals

  • int32 literal_int8 = 21

  • int32 literal_int16 = 22

  • int32 literal_int32 = 23

  • int64 literal_int64 = 24

  • uint32 literal_uint8 = 25

  • uint32 literal_uint16 = 26

  • uint32 literal_uint32 = 27

  • uint64 literal_uint64 = 28

  • float literal_f32 = 29

  • double literal_f64 = 30

  • BinaryExprNode binary_expr = 40

    binary expressions

  • AggregateExprNode aggregate_expr = 50

    aggregate expressions

  • IsNull is_null_expr = 60

    null checks

  • IsNotNull is_not_null_expr = 61

  • Not not_expr = 62

LogicalPlan is a nested type

Used in: Action, JoinNode

• optional LogicalPlanNode input = 1

• optional CsvTableScanNode csv_scan = 10

• optional ParquetTableScanNode parquet_scan = 11

• optional ProjectionNode projection = 20

• optional SelectionNode selection = 21

• optional LimitNode limit = 22

• optional AggregateNode aggregate = 23

• optional JoinNode join = 24

• optional SortNode sort = 25

• optional RepartitionNode repartition = 26

• optional EmptyRelationNode empty_relation = 27

• optional CreateExternalTableNode create_external_table = 28

• optional ExplainNode explain = 29

Used in: LogicalExprNode

• optional LogicalExprNode expr = 1

Used in: LogicalPlanNode

• string table_name = 1

• string path = 2

• optional ProjectionColumns projection = 3

• optional Schema schema = 4

• repeated LogicalExprNode filters = 5

PhysicalPlanNode is a nested type

Used in: Task

• optional PhysicalPlanNode input = 1

• optional ScanExecNode scan = 10

• optional ProjectionExecNode projection = 20

• optional SelectionExecNode selection = 21

• optional GlobalLimitExecNode global_limit = 22

• optional LocalLimitExecNode local_limit = 23

• optional HashAggregateExecNode hash_aggregate = 30

• optional ShuffleReaderExecNode shuffle_reader = 40

Used in: CsvTableScanNode, ParquetTableScanNode

• repeated string columns = 1

Used in: PhysicalPlanNode

• repeated LogicalExprNode expr = 1

Used in: LogicalPlanNode

• repeated LogicalExprNode expr = 1

Used in: LogicalPlanNode

• oneof partition_method

  • uint64 round_robin = 1

  • HashRepartition hash = 2

Used in: PhysicalPlanNode

• string path = 1

• repeated uint32 projection = 2

• optional Schema schema = 3

• string file_format = 4

  parquet or csv

• bool has_header = 5

  csv specific

• uint32 batch_size = 6

• repeated string filename = 8

  partition filenames

Used in: CreateExternalTableNode, CsvTableScanNode, ParquetTableScanNode, ScanExecNode, ShuffleReaderExecNode

• repeated Field columns = 1

Used in: PhysicalPlanNode

• optional LogicalExprNode expr = 2

Used in: LogicalPlanNode

• optional LogicalExprNode expr = 2

Mapping from shuffle id to executor id

Used in: Action, ShuffleReaderExecNode

• string job_uuid = 1

• uint32 stage_id = 2

• uint32 partition_id = 4

Mapping from shuffle id to executor id

Used in: Task

• string job_uuid = 1

• uint32 stage_id = 2

• uint32 partition_id = 4

• string executor_id = 5

• string executor_host = 6

• uint32 executor_port = 7

Used in: PhysicalPlanNode

• repeated ShuffleId shuffle_id = 1

• optional Schema schema = 2

Used in: LogicalPlanNode

• repeated LogicalExprNode expr = 1

Used in: Action

• string job_uuid = 1

• uint32 stage_id = 2

• uint32 task_id = 3

• uint32 partition_id = 4

• optional PhysicalPlanNode plan = 5

• repeated ShuffleLocation shuffle_loc = 6

  The task could need to read shuffle output from another task