package plan.protobuf

Get desktop application:
View/edit binary Protocol Buffers messages

Used in: AggExecNode

• HASH_AGG = 0

• SORT_AGG = 1

Used in: PhysicalPlanNode

• optional PhysicalPlanNode input = 1

• AggExecMode exec_mode = 2

• repeated PhysicalExprNode grouping_expr = 3

• repeated PhysicalExprNode agg_expr = 4

• repeated AggMode mode = 5

• repeated string grouping_expr_name = 6

• repeated string agg_expr_name = 7

• uint64 initial_input_buffer_offset = 8

• bool supports_partial_skipping = 9

Used in: PhysicalAggExprNode, WindowExprNode

• MIN = 0

• MAX = 1

• SUM = 2

• AVG = 3

• COUNT = 4

• COLLECT_LIST = 5

• COLLECT_SET = 6

• FIRST = 7

• FIRST_IGNORES_NULL = 8

• BLOOM_FILTER = 9

• BRICKHOUSE_COLLECT = 1000

• BRICKHOUSE_COMBINE_UNIQUE = 1001

• UDAF = 1002

Used in: AggExecNode

• PARTIAL = 0

• PARTIAL_MERGE = 1

• FINAL = 2

Used in: PhysicalAggExprNode

• bytes serialized = 1

• optional Schema input_schema = 2

Broke out into multiple message types so that type metadata did not need to be in separate message All types that are of the empty message types contain no additional metadata about the type

Used in: BoundReference, Dictionary, Field, PhysicalAggExprNode, PhysicalCastNode, PhysicalNamedStructExprNode, PhysicalScalarFunctionNode, PhysicalSparkScalarSubqueryWrapperExprNode, PhysicalSparkUDFWrapperExprNode, PhysicalTryCastNode, ProjectionExecNode, WindowExprNode

• oneof arrow_type_enum

  • EmptyMessage NONE = 1

    arrow::Type::NA

  • EmptyMessage BOOL = 2

    arrow::Type::BOOL

  • EmptyMessage UINT8 = 3

    arrow::Type::UINT8

  • EmptyMessage INT8 = 4

    arrow::Type::INT8

  • EmptyMessage UINT16 = 5

    represents arrow::Type fields in src/arrow/type.h

  • EmptyMessage INT16 = 6

  • EmptyMessage UINT32 = 7

  • EmptyMessage INT32 = 8

  • EmptyMessage UINT64 = 9

  • EmptyMessage INT64 = 10

  • EmptyMessage FLOAT16 = 11

  • EmptyMessage FLOAT32 = 12

  • EmptyMessage FLOAT64 = 13

  • EmptyMessage UTF8 = 14

  • EmptyMessage LARGE_UTF8 = 32

  • EmptyMessage BINARY = 15

  • int32 FIXED_SIZE_BINARY = 16

  • EmptyMessage LARGE_BINARY = 31

  • EmptyMessage DATE32 = 17

  • EmptyMessage DATE64 = 18

  • TimeUnit DURATION = 19

  • Timestamp TIMESTAMP = 20

  • TimeUnit TIME32 = 21

  • TimeUnit TIME64 = 22

  • IntervalUnit INTERVAL = 23

  • Decimal DECIMAL = 24

  • List LIST = 25

  • List LARGE_LIST = 26

  • FixedSizeList FIXED_SIZE_LIST = 27

  • Struct STRUCT = 28

  • Union UNION = 29

  • Dictionary DICTIONARY = 30

  • Map MAP = 33

Used in: PhysicalExprNode

• string uuid = 1

• optional PhysicalExprNode bloom_filter_expr = 2

• optional PhysicalExprNode value_expr = 3

Used in: PhysicalExprNode

• uint64 index = 1

• optional ArrowType data_type = 2

• bool nullable = 3

Used in: PhysicalPlanNode

• optional PhysicalPlanNode input = 1

• repeated PhysicalExprNode keys = 2

Used in: PhysicalPlanNode

• optional Schema schema = 1

• optional PhysicalPlanNode left = 2

• optional PhysicalPlanNode right = 3

• repeated JoinOn on = 4

• JoinType join_type = 5

• JoinSide broadcast_side = 6

• string cached_build_hash_map_id = 7

Used in: PhysicalPlanNode

• optional PhysicalPlanNode input = 1

• uint64 batch_size = 2

Used in: JoinFilter

• uint32 index = 1

• JoinSide side = 2

Used in: Statistics

• optional ScalarValue min_value = 1

• optional ScalarValue max_value = 2

• uint32 null_count = 3

• uint32 distinct_count = 4

• Day = 0

• DateMillisecond = 1

Used in: PhysicalPlanNode

• optional PhysicalPlanNode input = 1

• string debug_id = 2

Used in: ArrowType

• uint64 whole = 1

• int64 fractional = 2

Used in: ArrowType

• optional ArrowType key = 1

• optional ArrowType value = 2

Useful for representing an empty enum variant in rust E.G. enum example{One, Two(i32)} maps to message example{ oneof{ EmptyMessage One = 1; i32 Two = 2; } }

Used in: ArrowType

(message has no fields)

Used in: PhysicalPlanNode

• optional Schema schema = 1

• uint32 num_partitions = 2

Used in: PhysicalPlanNode

• optional PhysicalPlanNode input = 1

• optional Schema schema = 2

• repeated ExpandProjection projections = 3

Used in: ExpandExecNode

• repeated PhysicalExprNode expr = 1

Used in: PhysicalPlanNode

• uint32 num_partitions = 1

• optional Schema schema = 2

• string export_iter_provider_resource_id = 3

Used in: SortExecNode

• uint64 limit = 1

  wrap into a message to make it optional

Used in: FixedSizeList, GenerateExecNode, List, Map, Schema, Struct, Union, WindowExprNode

• string name = 1

  name of the field

• optional ArrowType arrow_type = 2

• bool nullable = 3

• repeated Field children = 4

  for complex data types like structs, unions

Used in: FileScanExecConf

• repeated PartitionedFile files = 1

Used in: PartitionedFile

• int64 start = 1

• int64 end = 2

Used in: OrcScanExecNode, ParquetScanExecNode

• int64 num_partitions = 1

• int64 partition_index = 2

• optional FileGroup file_group = 3

• optional Schema schema = 4

• repeated uint32 projection = 6

• optional ScanLimit limit = 7

• optional Statistics statistics = 8

• optional Schema partition_schema = 9

Used in: PhysicalPlanNode

• optional PhysicalPlanNode input = 1

• repeated PhysicalExprNode expr = 2

• int32 length = 1

Used in: ArrowType

• optional Field field_type = 1

• int32 list_size = 2

Used in: PhysicalPlanNode

• optional PhysicalPlanNode input = 1

• optional Generator generator = 2

• repeated string required_child_output = 3

• repeated Field generator_output = 4

• bool outer = 5

Used in: Generator

• Explode = 0

• PosExplode = 1

• JsonTuple = 2

• Udtf = 10000

Used in: Generator

• bytes serialized = 1

• optional Schema return_schema = 2

Used in: GenerateExecNode

• GenerateFunction func = 1

• optional GenerateUdtf udtf = 2

• repeated PhysicalExprNode child = 3

Used in: PhysicalPlanNode

• optional Schema schema = 1

• optional PhysicalPlanNode left = 2

• optional PhysicalPlanNode right = 3

• repeated JoinOn on = 4

• JoinType join_type = 5

• JoinSide build_side = 6

Used in: ArrowType

• YearMonth = 0

• DayTime = 1

• MonthDayNano = 2

Used in: PhysicalPlanNode

• uint32 num_partitions = 1

• optional Schema schema = 2

• string ipc_provider_resource_id = 3

Used in: PhysicalPlanNode

• optional PhysicalPlanNode input = 1

• string ipc_consumer_resource_id = 2

• optional PhysicalExprNode expression = 1

• repeated ColumnIndex column_indices = 2

• optional Schema schema = 3

Used in: BroadcastJoinExecNode, HashJoinExecNode, SortMergeJoinExecNode

• optional PhysicalExprNode left = 1

• optional PhysicalExprNode right = 2

Used in: BroadcastJoinExecNode, ColumnIndex, HashJoinExecNode

• LEFT_SIDE = 0

• RIGHT_SIDE = 1

Used in: BroadcastJoinExecNode, HashJoinExecNode, SortMergeJoinExecNode

• INNER = 0

• LEFT = 1

• RIGHT = 2

• FULL = 3

• SEMI = 4

• ANTI = 5

• EXISTENCE = 6

Used in: PhysicalPlanNode

• optional PhysicalPlanNode input = 1

• uint64 limit = 2

Used in: ArrowType

• optional Field field_type = 1

Used in: ArrowType

• optional Field key_type = 1

• optional Field value_type = 2

Used in: PhysicalPlanNode

• optional FileScanExecConf base_conf = 1

• repeated PhysicalExprNode pruning_predicates = 2

• string fsResourceId = 3

Used in: ParquetSinkExecNode

• string key = 1

• string value = 2

Used in: PhysicalPlanNode

• optional FileScanExecConf base_conf = 1

• repeated PhysicalExprNode pruning_predicates = 2

• string fsResourceId = 3

Used in: PhysicalPlanNode

• optional PhysicalPlanNode input = 1

• string fs_resource_id = 2

• int32 num_dyn_parts = 3

• repeated ParquetProp prop = 4

Unique identifier for a materialized partition of data

Used in: TaskDefinition

• string job_id = 1

• uint32 stage_id = 2

• uint32 partition_id = 4

• COLLECT_LEFT = 0

• PARTITIONED = 1

Used in: FileGroup

• string path = 1

• uint64 size = 2

• uint64 last_modified_ns = 3

• repeated ScalarValue partition_values = 4

• optional FileRange range = 5

Used in: PhysicalExprNode

• AggFunction agg_function = 1

• optional AggUdaf udaf = 2

• repeated PhysicalExprNode children = 3

• optional ArrowType return_type = 4

• optional PhysicalExprNode expr = 1

• string alias = 2

Used in: PhysicalExprNode

• optional PhysicalExprNode l = 1

• optional PhysicalExprNode r = 2

• string op = 3

Used in: PhysicalExprNode

• optional PhysicalExprNode expr = 1

• repeated PhysicalWhenThen when_then_expr = 2

• optional PhysicalExprNode else_expr = 3

Used in: PhysicalExprNode

• optional PhysicalExprNode expr = 1

• optional ArrowType arrow_type = 2

Used in: PhysicalExprNode

• string name = 1

• uint32 index = 2

physical expressions

Used in: AggExecNode, BloomFilterMightContainExprNode, BroadcastJoinBuildHashMapExecNode, ExpandProjection, FilterExecNode, Generator, JoinFilter, JoinOn, OrcScanExecNode, ParquetScanExecNode, PhysicalAggExprNode, PhysicalAliasNode, PhysicalBinaryExprNode, PhysicalCaseNode, PhysicalCastNode, PhysicalGetIndexedFieldExprNode, PhysicalGetMapValueExprNode, PhysicalHashRepartition, PhysicalInListNode, PhysicalIsNotNull, PhysicalIsNull, PhysicalLikeExprNode, PhysicalNamedStructExprNode, PhysicalNegativeNode, PhysicalNot, PhysicalSCAndExprNode, PhysicalSCOrExprNode, PhysicalScalarFunctionNode, PhysicalSortExprNode, PhysicalSparkUDFWrapperExprNode, PhysicalTryCastNode, PhysicalWhenThen, ProjectionExecNode, SortExecNode, StringContainsExprNode, StringEndsWithExprNode, StringStartsWithExprNode, WindowExecNode, WindowExprNode

• oneof ExprType

  • PhysicalColumn column = 1

    column references

  • ScalarValue literal = 2

  • BoundReference bound_reference = 3

  • PhysicalBinaryExprNode binary_expr = 4

    binary expressions

  • PhysicalAggExprNode agg_expr = 5

    aggregate expressions

  • PhysicalIsNull is_null_expr = 6

    null checks

  • PhysicalIsNotNull is_not_null_expr = 7

  • PhysicalNot not_expr = 8

  • PhysicalCaseNode case_ = 9

  • PhysicalCastNode cast = 10

  • PhysicalSortExprNode sort = 11

  • PhysicalNegativeNode negative = 12

  • PhysicalInListNode in_list = 13

  • PhysicalScalarFunctionNode scalar_function = 14

  • PhysicalTryCastNode try_cast = 15

  • PhysicalLikeExprNode like_expr = 20

    like/not like

  • PhysicalSCAndExprNode sc_and_expr = 3000

    sc_and/or

  • PhysicalSCOrExprNode sc_or_expr = 3001

  • PhysicalSparkUDFWrapperExprNode spark_udf_wrapper_expr = 10000

    spark udf wrapper

  • PhysicalSparkScalarSubqueryWrapperExprNode spark_scalar_subquery_wrapper_expr = 10001

    spark scalar subquery wrapper

  • PhysicalGetIndexedFieldExprNode get_indexed_field_expr = 10002

    GetIndexedField

  • PhysicalGetMapValueExprNode get_map_value_expr = 10003

    GetMapValue

  • PhysicalNamedStructExprNode named_struct = 11000

    CreateNamedStruct

  • StringStartsWithExprNode string_starts_with_expr = 20000

    string expressions

  • StringEndsWithExprNode string_ends_with_expr = 20001

  • StringContainsExprNode string_contains_expr = 20002

  • RowNumExprNode row_num_expr = 20100

    RowNum

  • BloomFilterMightContainExprNode bloom_filter_might_contain_expr = 20200

    BloomFilterMightContain

Used in: PhysicalExprNode

• optional PhysicalExprNode expr = 1

• optional ScalarValue key = 2

Used in: PhysicalExprNode

• optional PhysicalExprNode expr = 1

• optional ScalarValue key = 2

Used in: PhysicalRepartition

• repeated PhysicalExprNode hash_expr = 1

• uint64 partition_count = 2

Used in: PhysicalExprNode

• optional PhysicalExprNode expr = 1

• repeated PhysicalExprNode list = 2

• bool negated = 3

Used in: PhysicalExprNode

• optional PhysicalExprNode expr = 1

Used in: PhysicalExprNode

• optional PhysicalExprNode expr = 1

Used in: PhysicalExprNode

• bool negated = 1

• bool case_insensitive = 2

• optional PhysicalExprNode expr = 3

• optional PhysicalExprNode pattern = 4

Used in: PhysicalExprNode

• repeated PhysicalExprNode values = 1

• optional ArrowType return_type = 2

Used in: PhysicalExprNode

• optional PhysicalExprNode expr = 1

Used in: PhysicalExprNode

• optional PhysicalExprNode expr = 1

PhysicalPlanNode is a nested type

Used in: AggExecNode, BroadcastJoinBuildHashMapExecNode, BroadcastJoinExecNode, CoalesceBatchesExecNode, DebugExecNode, ExpandExecNode, FilterExecNode, GenerateExecNode, HashJoinExecNode, IpcWriterExecNode, LimitExecNode, ParquetSinkExecNode, ProjectionExecNode, RenameColumnsExecNode, RssShuffleWriterExecNode, ShuffleWriterExecNode, SortExecNode, SortMergeJoinExecNode, TaskDefinition, UnionInput, WindowExecNode

• oneof PhysicalPlanType

  • DebugExecNode debug = 1

  • ShuffleWriterExecNode shuffle_writer = 2

  • IpcReaderExecNode ipc_reader = 3

  • IpcWriterExecNode ipc_writer = 4

  • ParquetScanExecNode parquet_scan = 5

  • ProjectionExecNode projection = 6

  • SortExecNode sort = 7

  • FilterExecNode filter = 8

  • UnionExecNode union = 9

  • SortMergeJoinExecNode sort_merge_join = 10

  • HashJoinExecNode hash_join = 11

  • BroadcastJoinBuildHashMapExecNode broadcast_join_build_hash_map = 12

  • BroadcastJoinExecNode broadcast_join = 13

  • RenameColumnsExecNode rename_columns = 14

  • EmptyPartitionsExecNode empty_partitions = 15

  • AggExecNode agg = 16

  • LimitExecNode limit = 17

  • FFIReaderExecNode ffi_reader = 18

  • CoalesceBatchesExecNode coalesce_batches = 19

  • ExpandExecNode expand = 20

  • RssShuffleWriterExecNode rss_shuffle_writer = 21

  • WindowExecNode window = 22

  • GenerateExecNode generate = 23

  • ParquetSinkExecNode parquet_sink = 24

  • OrcScanExecNode orc_scan = 25

Used in: PhysicalRepartition

• optional SortExecNode sort_expr = 1

• uint64 partition_count = 2

• repeated ScalarValue list_value = 3

Used in: RssShuffleWriterExecNode, ShuffleWriterExecNode, TaskDefinition

• oneof RepartitionType

  • PhysicalSingleRepartition single_repartition = 1

  • PhysicalHashRepartition hash_repartition = 2

  • PhysicalRoundRobinRepartition round_robin_repartition = 3

  • PhysicalRangeRepartition range_repartition = 4

Used in: PhysicalRepartition

• uint64 partition_count = 1

Used in: PhysicalExprNode

• optional PhysicalExprNode left = 1

• optional PhysicalExprNode right = 2

Used in: PhysicalExprNode

• optional PhysicalExprNode left = 1

• optional PhysicalExprNode right = 2

Used in: PhysicalExprNode

• string name = 1

• ScalarFunction fun = 2

• repeated PhysicalExprNode args = 3

• optional ArrowType return_type = 4

Used in: PhysicalRepartition

• uint64 partition_count = 1

Used in: PhysicalExprNode

• optional PhysicalExprNode expr = 1

• bool asc = 2

• bool nulls_first = 3

Used in: PhysicalExprNode

• bytes serialized = 1

• optional ArrowType return_type = 2

• bool return_nullable = 3

Used in: PhysicalExprNode

• bytes serialized = 1

• optional ArrowType return_type = 2

• bool return_nullable = 3

• repeated PhysicalExprNode params = 4

• string expr_string = 5

Used in: PhysicalExprNode

• optional PhysicalExprNode expr = 1

• optional ArrowType arrow_type = 2

Used in: PhysicalCaseNode

• optional PhysicalExprNode when_expr = 1

• optional PhysicalExprNode then_expr = 2

Contains all valid datafusion scalar type except for List

• BOOL = 0

  arrow::Type::BOOL

• UINT8 = 1

  arrow::Type::UINT8

• INT8 = 2

  arrow::Type::INT8

• UINT16 = 3

  represents arrow::Type fields in src/arrow/type.h

• INT16 = 4

• UINT32 = 5

• INT32 = 6

• UINT64 = 7

• INT64 = 8

• FLOAT32 = 9

• FLOAT64 = 10

• UTF8 = 11

• LARGE_UTF8 = 12

• DATE32 = 13

• NULL = 14

• DECIMAL128 = 15

• DATE64 = 16

• TIMESTAMP_SECOND = 17

• TIMESTAMP_MILLISECOND = 18

• TIMESTAMP_MICROSECOND = 19

• TIMESTAMP_NANOSECOND = 20

• INTERVAL_YEARMONTH = 21

• INTERVAL_DAYTIME = 22

Used in: PhysicalPlanNode

• optional PhysicalPlanNode input = 1

• repeated PhysicalExprNode expr = 2

• repeated string expr_name = 3

• repeated ArrowType data_type = 4

Used in: PhysicalPlanNode

• optional PhysicalPlanNode input = 1

• repeated string renamed_column_names = 2

Used in: PhysicalExprNode

(message has no fields)

Used in: PhysicalPlanNode

• optional PhysicalPlanNode input = 1

• optional PhysicalRepartition output_partitioning = 2

• string rss_partition_writer_resource_id = 3

Used in: PhysicalScalarFunctionNode

• Abs = 0

• Acos = 1

• Asin = 2

• Atan = 3

• Ascii = 4

• Ceil = 5

• Cos = 6

• Digest = 7

• Exp = 8

• Floor = 9

• Ln = 10

• Log = 11

• Log10 = 12

• Log2 = 13

• Round = 14

• Signum = 15

• Sin = 16

• Sqrt = 17

• Tan = 18

• Trunc = 19

• NullIf = 20

• RegexpMatch = 21

• BitLength = 22

• Btrim = 23

• CharacterLength = 24

• Chr = 25

• Concat = 26

• ConcatWithSeparator = 27

• DatePart = 28

• DateTrunc = 29

• InitCap = 30

• Left = 31

• Lpad = 32

• Lower = 33

• Ltrim = 34

• MD5 = 35

• OctetLength = 37

  NullIf=36;

• Random = 38

• RegexpReplace = 39

• Repeat = 40

• Replace = 41

• Reverse = 42

• Right = 43

• Rpad = 44

• Rtrim = 45

• SplitPart = 50

  SHA224=46; SHA256=47; SHA384=48; SHA512=49;

• StartsWith = 51

• Strpos = 52

• Substr = 53

• ToHex = 54

• ToTimestamp = 55

• ToTimestampMillis = 56

• ToTimestampMicros = 57

• ToTimestampSeconds = 58

• Now = 59

• Translate = 60

• Trim = 61

• Upper = 62

• Coalesce = 63

• SparkExtFunctions = 10000

Used in: ColumnStats, PartitionedFile, PhysicalExprNode, PhysicalGetIndexedFieldExprNode, PhysicalGetMapValueExprNode, PhysicalRangeRepartition

• bytes ipc_bytes = 1

Used in: FileScanExecConf

• uint32 limit = 1

  wrap into a message to make it optional

Used in: AggUdaf, BroadcastJoinExecNode, EmptyPartitionsExecNode, ExpandExecNode, FFIReaderExecNode, FileScanExecConf, GenerateUdtf, HashJoinExecNode, IpcReaderExecNode, JoinFilter, SortMergeJoinExecNode, UnionExecNode

• repeated Field columns = 1

Used in: PhysicalPlanNode

• optional PhysicalPlanNode input = 1

• optional PhysicalRepartition output_partitioning = 2

• string output_data_file = 3

• string output_index_file = 4

Used in: PhysicalPlanNode, PhysicalRangeRepartition

• optional PhysicalPlanNode input = 1

• repeated PhysicalExprNode expr = 2

• optional FetchLimit fetch_limit = 3

Used in: PhysicalPlanNode

• optional Schema schema = 1

• optional PhysicalPlanNode left = 2

• optional PhysicalPlanNode right = 3

• repeated JoinOn on = 4

• repeated SortOptions sort_options = 5

• JoinType join_type = 6

Used in: SortMergeJoinExecNode

• bool asc = 1

• bool nulls_first = 2

Used in: FileScanExecConf

• int64 num_rows = 1

• int64 total_byte_size = 2

• repeated ColumnStats column_stats = 3

• bool is_exact = 4

Used in: PhysicalExprNode

• optional PhysicalExprNode expr = 1

• string infix = 2

Used in: PhysicalExprNode

• optional PhysicalExprNode expr = 1

• string suffix = 2

Used in: PhysicalExprNode

• optional PhysicalExprNode expr = 1

• string prefix = 2

Used in: ArrowType

• repeated Field sub_field_types = 1

• optional PartitionId task_id = 1

• optional PhysicalPlanNode plan = 2

• optional PhysicalRepartition output_partitioning = 3

  Output partition for shuffle writer

Used in: ArrowType, Timestamp

• Second = 0

• Millisecond = 1

• Microsecond = 2

• Nanosecond = 3

Used in: ArrowType

• TimeUnit time_unit = 1

• string timezone = 2

Used in: ArrowType

• repeated Field union_types = 1

• UnionMode union_mode = 2

Used in: PhysicalPlanNode

• repeated UnionInput input = 1

• optional Schema schema = 2

• uint32 num_partitions = 3

• uint32 cur_partition = 4

Used in: UnionExecNode

• optional PhysicalPlanNode input = 1

• uint32 partition = 2

Used in: Union

• sparse = 0

• dense = 1

Used in: PhysicalPlanNode

• optional PhysicalPlanNode input = 1

• repeated WindowExprNode window_expr = 2

• repeated PhysicalExprNode partition_spec = 3

• repeated PhysicalExprNode order_spec = 4

• optional WindowGroupLimit group_limit = 5

• bool output_window_cols = 6

Used in: WindowExecNode

• optional Field field = 1

• optional ArrowType return_type = 1000

• WindowFunctionType func_type = 2

• WindowFunction window_func = 3

• AggFunction agg_func = 4

• repeated PhysicalExprNode children = 5

Used in: WindowExprNode

• ROW_NUMBER = 0

• RANK = 1

• DENSE_RANK = 2

Used in: WindowExprNode

• Window = 0

• Agg = 1

Used in: WindowExecNode

• uint32 k = 1