package spark.spark_operator

Get desktop application:
View/edit binary Protocol Buffers messages

Used in: HashAggregate

• Partial = 0

• Final = 1

Used in: HashJoin

• BuildLeft = 0

• BuildRight = 1

Used in: ShuffleWriter

• None = 0

• Zstd = 1

• Lz4 = 2

• Snappy = 3

Used in: LowerWindowFrameBound, UpperWindowFrameBound

(message has no fields)

Used in: Operator

• repeated spark_expression.Expr project_list = 1

• int32 num_expr_per_project = 3

Used in: Operator

• optional spark_expression.Expr predicate = 1

• bool use_datafusion_filter = 2

Used in: UpperWindowFrameBound

• int64 offset = 1

Used in: Operator

• repeated spark_expression.Expr grouping_exprs = 1

• repeated spark_expression.AggExpr agg_exprs = 2

• repeated spark_expression.Expr result_exprs = 3

• AggregateMode mode = 5

Used in: Operator

• repeated spark_expression.Expr left_join_keys = 1

• repeated spark_expression.Expr right_join_keys = 2

• JoinType join_type = 3

• optional spark_expression.Expr condition = 4

• BuildSide build_side = 5

Used in: HashJoin, SortMergeJoin

• Inner = 0

• LeftOuter = 1

• RightOuter = 2

• FullOuter = 3

• LeftSemi = 4

• RightSemi = 5

• LeftAnti = 6

• RightAnti = 7

Used in: Operator

• int32 limit = 1

• int32 offset = 2

Used in: WindowFrame

• oneof lower_frame_bound_struct

  • UnboundedPreceding unboundedPreceding = 1

  • Preceding preceding = 2

  • CurrentRow currentRow = 3

Used in: Operator

• repeated spark_expression.DataType fields = 1

• string source = 2

  The source of the scan (e.g. file scan, broadcast exchange, shuffle, etc). This is purely for informational purposes when viewing native query plans in debug mode.

• repeated SparkStructField required_schema = 3

• repeated SparkStructField data_schema = 4

• repeated SparkStructField partition_schema = 5

• repeated spark_expression.Expr data_filters = 6

• repeated SparkFilePartition file_partitions = 7

• repeated int64 projection_vector = 8

• string session_timezone = 9

The basic message representing a Spark operator.

Used in: Window

• repeated Operator children = 1

  The child operators of this

• uint32 plan_id = 2

  Spark plan ID

• oneof op_struct

  • Scan scan = 100

  • Projection projection = 101

  • Filter filter = 102

  • Sort sort = 103

  • HashAggregate hash_agg = 104

  • Limit limit = 105

  • ShuffleWriter shuffle_writer = 106

  • Expand expand = 107

  • SortMergeJoin sort_merge_join = 108

  • HashJoin hash_join = 109

  • Window window = 110

  • NativeScan native_scan = 111

Used in: LowerWindowFrameBound

• int64 offset = 1

Used in: Operator

• repeated spark_expression.Expr project_list = 1

Used in: Operator

• repeated spark_expression.DataType fields = 1

• string source = 2

  The source of the scan (e.g. file scan, broadcast exchange, shuffle, etc). This is purely for informational purposes when viewing native query plans in debug mode.

Used in: Operator

• optional spark_partitioning.Partitioning partitioning = 1

• string output_data_file = 3

• string output_index_file = 4

• CompressionCodec codec = 5

• int32 compression_level = 6

• bool tracing_enabled = 7

Used in: Operator

• repeated spark_expression.Expr sort_orders = 1

• optional int32 fetch = 3

Used in: Operator

• repeated spark_expression.Expr left_join_keys = 1

• repeated spark_expression.Expr right_join_keys = 2

• JoinType join_type = 3

• repeated spark_expression.Expr sort_options = 4

• optional spark_expression.Expr condition = 5

This name and the one above are not great, but they correspond to the (unfortunate) Spark names. I prepended "Spark" since I think there's a name collision on the native side, but we can revisit.

Used in: NativeScan

• repeated SparkPartitionedFile partitioned_file = 1

Used in: SparkFilePartition

• string file_path = 1

• int64 start = 2

• int64 length = 3

• int64 file_size = 4

• repeated spark_expression.Expr partition_values = 5

Used in: NativeScan

• string name = 1

• optional spark_expression.DataType data_type = 2

• bool nullable = 3

Used in: UpperWindowFrameBound

(message has no fields)

Used in: LowerWindowFrameBound

(message has no fields)

Used in: WindowFrame

• oneof upper_frame_bound_struct

  • UnboundedFollowing unboundedFollowing = 1

  • Following following = 2

  • CurrentRow currentRow = 3

Used in: Operator

• repeated WindowExpr window_expr = 1

• repeated spark_expression.Expr order_by_list = 2

• repeated spark_expression.Expr partition_by_list = 3

• optional Operator child = 4

Used in: Window

• optional spark_expression.Expr built_in_window_function = 1

• optional spark_expression.AggExpr agg_func = 2

• optional WindowSpecDefinition spec = 3

Used in: WindowSpecDefinition

• WindowFrameType frame_type = 1

• optional LowerWindowFrameBound lower_bound = 2

• optional UpperWindowFrameBound upper_bound = 3

Used in: WindowFrame

• Rows = 0

• Range = 1

Used in: WindowExpr

• repeated spark_expression.Expr partitionSpec = 1

• repeated spark_expression.Expr orderSpec = 2

• optional WindowFrame frameSpecification = 3