package spark.spark_operator

Get desktop application:
View/edit binary Protocol Buffers messages

Used in: HashAggregate

• Partial = 0

• Final = 1

• PartialMerge = 2

Used in: Operator

• JoinType join_type = 1

• BuildSide build_side = 2

• optional spark_expression.Expr condition = 3

Used in: BroadcastNestedLoopJoin, HashJoin

• BuildLeft = 0

• BuildRight = 1

Used in: ParquetWriter, ShuffleWriter

• None = 0

• Zstd = 1

• Lz4 = 2

• Snappy = 3

Used in: CsvScan

• bool has_header = 1

• string delimiter = 2

• string quote = 3

• string escape = 4

• optional string comment = 5

• string terminator = 7

• bool truncated_rows = 8

Used in: Operator

• repeated SparkStructField data_schema = 1

• repeated SparkStructField partition_schema = 2

• repeated int32 projection_vector = 3

• repeated SparkFilePartition file_partitions = 4

• map<string, string> object_store_options = 5

• optional CsvOptions csv_options = 6

Used in: LowerWindowFrameBound, UpperWindowFrameBound

(message has no fields)

Helper message for deduplicating delete file lists

Used in: IcebergScanCommon

• repeated IcebergDeleteFile delete_files = 1

Used in: Operator

• repeated spark_expression.Expr project_list = 1

• int32 num_expr_per_project = 3

Used in: Operator

• optional spark_expression.Expr child = 1

  The array expression to explode into multiple rows

• bool outer = 2

  Whether this is explode_outer (produces null row for empty/null arrays)

• repeated spark_expression.Expr project_list = 3

  Expressions for other columns to project alongside the exploded values

• bool position = 4

  Whether to emit a position column alongside the exploded values (posexplode)

Used in: Operator

• optional spark_expression.Expr predicate = 1

Used in: UpperWindowFrameBound

• int64 offset = 1

  Used for ROWS frames. Integer row count.

• optional spark_expression.Literal range_offset = 2

  Used for RANGE frames. Carries the typed offset value so the native side can build a ScalarValue whose type matches the ORDER BY column.

Used in: Operator

• repeated spark_expression.Expr grouping_exprs = 1

• repeated spark_expression.AggExpr agg_exprs = 2

• AggregateMode mode = 5

• repeated AggregateMode expr_modes = 6

  Per-expression modes for mixed-mode aggregates (e.g., PartialMerge + Partial). When set, each entry corresponds to agg_exprs at the same index. When empty, all expressions use the `mode` field.

• int32 initial_input_buffer_offset = 7

  Offset in the child's output where aggregate buffer attributes start. Used by PartialMerge to locate state fields in the input.

Used in: Operator

• repeated spark_expression.Expr left_join_keys = 1

• repeated spark_expression.Expr right_join_keys = 2

• JoinType join_type = 3

• optional spark_expression.Expr condition = 4

• BuildSide build_side = 5

• bool null_aware_anti_join = 6

  True for BroadcastHashJoinExec null-aware anti-joins (NOT IN subquery semantics). When true, any null in the build side suppresses all left rows.

Iceberg delete file for MOR tables (positional or equality deletes) Positional: (file_path, row_position) pairs to skip Equality: Column values to filter out (specified by equality_ids)

Used in: DeleteFileList

• string file_path = 1

  Delete file path

• string content_type = 2

  POSITION_DELETES or EQUALITY_DELETES

• int32 partition_spec_id = 3

  Partition spec ID

• repeated int32 equality_ids = 4

  Equality field IDs (empty for positional deletes)

Iceberg FileScanTask containing data file, delete files, and residual filter

Used in: IcebergScan

• string data_file_path = 1

  Data file path (e.g., s3://bucket/warehouse/db/table/data/00000-0-abc.parquet)

• uint64 start = 2

  Byte range to read (for split files)

• uint64 length = 3

• optional uint64 record_count = 4

  Record count if reading entire file

• uint64 file_size_in_bytes = 5

  Total file size from the manifest entry, used to skip stat/HEAD calls

• uint32 schema_idx = 15

  Indices into IcebergScan deduplication pools

• optional uint32 partition_type_idx = 16

• optional uint32 partition_spec_idx = 17

• optional uint32 name_mapping_idx = 18

• uint32 project_field_ids_idx = 19

• optional uint32 partition_data_idx = 20

• optional uint32 delete_files_idx = 21

• optional uint32 residual_idx = 22

Used in: Operator

• optional IcebergScanCommon common = 1

  Common data shared across partitions (pools, metadata, catalog props)

• repeated IcebergFileScanTask file_scan_tasks = 2

  Single partition's file scan tasks

Common data shared by all partitions in split mode (sent once, captured in closure)

Used in: IcebergScan

• map<string, string> catalog_properties = 1

  Catalog-specific configuration for FileIO (credentials, S3/GCS config, etc.)

• string metadata_location = 2

  Table metadata file path for FileIO initialization

• repeated SparkStructField required_schema = 3

  Schema to read

• repeated string schema_pool = 4

  Deduplication pools (must contain all entries for cross-partition deduplication)

• repeated string partition_type_pool = 5

• repeated string partition_spec_pool = 6

• repeated string name_mapping_pool = 7

• repeated ProjectFieldIdList project_field_ids_pool = 8

• repeated PartitionData partition_data_pool = 9

• repeated DeleteFileList delete_files_pool = 10

• repeated spark_expression.Expr residual_pool = 11

• uint32 data_file_concurrency_limit = 12

  Number of data files to read concurrently within a single task

• string catalog_name = 13

  Spark V2 catalog name that loaded this table. Forwarded as the dispatchKey to CometS3CredentialDispatcher.ensureInitialized so two catalogs sharing one provider class get isolated provider instances. Empty string when the table has no catalog identity (e.g. HadoopTables loaded by raw path).

Used in: BroadcastNestedLoopJoin, HashJoin, SortMergeJoin

• Inner = 0

• LeftOuter = 1

• RightOuter = 2

• FullOuter = 3

• LeftSemi = 4

• LeftAnti = 5

Used in: Operator

• int32 limit = 1

• int32 offset = 2

Used in: WindowFrame

• oneof lower_frame_bound_struct

  • UnboundedPreceding unboundedPreceding = 1

  • Preceding preceding = 2

  • CurrentRow currentRow = 3

Used in: Operator

• optional NativeScanCommon common = 1

  Common data shared across partitions (schemas, filters, projections, config)

• optional SparkFilePartition file_partition = 2

  Single partition's file list (injected at execution time)

Common data shared by all partitions in split mode (sent once at planning)

Used in: NativeScan

• repeated SparkStructField required_schema = 1

• repeated SparkStructField data_schema = 2

• repeated SparkStructField partition_schema = 3

• repeated spark_expression.Expr data_filters = 4

• repeated int64 projection_vector = 5

• string session_timezone = 6

• repeated spark_expression.Expr default_values = 7

• repeated int64 default_values_indexes = 8

• bool case_sensitive = 9

• map<string, string> object_store_options = 10

• bool encryption_enabled = 11

• string source = 12

• repeated spark_expression.DataType fields = 13

• bool return_null_struct_if_all_fields_missing = 14

  SPARK-53535 (Spark 4.1+): when reading a struct whose requested fields are all missing in the Parquet file, true returns the entire struct as null (legacy pre-4.1 behavior); false preserves the parent struct's nullness from the file so non-null parents return a struct of all-null fields.

• bool use_field_id = 15

  True when spark.sql.parquet.fieldId.read.enabled is set and the requested schema actually carries parquet.field.id metadata. When false the native scan keeps its existing name-based path with no extra work.

• bool ignore_missing_field_id = 16

  True when spark.sql.parquet.fieldId.read.ignoreMissing is set.

• bool allow_type_promotion = 17

  Whether widening type promotion is allowed (e.g. INT32 -> INT64, FLOAT -> DOUBLE). Set from Comet's per-Spark-version constant in ShimCometConf (false on 3.x, true on 4.x). When false, reading a column with a disallowed promoted type throws an error matching Spark's SchemaColumnConvertNotSupportedException behavior.

• bool allow_timestamp_ltz_to_ntz = 18

  When true, reading a Parquet TimestampLTZ column as TimestampNTZ is permitted (Spark 4.0+, SPARK-47447); when false, it is rejected with SchemaColumnConvertNotSupportedException (Spark 3.x, SPARK-36182). Set from Comet's per-Spark-version constant in ShimCometConf.

The basic message representing a Spark operator.

Used in: Window

• repeated Operator children = 1

  The child operators of this

• uint32 plan_id = 2

  Spark plan ID

• oneof op_struct

  • Scan scan = 100

  • Projection projection = 101

  • Filter filter = 102

  • Sort sort = 103

  • HashAggregate hash_agg = 104

  • Limit limit = 105

  • ShuffleWriter shuffle_writer = 106

  • Expand expand = 107

  • SortMergeJoin sort_merge_join = 108

  • HashJoin hash_join = 109

  • Window window = 110

  • NativeScan native_scan = 111

  • IcebergScan iceberg_scan = 112

  • ParquetWriter parquet_writer = 113

  • Explode explode = 114

  • CsvScan csv_scan = 115

  • ShuffleScan shuffle_scan = 116

  • BroadcastNestedLoopJoin broadcast_nested_loop_join = 117

Used in: Operator

• string output_path = 1

• CompressionCodec compression = 2

• repeated string column_names = 4

• optional string work_dir = 5

  Working directory for temporary files (used by FileCommitProtocol) If not set, files are written directly to output_path

• optional string job_id = 6

  Job ID for tracking this write operation

• optional int32 task_attempt_id = 7

  Task attempt ID for this specific task

• map<string, string> object_store_options = 8

  Options for configuring object stores such as AWS S3, GCS, etc. The key-value pairs are taken from Hadoop configuration for compatibility with Hadoop FileSystem implementations of object stores. The configuration values have hadoop. or spark.hadoop. prefix trimmed. For instance, the configuration value "spark.hadoop.fs.s3a.access.key" will be stored as "fs.s3a.access.key" in the map.

Collection of partition values for a single partition

Used in: IcebergScanCommon

• repeated PartitionValue values = 1

Partition value for Iceberg partition data

Used in: PartitionData

• int32 field_id = 1

• oneof value

  • int32 int_val = 2

  • int64 long_val = 3

  • int64 date_val = 4

    days since epoch

  • int64 timestamp_val = 5

    microseconds since epoch

  • int64 timestamp_tz_val = 6

    microseconds with timezone

  • string string_val = 7

  • double double_val = 8

  • float float_val = 9

  • bytes decimal_val = 10

    unscaled BigInteger bytes

  • bool bool_val = 11

  • bytes uuid_val = 12

  • bytes fixed_val = 13

  • bytes binary_val = 14

• bool is_null = 15

Used in: LowerWindowFrameBound

• int64 offset = 1

  Used for ROWS frames. Integer row count.

• optional spark_expression.Literal range_offset = 2

  Used for RANGE frames. Carries the typed offset value so the native side can build a ScalarValue whose type matches the ORDER BY column.

Helper message for deduplicating field ID lists

Used in: IcebergScanCommon

• repeated int32 field_ids = 1

Used in: Operator

• repeated spark_expression.Expr project_list = 1

Used in: Operator

• repeated spark_expression.DataType fields = 1

• string source = 2

  The source of the scan (e.g. file scan, broadcast exchange, shuffle, etc). This is purely for informational purposes when viewing native query plans in debug mode.

Used in: Operator

• repeated spark_expression.DataType fields = 1

• string source = 2

  Informational label for debug output (e.g., "CometShuffleExchangeExec [id=5]")

Used in: Operator

• optional spark_partitioning.Partitioning partitioning = 1

• string output_data_file = 3

• string output_index_file = 4

• CompressionCodec codec = 5

• int32 compression_level = 6

• bool tracing_enabled = 7

• int32 write_buffer_size = 8

  Size of the write buffer in bytes used when writing shuffle data to disk. Larger values may improve write performance but use more memory.

• repeated SparkStructField expected_output_schema = 9

  Spark-declared output schema of the writer's child. When the child is an inlined native subtree, the native planner casts the child's actual output to this schema before serializing to shuffle blocks, since there is no FFI boundary or ScanExec between them to absorb DataFusion-vs-Spark type drift. Empty when the child is a placeholder Scan; that path already has a cast point upstream.

Used in: Operator

• repeated spark_expression.Expr sort_orders = 1

• optional int32 fetch = 3

• optional int32 skip = 4

Used in: Operator

• repeated spark_expression.Expr left_join_keys = 1

• repeated spark_expression.Expr right_join_keys = 2

• JoinType join_type = 3

• repeated spark_expression.Expr sort_options = 4

• optional spark_expression.Expr condition = 5

This name and the one above are not great, but they correspond to the (unfortunate) Spark names. I prepended "Spark" since I think there's a name collision on the native side, but we can revisit.

Used in: CsvScan, NativeScan

• repeated SparkPartitionedFile partitioned_file = 1

Used in: SparkFilePartition

• string file_path = 1

• int64 start = 2

• int64 length = 3

• int64 file_size = 4

• repeated spark_expression.Expr partition_values = 5

Used in: CsvScan, IcebergScanCommon, NativeScanCommon, ShuffleWriter

• string name = 1

• optional spark_expression.DataType data_type = 2

• bool nullable = 3

• map<string, string> metadata = 4

  Spark field metadata. Currently used to carry parquet.field.id through to the native side. Empty when not needed.

Used in: UpperWindowFrameBound

(message has no fields)

Used in: LowerWindowFrameBound

(message has no fields)

Used in: WindowFrame

• oneof upper_frame_bound_struct

  • UnboundedFollowing unboundedFollowing = 1

  • Following following = 2

  • CurrentRow currentRow = 3

Used in: Operator

• repeated WindowExpr window_expr = 1

• repeated spark_expression.Expr order_by_list = 2

• repeated spark_expression.Expr partition_by_list = 3

• optional Operator child = 4

Used in: Window

• optional spark_expression.Expr built_in_window_function = 1

• optional spark_expression.AggExpr agg_func = 2

• optional WindowSpecDefinition spec = 3

• bool ignore_nulls = 4

• optional spark_expression.DataType result_type = 5

  Spark's expected result type. Used to cast the native window function output when DataFusion's return type differs (e.g. row_number returns UInt64 but Spark expects Int32).

Used in: WindowSpecDefinition

• WindowFrameType frame_type = 1

• optional LowerWindowFrameBound lower_bound = 2

• optional UpperWindowFrameBound upper_bound = 3

Used in: WindowFrame

• Rows = 0

• Range = 1

Used in: WindowExpr

• repeated spark_expression.Expr partitionSpec = 1

• repeated spark_expression.Expr orderSpec = 2

• optional WindowFrame frameSpecification = 3