package spark.spark_operator

Mouse Melon logoGet desktop application:
View/edit binary Protocol Buffers messages

enum AggregateMode

operator.proto:1092

Used in: HashAggregate

message ArrowPythonFunction

operator.proto:917

Used in: ArrowPythonUdf

message ArrowPythonUdf

operator.proto:905

Spark 4.1+ scalar @arrow_udf evaluation. The input columns pass through, and one result column is appended for each function in order.

Used in: Operator

message BroadcastNestedLoopJoin

operator.proto:1190

Used in: Operator

enum BuildSide

operator.proto:1210

Used in: BroadcastNestedLoopJoin, HashJoin

enum CompressionCodec

operator.proto:992

Used in: IcebergParquetWriteSettings, ParquetWriter, ShuffleWriter

message ContribScan

operator.proto:95

A `google.protobuf.Any`-shaped envelope for out-of-tree contrib scan messages. Hand-rolled (rather than importing the well-known `google.protobuf.Any`) so the wire format needs no well-known-type import -- Comet compiles this .proto with two toolchains (Rust `prost-build` and the Maven `protoc-jar` plugin) and the latter cannot reliably resolve the bundled well-known types. Field layout is deliberately identical to `Any` (`type_url` + serialized `value` bytes) so the JVM can populate it from a standard `Any.pack(...)` and the native side routes purely on `type_url`.

Used in: Operator

message CrossJoin

operator.proto:1197

Spark's CartesianProductExec without a condition (one with a condition falls back to Spark).

Used in: Operator

(message has no fields)

message CsvOptions

operator.proto:286

Used in: CsvScan

message CsvScan

operator.proto:277

Used in: Operator

message CurrentRow

operator.proto:1271

Used in: LowerWindowFrameBound, UpperWindowFrameBound

(message has no fields)

message DeleteFileList

operator.proto:502

Helper message for deduplicating delete file lists

Used in: IcebergScanCommon

message DeltaDvDescriptor

operator.proto:753

Mirrors delta_kernel::actions::deletion_vector::DeletionVectorDescriptor. Carries everything an executor needs to read + decode the DV bitmap, without materialising it on the driver.

Used in: DeltaScanTask

message DeltaPartitionValue

operator.proto:770

Used in: DeltaScanTask

message DeltaScan

operator.proto:702

Operator payload for a Delta scan.

message DeltaScanCommon

operator.proto:582

Per-scan invariants. Lives at the head of every Delta scan operator payload.

Used in: DeltaScan

message DeltaScanTask

operator.proto:713

Used in: DeltaScan, DeltaScanTaskList

message DeltaScanTaskList

operator.proto:776

JNI return value from planDeltaScan -- the list of tasks for a given snapshot.

message DeltaSparkDvDescriptor

operator.proto:238

Descriptor for a Delta deletion vector, derived from the Delta protocol's DeletionVectorDescriptor. The JVM side (which has delta-spark on the classpath) resolves UUID-relative paths to absolute URLs and Z85-decodes inline bitmaps, so the native side needs neither codec. Executors fetch on-disk bitmaps with a single ranged object-store read; only this small descriptor crosses JNI.

Used in: DeltaSparkPartitionedFile

message DeltaSparkFilePartition

operator.proto:264

Single partition's Delta file list (injected at execution time). Field name matches SparkFilePartition.partitioned_file for consistency.

Used in: DeltaSparkScan

message DeltaSparkPartitionedFile

operator.proto:257

A data file plus its optional deletion vector.

Used in: DeltaSparkFilePartition

message DeltaSparkScan

operator.proto:268

message DeltaSparkScanCommon

operator.proto:210

Delta-table-wide data shared by all partitions (sent once at planning). Produced by the contrib Delta module; the native handler is compiled only when the `delta` Cargo feature is enabled.

Used in: DeltaSparkScan

message Expand

operator.proto:1098

Used in: Operator

message Explode

operator.proto:1150

Used in: Operator

message Filter

operator.proto:929

Used in: Operator

message Following

operator.proto:1261

Used in: UpperWindowFrameBound

message HashAggregate

operator.proto:943

Used in: Operator

message HashJoin

operator.proto:1161

Used in: Operator

message IcebergBinaryPredicate

operator.proto:376

Used in: IcebergPredicate

message IcebergDecimal

operator.proto:343

A decimal literal. `scale`/`precision` let the native side build a self-typed Datum for predicate pushdown (iceberg-rust's Datum is self-typed); partition-value decode reads only `unscaled`. Predicate decimal pushdown is deferred (iceberg-rust page-index gap), so today only `unscaled` is consumed.

Used in: IcebergLiteral

message IcebergDeleteFile

operator.proto:540

Iceberg delete file for MOR tables (positional or equality deletes) Positional: (file_path, row_position) pairs to skip Equality: Column values to filter out (specified by equality_ids)

Used in: IcebergScanCommon

message IcebergFileScanTask

operator.proto:508

Iceberg FileScanTask containing data file, delete files, and residual filter

Used in: IcebergScan

message IcebergLiteral

operator.proto:320

A typed Iceberg primitive value, shared by partition values and predicate literals. The oneof case is the type tag (iceberg-rust's Datum is self-typed and Predicate::bind does not coerce, so the type must be known at construction). Grows by adding a case as Iceberg adds primitives (GEOMETRY/VARIANT/nanosecond timestamps); unrecognized values decode to no-op, never an error.

Used in: IcebergBinaryPredicate, IcebergSetPredicate, PartitionValue

message IcebergLogicalPredicate

operator.proto:378

Used in: IcebergPredicate

message IcebergParquetWriteSettings

operator.proto:821

Parquet-rs `WriterProperties` knobs derived from Iceberg table properties on the JVM driver. All fields use Iceberg defaults when absent in the table properties; the rust side does not re-apply defaults. Iceberg's metrics modes (`write.metadata.metrics.*`) are deliberately NOT translated: the native writer always emits full, untruncated footer statistics (matching parquet-mr, which never truncates), and the JVM re-derives every manifest `DataFile`'s metrics from the footer with Iceberg's own `MetricsConfig` logic before commit. See `CometIcebergWriteExec`.

Used in: IcebergWriteCommon

message IcebergPredicate

operator.proto:364

Iceberg residual predicate for row-group pruning, mirroring iceberg::expr::Predicate. Only a pruning hint: the post-scan CometFilter enforces correctness, so an unrepresentable predicate is simply omitted. Carries no expr_id/query_context (those are for native ANSI error reporting on executable expressions; a residual never errors), which also lets identical residuals dedup.

Used in: IcebergLogicalPredicate, IcebergScanCommon

enum IcebergPredicateOperator

operator.proto:384

Mirrors iceberg-rust PredicateOperator (crates/iceberg/src/expr/mod.rs) so the mapping on both sides is 1:1. The driver emits a subset by design: NOT_IN never prunes from column stats (inherent, matches Iceberg-Java), and IS_NAN/STARTS_WITH are not produced by the current residual path. Unspecified/unrecognized decodes to no-pushdown.

Used in: IcebergBinaryPredicate, IcebergSetPredicate, IcebergUnaryPredicate

message IcebergScan

operator.proto:461

Used in: Operator

message IcebergScanCommon

operator.proto:403

Common data shared by all partitions in split mode (sent once, captured in closure)

Used in: IcebergScan

message IcebergSetPredicate

operator.proto:377

Used in: IcebergPredicate

message IcebergUnaryPredicate

operator.proto:375

Used in: IcebergPredicate

message IcebergWrite

operator.proto:888

Single Iceberg write operator. Per-task fields are populated by the JVM exec wrapper at task launch (matching how `ParquetWriter.output_path` is filled in by the Parquet write execs).

Used in: Operator

message IcebergWriteCommon

operator.proto:841

Broadcast payload -- one of these per write, identical for every task.

Used in: IcebergWrite

enum IcebergWriterMode

operator.proto:807

Picks which iceberg-rust writer the rust side instantiates. Resolved once on the JVM driver (via `spec.isUnpartitioned()` and `SparkWriteConf.useFanoutWriter`) so the rust side never re-derives the choice.

Used in: IcebergWriteCommon

enum JoinType

operator.proto:1200

Used in: BroadcastNestedLoopJoin, HashJoin, SortMergeJoin

message LanceScan

operator.proto:491

message LanceScanCommon

operator.proto:471

Common data shared by all partitions for native Lance scans.

Used in: LanceScan

message LanceScanPartition

operator.proto:486

Used in: LanceScan

message Limit

operator.proto:966

Used in: Operator

message LocalPartitionWriter

operator.proto:1013

Local shuffle output consists of a data file. The partition offsets are returned to the JVM via JNI

Used in: PartitionWriter

message LowerWindowFrameBound

operator.proto:1237

Used in: WindowFrame

enum MergeActionContext

operator.proto:1129

Used in: MergeInstruction

message MergeInstruction

operator.proto:1137

Used in: MergeRows

message MergeOutputRow

operator.proto:1146

Used in: MergeInstruction

message MergeRows

operator.proto:1108

Native counterpart of Spark's `MergeRowsExec` (row-level MERGE dispatch). Mirrors the real Spark 4.x bytecode shape: `isSourceRowPresent` / `isTargetRowPresent` are predicates (not column ordinals), and each instruction is uniformly `condition + outputs`, where the number of output row projections (0/1/2) distinguishes Discard/Keep/Split -- there is no separate instruction-kind enum on the Spark side, so we don't invent one here either.

Used in: Operator

message NativeScan

operator.proto:192

Used in: Operator

message NativeScanCommon

operator.proto:148

Common data shared by all partitions in split mode (sent once at planning)

Used in: DeltaSparkScan, NativeScan

message Operator

operator.proto:32

The basic message representing a Spark operator.

Used in: spark_native_executors.Stage, Window

message ParquetWriter

operator.proto:1054

Used in: Operator

message PartitionData

operator.proto:356

Collection of partition values for a single partition

Used in: IcebergScanCommon

message PartitionValue

operator.proto:350

Partition value for Iceberg partition data

Used in: PartitionData

message PartitionWriter

operator.proto:1005

Selects where encoded shuffle partitions are written.

Used in: ShuffleWriter

message Preceding

operator.proto:1253

Used in: LowerWindowFrameBound

message ProjectFieldIdList

operator.proto:497

Helper message for deduplicating field ID lists

Used in: IcebergScanCommon

message Projection

operator.proto:899

Used in: Operator

message RangeScan

operator.proto:984

Spark's `RangeExec`, which produces the rows of `spark.range` and SQL `range()`. The native side computes its partition's values from these fields and the partition index, the way Spark's `RangeExec` does.

Used in: Operator

enum RankLikeFunction

operator.proto:1299

Used in: WindowGroupLimit

message RssPartitionWriter

operator.proto:1022

Marker for remote shuffle output. The task-owned callback is bound outside the serialized plan and is never transmitted through protobuf.

Used in: PartitionWriter

(message has no fields)

message Sample

operator.proto:974

Sampling without replacement. Spark's SampleExec also supports sampling with replacement, but that path is not serialized to native yet, so there is no `with_replacement` field.

Used in: Operator

message Scan

operator.proto:133

Used in: Operator

message ShuffleScan

operator.proto:141

Used in: Operator

message ShuffleWriter

operator.proto:1024

Used in: Operator

message Sort

operator.proto:935

Used in: Operator

message SortMergeJoin

operator.proto:1180

Used in: Operator

message SparkFilePartition

operator.proto:120

This name and the one above are not great, but they correspond to the (unfortunate) Spark names. I prepended "Spark" since I think there's a name collision on the native side, but we can revisit.

Used in: CsvScan, NativeScan, TextScan

message SparkPartitionedFile

operator.proto:110

Used in: DeltaSparkPartitionedFile, SparkFilePartition

message SparkStructField

operator.proto:124

Used in: spark_native_executors.Fragment, spark_native_executors.Stage, CsvScan, DeltaScanCommon, IcebergScanCommon, LanceScanCommon, NativeScanCommon, ParquetWriter, ShuffleWriter, TextScan

message TextOptions

operator.proto:308

Used in: TextScan

message TextScan

operator.proto:297

Native Text scan, mirroring Spark's `text` data source (a single `value: string` column).

Used in: Operator

message UnboundedFollowing

operator.proto:1270

Used in: UpperWindowFrameBound

(message has no fields)

message UnboundedPreceding

operator.proto:1269

Used in: LowerWindowFrameBound

(message has no fields)

message UpperWindowFrameBound

operator.proto:1245

Used in: WindowFrame

message Window

operator.proto:1279

Used in: Operator

message WindowExpr

operator.proto:1215

Used in: Window

message WindowFrame

operator.proto:1231

Used in: WindowSpecDefinition

enum WindowFrameType

operator.proto:1226

Used in: WindowFrame

message WindowGroupLimit

operator.proto:1292

Top-K rows per partition group. Corresponds to Spark's WindowGroupLimitExec (Spark 3.5+). Rank-like function drives the semantics; the operator only accepts ROW_NUMBER, RANK, and DENSE_RANK.

Used in: Operator

message WindowSpecDefinition

operator.proto:1273

Used in: WindowExpr