Get desktop application:
View/edit binary Protocol Buffers messages
Used in:
Used in:
Spark's pickled (function, returnType) command. Broadcast-backed commands are rejected by the JVM planner until broadcast support is implemented.
Empty string for a positional argument, otherwise Python keyword name.
Spark 4.1+ scalar @arrow_udf evaluation. The input columns pass through, and one result column is appended for each function in order.
Used in:
Spark's Arrow writer splits input batches before calling a scalar Arrow UDF. A non-positive value disables the limit.
Spark also stops a batch once its Arrow input buffers reach this size.
Spark 4.2 accepts any result that pyarrow.RecordBatch.from_arrays accepts, such as a list or NumPy array. Spark 4.1 requires a pyarrow.Array.
Used in:
Used in: ,
Used in: , ,
Parquet write only. The shuffle writer rejects this codec.
Iceberg write only; Iceberg inherits Parquet's full codec set. The shuffle and ParquetWriter paths reject this codec.
A `google.protobuf.Any`-shaped envelope for out-of-tree contrib scan messages. Hand-rolled (rather than importing the well-known `google.protobuf.Any`) so the wire format needs no well-known-type import -- Comet compiles this .proto with two toolchains (Rust `prost-build` and the Maven `protoc-jar` plugin) and the latter cannot reliably resolve the bundled well-known types. Field layout is deliberately identical to `Any` (`type_url` + serialized `value` bytes) so the JVM can populate it from a standard `Any.pack(...)` and the native side routes purely on `type_url`.
Used in:
Type name of `value`, namespaced to the owning contrib -- e.g. `type.googleapis.com/comet.contrib.delta.DeltaScan`. The native dispatcher matches on this to pick the contrib that decodes `value`; core itself names no contrib type. This is the contrib's identifier for its own message, NOT necessarily the proto package the message is compiled under today. Contrib scan messages currently live in this file only because a contrib proto build pipeline does not exist yet; they are meant to move under `contrib/<name>/proto`. Keying on a contrib-owned name keeps that relocation invisible on the wire.
The serialized contrib scan message (e.g. an encoded `DeltaScan`).
Spark's CartesianProductExec without a condition (one with a condition falls back to Spark).
Used in:
(message has no fields)
Used in:
Used in:
Used in: ,
(message has no fields)
Helper message for deduplicating delete file lists
Used in:
Indices into IcebergScanCommon.delete_file_pool
Mirrors delta_kernel::actions::deletion_vector::DeletionVectorDescriptor. Carries everything an executor needs to read + decode the DV bitmap, without materialising it on the driver.
Used in:
"u" (UUID-named, relative to <table_root>/_delta_log/deletion_vectors/) "p" (absolute URI in path_or_inline_dv) "i" (inline -- bytes carried in inline_bytes)
For "u": Delta-encoded UUID path component. For "p": absolute URI. For "i": empty.
Byte offset within the DV file ("u"/"p" only). Absent ≡ 0.
Size in bytes of the serialised RoaringBitmap entry.
Number of deleted rows (descriptor metadata; the executor doesn't need to count).
Inline-DV bytes (storage_type="i" only). Empty for "u"/"p".
Used in:
Operator payload for a Delta scan.
Absolute URL of the Delta table root, used by the executor to resolve relative ("u" storage-type) DV paths via kernel's `absolute_path` -- kernel joins `_delta_log/deletion_vectors/<uuid>` onto this, so it MUST end with `/` (the driver normalises before sending). Empty for tables with no DVs (executor simply never decodes), but populate it whenever known.
Per-scan invariants. Lives at the head of every Delta scan operator payload.
Used in:
Required (output) schema -- pure-logical at every nesting level. The kernel-read path physicalises it via `column_mappings`; partition columns are split out by name.
Partition columns (stored in add actions, not in file paths).
Cloud storage credentials / options, keyed by Hadoop-style names.
Root path of the Delta table (file:// or s3://...).
Optional snapshot version; unset = latest.
Per-file parallelism knob within a single Spark task.
Informational label for debug output.
When true the DataSourceExec swallows object-store NotFound errors as empty streams; matches Spark's `spark.sql.files.ignoreMissingFiles=true` semantics.
When true the native parquet reader matches Spark schema fields to parquet file fields by `PARQUET:field_id` metadata rather than by name. Used by Delta tables with `delta.columnMapping.mode = id` (the contrib copies `delta.columnMapping.id` to `PARQUET:field_id` on each schema field).
Delta synthetic columns to append to the scan output via `DeltaSyntheticColumnsExec`. The corresponding columns must NOT appear in `data_schema` or `required_schema` -- they're synthesised at exec time from row positions / DV bitmaps and inserted after the parquet read.
Row-tracking synthesis (row_id = baseRowId + physical_row_index, row_commit_version = defaultRowCommitVersion). Per-task baseRowId / defaultRowCommitVersion travel on `DeltaScanTask` (fields 6 + 7).
When synthetic columns aren't a contiguous suffix of `required_schema`, this sequence tells the native dispatcher how to project the final output. Each entry is an index into [parquet_output, appended_synthetics] -- positive means a column from the (stripped) parquet read at that index; the synthetic tail comes after, in the canonical row_index, is_row_deleted, row_id, row_commit_version order. Empty = synthetics are already a suffix; no reorder needed.
Override for the row_index column name. Defaults to "__delta_internal_row_index" when empty. Delta plans that read `_metadata.row_index` from row-tracking-enabled tables expose the row index as `_tmp_metadata_row_index` (the Delta-internal physical name from `DeltaParquetFileFormat.TMP_METADATA_ROW_INDEX_COLUMN_NAME`) rather than the canonical `__delta_internal_row_index`. Setting this lets the contrib produce a column with the alternate name from the same native synthesis.
Spark `_metadata.*` virtual columns the scan must surface. Delta's `PreprocessTableWithDVs` strategy and the row-tracking projection both inject these into the FileScan output: `file_path`, `file_name`, `file_size`, `file_block_start`, `file_block_length`, `file_modification_time`. Each appears as a flat top-level column in the scan's output schema. Native synthesizes them per-task from `DeltaScanTask` data (file_path, file_size, modification_time) and per-batch derivations. Empty list = no metadata columns to emit. Names are emitted in the order given, appended AFTER synthetic columns (row_index / is_row_deleted / row_id / row_commit_version) unless `final_output_indices` reorders them.
Kernel-built data-column schemas for the kernel-read path, produced by the driver from `delta-kernel-rs`'s `Scan` (`scan.physical_schema()` / `scan.logical_schema()`), serialized as Arrow IPC schema messages. The physical schema carries the correct physical names AND field-ids at EVERY nesting level (kernel's own column-mapping resolution); the logical schema is the matching pure-logical shape. The executor reads parquet with the physical schema and relabels physical->logical via the schema pair -- no Comet-side physicalisation. Empty when the driver didn't build them (e.g. the non-kernel-read add-files path), in which case the executor falls back to physicalising `required_schema` via `column_mappings`.
Stage B/C: when true, DeltaKernelScanExec produces ALL synthetic columns itself (row_index / row_id / file_path via kernel metadata columns; is_row_deleted by inverting the decoded DV; row_commit_version + Spark `_metadata.*` as per-file constants) and assembles the full output by name -- so the standalone DeltaSyntheticColumnsExec is NOT stacked on top and no `final_output_indices` reorder is needed. Default false (legacy DeltaSyntheticColumnsExec path). When set, `required_schema` is the FULL output (synthetics NOT stripped) and the per-file synthesis inputs ride on DeltaScanTask (base_row_id / default_row_commit_version / modification_time / byte range / dv).
Change Data Feed (CDF / readChangeFeed): when true, DeltaKernelScanExec reads table changes via delta-kernel-rs's `TableChanges` API (single partition reconstructs TableChanges(start, end) and calls `execute()`), emitting the data columns plus `_change_type` / `_commit_version` / `_commit_timestamp`. The version range is below; `required_schema` is the full CDF output (assembled by name). No per-file tasks -- the worker enumerates + reads the whole range itself (kernel's CDF per-file API is pub(crate); only execute() is public). Default false.
Inclusive start version of the CDF range.
Inclusive end version of the CDF range. `has_cdf_end_version` distinguishes "latest" (unset).
Delta's test-only deletion-vector filename prefix (`spark.databricks.delta.testOnly.dvFileNamePrefix`, default `test%dv%prefix-` under `Utils.isTesting`, empty in production). Delta prepends it to the DV filename at write time (`<prefix>deletion_vector_<uuid>.bin`) but delta-kernel-rs resolves the un-prefixed name, so executor-side DV reads splice it back in for "u"-storage DVs. Empty = no-op (production).
Used in: ,
File-splitting byte range. Both unset = read whole file.
File modification time in epoch milliseconds. Surfaced through Spark's `_metadata.file_modification_time` column when emit_metadata_columns includes "file_modification_time". Unset = 0 (rare; AddFile usually carries one).
Deletion-vector descriptor for this file (absent = no DV applies). Mirrors delta_kernel::actions::deletion_vector::DeletionVectorDescriptor. The executor reads the actual DV bitmap on-task via dv_reader::read_dv_indexes -- the driver only ships the path/offset/size, so the per-scan-exec heap stays KB-scale regardless of DV size.
Kernel's own fully-resolved physical->logical transform for THIS file, serialized as JSON (serde of `delta_kernel::expressions::Expression`). The driver gets it from kernel's scan enumeration (`ScanMetadata.scan_file_transforms`, one per scan file), where partition values and `baseRowId` (for row_id `GenerateRowId`) are already baked in as literals. The executor deserializes it and applies `transform_to_logical` -- so partition injection, column-mapping relabel (incl. nested), and row-tracking come from kernel rather than Comet-side reconstruction. Empty when the driver has no kernel transform for this file (e.g. a plain table with no mapping / partitions / row-tracking -> identity), in which case the executor uses a pass-through read.
JNI return value from planDeltaScan -- the list of tasks for a given snapshot.
Kernel-built data-column schemas (Arrow IPC), produced by the driver from the projected `Scan` (`scan.physical_schema()` / `scan.logical_schema()`). Carried back to Scala and copied onto `DeltaScanCommon.kernel_physical_schema` / `kernel_logical_schema`. Empty when the driver had no projection to build the scan against.
Descriptor for a Delta deletion vector, derived from the Delta protocol's DeletionVectorDescriptor. The JVM side (which has delta-spark on the classpath) resolves UUID-relative paths to absolute URLs and Z85-decodes inline bitmaps, so the native side needs neither codec. Executors fetch on-disk bitmaps with a single ranged object-store read; only this small descriptor crosses JNI.
Used in:
Original storage form, for diagnostics: "u" (UUID-relative), "i" (inline), "p" (absolute path).
Absolute URL of the DV file (on-disk forms). At descriptor.offset the file holds [i32 BE size][bitmap data][i32 BE CRC32-of-data].
The bitmap data (magic + RoaringBitmapArray), already unframed and Z85-decoded (inline form).
Byte offset of the size-prefixed bitmap within the DV file.
Length of the bitmap data (excluding the size/CRC framing).
Number of deleted rows encoded in the bitmap.
Single partition's Delta file list (injected at execution time). Field name matches SparkFilePartition.partitioned_file for consistency.
Used in:
A data file plus its optional deletion vector.
Used in:
Reuses the parquet scan's common data (schemas, filters, projections, object-store options, reader flags) -- the Delta read path delegates to the same native parquet machinery as NativeScan.
Delta-table-wide data shared by all partitions (sent once at planning). Produced by the contrib Delta module; the native handler is compiled only when the `delta` Cargo feature is enabled.
Used in:
Table root URL, used to resolve relative deletion-vector paths.
Column mapping mode: "none", "name", or "id".
Key for split-mode plan-data injection: a hash of the table root, required schema, data filters, projection and fields, prefixed with common.source and ending with the scan op's plan_id, so two scans of one table in one native plan get distinct keys. Read only on the JVM side.
Effective datetime rebase read modes (LegacyBehaviorPolicy values of spark.sql.parquet.datetimeRebaseModeInRead / int96RebaseModeInRead, resolved through ParquetOptions so per-relation options win, exactly as ParquetFileFormat.buildReaderWithPartitionValues resolves them). Consulted only for files whose footer metadata does not decide the rebase policy on its own (no org.apache.spark.version key), mirroring DataSourceUtils.getRebaseSpec's modeByConfig fallback. Empty (an older producer) is read as EXCEPTION, the conservative refuse-ancient posture.
Used in:
Used in:
The array or map expression to explode into multiple rows
Whether this is explode_outer (produces null row for empty/null collections)
Expressions for other columns to project alongside the exploded values
Whether to emit a position column alongside the exploded values (posexplode)
Used in:
Explicit opt-in for a deterministic residual. Absent metadata preserves the safe default.
Used in:
Used for ROWS frames. Integer row count.
Used for RANGE frames. Carries the typed offset value so the native side can build a ScalarValue whose type matches the ORDER BY column.
Used in:
Per-expression modes for mixed-mode aggregates (e.g., PartialMerge + Partial). When set, each entry corresponds to agg_exprs at the same index. When empty, all expressions use the `mode` field.
Offset in the child's output where aggregate buffer attributes start. Used by PartialMerge to locate state fields in the input.
Set for Spark's SortAggregateExec, which reports its output as ordered by the grouping keys (ascending, nulls first). The native planner sorts the aggregate output on the grouping columns unless the aggregate already emits its groups in that order.
Attach this partial MIN/MAX execution's live bound to an eligible native reader.
Used in:
True for BroadcastHashJoinExec null-aware anti-joins (NOT IN subquery semantics). When true, any null in the build side suppresses all left rows.
Opt-in completed-build filtering of probe batches and eligible native Parquet readers. Older serialized plans leave it false.
The streamed side's sort order, or the longest prefix of it Comet can express, as `SortOrder` expressions over its output, when Spark reports that order for the join and the native join keeps it only if the probe input declares it: a left outer join built on the right, or a right outer join built on the left. Empty otherwise.
Used in:
A decimal literal. `scale`/`precision` let the native side build a self-typed Datum for predicate pushdown (iceberg-rust's Datum is self-typed); partition-value decode reads only `unscaled`. Predicate decimal pushdown is deferred (iceberg-rust page-index gap), so today only `unscaled` is consumed.
Used in:
unscaled value, two's-complement big-endian
Iceberg delete file for MOR tables (positional or equality deletes) Positional: (file_path, row_position) pairs to skip Equality: Column values to filter out (specified by equality_ids)
Used in:
Index into IcebergScanCommon.delete_file_path_pool.
POSITION_DELETES or EQUALITY_DELETES
Partition spec ID
Equality field IDs (empty for positional deletes)
Deletion vector coordinates, set only when file_format is PUFFIN. referenced_data_file is the data file the vector applies to; content_offset and content_size_in_bytes locate the deletion-vector-v1 blob within its Puffin file. Iceberg also populates referencedDataFile on file-scoped Parquet position deletes, so file_format, not the presence of this field, is what identifies a deletion vector.
Serialized StandardKeyMetadata for an encrypted delete file, from DeleteFile.keyMetadata(). Absent for unencrypted tables. Decoded directly by iceberg-rust; no KMS unwrap needed.
Delete-file record count. Required by iceberg-rust for deletion vectors, which check it against the cardinality decoded from the blob; unused for Parquet delete files.
"PARQUET" for a delete file, "PUFFIN" for a deletion vector. iceberg-rust selects its deletion-vector reader on this.
Iceberg FileScanTask containing data file, delete files, and residual filter
Used in:
Data file path (e.g., s3://bucket/warehouse/db/table/data/00000-0-abc.parquet)
Byte range to read (for split files)
Record count if reading entire file
Total file size from the manifest entry, used to skip stat/HEAD calls
Serialized StandardKeyMetadata (plaintext DEK + AAD prefix) for an encrypted data file, taken verbatim from Iceberg's DataFile.keyMetadata(). Absent for unencrypted tables. iceberg-rust decodes this directly; no KMS unwrap is needed on the native side.
Indices into IcebergScan deduplication pools
A typed Iceberg primitive value, shared by partition values and predicate literals. The oneof case is the type tag (iceberg-rust's Datum is self-typed and Predicate::bind does not coerce, so the type must be known at construction). Grows by adding a case as Iceberg adds primitives (GEOMETRY/VARIANT/nanosecond timestamps); unrecognized values decode to no-op, never an error.
Used in: , ,
days since epoch
micros since epoch, no tz
micros since epoch, utc
Used in:
Parquet-rs `WriterProperties` knobs derived from Iceberg table properties on the JVM driver. All fields use Iceberg defaults when absent in the table properties; the rust side does not re-apply defaults. Iceberg's metrics modes (`write.metadata.metrics.*`) are deliberately NOT translated: the native writer always emits full, untruncated footer statistics (matching parquet-mr, which never truncates), and the JVM re-derives every manifest `DataFile`'s metrics from the footer with Iceberg's own `MetricsConfig` logic before commit. See `CometIcebergWriteExec`.
Used in:
Codec-dependent: zstd 1-22, gzip 0-9, brotli 0-11. JVM side substitutes parquet-mr's per-codec default when the property is unset and parquet-rs's default would diverge (zstd: 3 instead of parquet-rs's 1). Absent => use parquet-rs's natural default.
Iceberg `write.parquet.row-group-size-bytes` (default 128 MiB).
Iceberg `write.parquet.page-size-bytes` (default 1 MiB).
Iceberg `write.parquet.dict-size-bytes` (default 2 MiB).
Iceberg `write.parquet.page-row-limit` (default 20_000).
String written into parquet file metadata. JVM-side default is `"Apache Iceberg <ver> (Comet)"`.
Iceberg residual predicate for row-group pruning, mirroring iceberg::expr::Predicate. Only a pruning hint: the post-scan CometFilter enforces correctness, so an unrepresentable predicate is simply omitted. Carries no expr_id/query_context (those are for native ANSI error reporting on executable expressions; a residual never errors), which also lets identical residuals dedup.
Used in: ,
IS_NULL / NOT_NULL / IS_NAN / NOT_NAN
column OP literal
IN only emitted; not_in is inherently unprunable from stats
Mirrors iceberg-rust PredicateOperator (crates/iceberg/src/expr/mod.rs) so the mapping on both sides is 1:1. The driver emits a subset by design: NOT_IN never prunes from column stats (inherent, matches Iceberg-Java), and IS_NAN/STARTS_WITH are not produced by the current residual path. Unspecified/unrecognized decodes to no-pushdown.
Used in: , ,
Used in:
Common data shared across partitions (pools, metadata, catalog props). In the per-partition message the driver ships, it holds only that partition's per-file pool entries.
Single partition's file scan tasks
Common data shared by all partitions in split mode (sent once, captured in closure)
Used in:
Catalog-specific configuration for FileIO (credentials, S3/GCS config, etc.)
Table metadata file path for FileIO initialization
Schema to read
Deduplication pools. In the common block every task gets, the per-file pools (partition_data_pool, delete_files_pool, residual_pool, delete_file_pool, delete_file_path_pool) are empty: each partition's IcebergScan.common holds the entries its tasks use, and the JVM merges the two before the plan reaches native. The driver ships a partition's delete files, their paths, and its residuals apart from its message, shared with the other partitions that use them, and the executor JVM adds them to the partition's pools.
Index-aligned with partition_spec_pool: entry i is the StructType JSON for the spec at partition_spec_pool[i]. Native uses this pairing to recover each entry's spec_id when merging partition types across historical specs (see parse_file_scan_tasks_from_common in planner.rs), which partition_type_pool cannot express on its own since it is only resolved field types, not specs.
Number of data files to read concurrently within a single task
Spark V2 catalog name that loaded this table. Forwarded as the dispatchKey to CometS3CredentialDispatcher.ensureInitialized so two catalogs sharing one provider class get isolated provider instances. Empty string when the table has no catalog identity (e.g. HadoopTables loaded by raw path).
Flat pool of unique delete files, referenced by index from DeleteFileList. Iceberg's default partition delete granularity has one delete file apply to many data files, so interning each delete file here (rather than embedding copies per delete_files_pool set) keeps a partition's pools from growing with the number of references.
Flat pool of unique delete-file paths, referenced by index from IcebergDeleteFile. A V3 deletion vector's path is the Puffin file holding it, and every vector written by one commit shares that file, so the path would otherwise repeat once per data file: delete_file_pool cannot fold those entries together because each carries a distinct content offset.
Iceberg's SparkScan.hashCode() (folds in pushed filters, snapshot, branch, and read schema). A self-join/self-merge can read the same table (same metadata_location) twice with different projections or filters, so PlanDataInjector keys planning data by metadata_location, scan_hash_code and the operator's plan_id rather than metadata_location alone.
Used in:
Used in:
Single Iceberg write operator. Per-task fields are populated by the JVM exec wrapper at task launch (matching how `ParquetWriter.output_path` is filled in by the Parquet write execs).
Used in:
Spark `TaskContext.get().partitionId()` -- 0-padded into the filename prefix `{partition_id:05d}-{task_attempt_id}-{operation_id}`.
Spark `TaskContext.get().taskAttemptId()`.
Broadcast payload -- one of these per write, identical for every task.
Used in:
Catalog properties for FileIO (same shape as `IcebergScanCommon.catalog_properties`).
Iceberg table metadata file path. Kept on the wire for plan-debug visibility (shows up in `IcebergWriteExec`'s `DisplayAs`); `FileIO` itself is initialised from `data_location` since both paths resolve to the same storage scheme.
Iceberg `SchemaParser.toJson(table.schema())`. Carries the schema id inside the JSON; the rust side parses it via `serde_json::from_str::<Schema>`.
Iceberg `PartitionSpecParser.toJson(table.spec())`. Same wire format as one entry in `IcebergScanCommon.partition_spec_pool`. Carries the spec id inside the JSON.
Iceberg `SparkWriteConf.outputSortOrderId(...)`. The rust side ignores it (iceberg-rust's writer does not expose the field); the JVM stamps it onto each decoded `DataFile` before commit. It rides in the common payload so the exec wrapper can read it back from the plan.
Directory under which data files are written; resolved on the JVM via the table's `LocationProvider` plus any `write.data.path` override.
Iceberg `OutputFileFactory` operation UUID -- embedded in filenames so files from the same logical write are correlated across tasks.
Target file size in bytes; the rust rolling writer rolls files at this threshold.
Spark V2 catalog name that loaded this table. Forwarded as the dispatchKey to CometS3CredentialDispatcher.ensureInitialized (and the access intent to the credential provider) so two catalogs sharing one provider class get isolated provider instances. Empty string when the table has no catalog identity (e.g. HadoopTables loaded by raw path). Mirrors `IcebergScanCommon.catalog_name`.
Picks which iceberg-rust writer the rust side instantiates. Resolved once on the JVM driver (via `spec.isUnpartitioned()` and `SparkWriteConf.useFanoutWriter`) so the rust side never re-derives the choice.
Used in:
Used in: , ,
Common data shared by all partitions for native Lance scans.
Used in:
Used in:
Used in:
Local shuffle output consists of a data file. The partition offsets are returned to the JVM via JNI
Used in:
Used in:
Used in:
Used in:
Always present on the Spark side (Keep/Discard/Split.condition are non-optional).
0 output rows = Discard, 1 = Keep, 2 = Split.
Present only for a one-row Keep. Required when semantic_metrics_required is true.
Used in:
Native counterpart of Spark's `MergeRowsExec` (row-level MERGE dispatch). Mirrors the real Spark 4.x bytecode shape: `isSourceRowPresent` / `isTargetRowPresent` are predicates (not column ordinals), and each instruction is uniformly `condition + outputs`, where the number of output row projections (0/1/2) distinguishes Discard/Keep/Split -- there is no separate instruction-kind enum on the Spark side, so we don't invent one here either.
Used in:
Ordinal (into the child row) of the target row-id column used for cardinality dedup. Mirrors `MergeRowsExec.checkCardinality`: present iff the check is on, in which case MERGE_CARDINALITY_VIOLATION is raised if a target row matches more than one source row.
Schema of the emitted rows, matching `MergeRowsExec.output`.
True when every Keep instruction must retain its Spark action context so the native operator can populate Spark 4.1+'s semantic MERGE counters.
Used in:
Common data shared across partitions (schemas, filters, projections, config)
Single partition's file list (injected at execution time)
Hash part of the key under which this scan's planning data is stored and looked up at execution time; the other parts are common.source and the operator's plan_id. Derived once on the driver (CometNativeScanExec) so executors rebuild the key instead of hashing the common per task. JVM-consumed only; the native side ignores it.
Common data shared by all partitions in split mode (sent once at planning)
Used in: ,
SPARK-53535 (Spark 4.1+): when none of the requested leaves of a struct, list or map exist in the Parquet file, true returns the entire value as null (legacy pre-4.1 behavior); false preserves the parent's nullness from the file, so non-null parents keep their structure with all-null leaves.
True when spark.sql.parquet.fieldId.read.enabled is set and the requested schema actually carries parquet.field.id metadata. When false the native scan keeps its existing name-based path with no extra work.
True when the requested schema carries a field id and spark.sql.parquet.fieldId.read.ignoreMissing is not set. The native scan then refuses a file whose Parquet schema carries no field id, as Spark's ParquetReadSupport does whether or not the read flag is set.
Whether widening type promotion is allowed (e.g. INT32 -> INT64, FLOAT -> DOUBLE). Set from Comet's per-Spark-version constant in ShimCometConf (false on 3.x, true on 4.x). When false, reading a column with a disallowed promoted type throws an error matching Spark's SchemaColumnConvertNotSupportedException behavior.
When true, reading a Parquet TimestampLTZ column as TimestampNTZ is permitted (Spark 4.0+, SPARK-47447); when false, it is rejected with SchemaColumnConvertNotSupportedException (Spark 3.x, SPARK-36182). Set from Comet's per-Spark-version constant in ShimCometConf.
True when Spark supplied a Parquet data filter, including when Comet could not serialize any of those filters into data_filters.
The basic message representing a Spark operator.
Used in: ,
The child operators of this
Spark plan ID
Pool of distinct SQL texts referenced by `QueryContext.sql_text_idx`. Every expression that carries a QueryContext needs the full query text to render Spark's `== SQL ... ==` error block, and in a typical plan they all want the *same* text -- embedding it per expression made the query text over 95% of the serialized plan. The JVM interns the texts into this pool on the root operator of each serialized native block (see `QueryContextInterner`) and leaves `QueryContext.sql_text` empty. Only populated on the root operator; nested operators leave it empty.
Extension point for optional, out-of-tree contrib scans (Delta, Lance, ...). The concrete scan message (e.g. `DeltaScan`) is packed into this envelope on the JVM side and dispatched by `type_url` on the native side. Using a single permanent field -- rather than a new oneof variant per format -- means core's proto never has to change when a contrib is added, and avoids field-number collisions between independent contrib PRs. Only decoded when core is built with the matching contrib Cargo feature (e.g. `--features contrib-delta`); default builds reject any `contrib_scan` with a clear "not compiled in" error, so the contrib has zero runtime cost.
Used in:
The fully-qualified path of the Parquet file this task writes, set per task from FileCommitProtocol.newTaskTempFile by CometWriteFilesExec (Spark 4.0+) or CometNativeWriteExec (Spark 3.x). Naming and staging are owned by Spark's commit protocol so that task-attempt isolation, speculative execution, and committers that track individual files (S3A magic, streaming manifest) all behave as they do for Spark's own writer. The native writer uses this path verbatim, and creates the file itself unless `jvm_output_stream` is set.
Options for configuring object stores such as AWS S3, GCS, etc. The key-value pairs are taken from Hadoop configuration for compatibility with Hadoop FileSystem implementations of object stores. The configuration values have hadoop. or spark.hadoop. prefix trimmed. For instance, the configuration value "spark.hadoop.fs.s3a.access.key" will be stored as "fs.s3a.access.key" in the map.
Catalyst's target schema, including top-level nullability and Parquet field IDs. Nested collection field IDs are carried by the corresponding DataType messages.
Whether the JVM has already created `output_path` through Hadoop's FileSystem API and registered its output stream with the plan (Native.setOutputStream). The native writer then writes the file's bytes to that stream instead of opening the path itself, and the JVM closes the file afterwards. Set per task by CometWriteFilesExec.writeFile, which both write operators call, for every file that is neither local nor on HDFS. See HadoopFileOutput for why.
The Spark version (`SPARK_VERSION_SHORT`) written into the footer's key-value metadata as `org.apache.spark.version`, as Spark's own ParquetWriteSupport does. Spark's datetime rebase logic reads it back to tell files written by Spark 3.0+ from older ones. Optional so native callers such as tests can omit it; when absent or empty the key is not written.
Collection of partition values for a single partition
Used in:
Partition value for Iceberg partition data
Used in:
Selects where encoded shuffle partitions are written.
Used in:
Used in:
Used for ROWS frames. Integer row count.
Used for RANGE frames. Carries the typed offset value so the native side can build a ScalarValue whose type matches the ORDER BY column.
Helper message for deduplicating field ID lists
Used in:
Used in:
Spark's `RangeExec`, which produces the rows of `spark.range` and SQL `range()`. The native side computes its partition's values from these fields and the partition index, the way Spark's `RangeExec` does.
Used in:
Spark's element count. Comet sends only ranges whose count fits in a long.
Used in:
Marker for remote shuffle output. The task-owned callback is bound outside the serialized plan and is never transmitted through protobuf.
Used in:
(message has no fields)
Sampling without replacement. Spark's SampleExec also supports sampling with replacement, but that path is not serialized to native yet, so there is no `with_replacement` field.
Used in:
Seed before the partition index is added, which happens on the native side.
Used in:
The source of the scan (e.g. file scan, broadcast exchange, shuffle, etc). This is purely for informational purposes when viewing native query plans in debug mode.
Used in:
Informational label for debug output (e.g., "CometShuffleExchangeExec [id=5]")
Used in:
Retained for compatibility with native binaries that predate partition_writer. Local plans also carry this path in partition_writer.local.
Size of the write buffer in bytes used when writing shuffle data to disk. Larger values may improve write performance but use more memory.
Spark-declared output schema of the writer's child. When the child is an inlined native subtree, the native planner casts the child's actual output to this schema before serializing to shuffle blocks, since there is no FFI boundary or ScanExec between them to absorb DataFusion-vs-Spark type drift. Empty when the child is a placeholder Scan; that path already has a cast point upstream.
Maximum number of bytes that the writer buffers in memory before spilling to disk. Zero means no limit, in which case spilling is driven only by memory pool pressure.
Explicit output destination. When absent, use the legacy output file fields.
Store the offsets of variable-width columns as lengths in local shuffle blocks, which compress better. The native writer applies it only with zstd for now.
Used in:
Attach this TopK execution's live threshold to an eligible native reader.
Used in:
Spark sets this for a right sort reading a single integer equality key from a Comet shuffle.
This name and the one above are not great, but they correspond to the (unfortunate) Spark names. I prepended "Spark" since I think there's a name collision on the native side, but we can revisit.
Used in: , ,
Used in: ,
Used in: , , , , , , , , ,
Spark field metadata. Currently used to carry parquet.field.id through to the native side. Empty when not needed.
Used in:
When true, each file is read as a single row instead of being split into lines.
Custom line separator, as the exact bytes Spark computed (lineSeparatorInRead, already encoded with the file's charset). When unset, lines are split on `\n`, `\r\n`, or `\r`.
Native Text scan, mirroring Spark's `text` data source (a single `value: string` column).
Used in:
Used in:
(message has no fields)
Used in:
(message has no fields)
Used in:
Used in:
The partition keys ascending, when Spark planned no sort below the window because its input reports that order. Empty otherwise.
Used in:
Spark's expected result type. Used to cast the native window function output when DataFusion's return type differs (e.g. row_number returns UInt64 but Spark expects Int32).
Used in:
Used in:
Top-K rows per partition group. Corresponds to Spark's WindowGroupLimitExec (Spark 3.5+). Rank-like function drives the semantics; the operator only accepts ROW_NUMBER, RANK, and DENSE_RANK.
Used in:
Used in: