Proto commits in apache/datafusion-comet

These commits are when the Protocol Buffers files have changed: (only the last 100 relevant commits are shown)

2026-08-19

feat: build gate + inert wiring for contrib Delta scans [Delta contrib split, part 2] (#4952) Part 2 of the Delta contrib split: the build gate and the inert core wiring an out-of-tree scan contrib plugs into. Nothing here is reachable on a default build -- no contrib is registered, no contrib class is compiled, and the native library carries zero contrib symbols. Core gains two format-agnostic extension points, both discovered at runtime so core holds no compile-time reference to any contrib: - `CometScanContrib`, a ServiceLoader-discovered hook (mirroring `PlanDataInjector`) that lets a contrib claim a V1 or V2 scan before Comet's built-in handling runs, plus `CometContribScanMarker` so `CometExecRule` can route a contrib's scan node to the contrib's own serde handler by a plain type test. - `ContribScan contrib_scan = 200`, a single permanent `Any`-shaped proto envelope (`type_url` + packed `value`) dispatched by `type_url` on the native side. Core's oneof never grows per-format, so independent contrib PRs cannot collide on a field number -- as `main` taking field 118 for `Sample` has since demonstrated. Plus the build machinery: the `contrib-delta` Maven profile and Cargo feature, and `dev/verify-contrib-delta-gate.sh`, which asserts a default build compiles no contrib classes, packages no contrib `META-INF/services` files, and links no contrib symbols. Where the hooks sit, and why. Both run *before* Comet's built-in guards for their scan kind, because a contrib may support things the built-in scan does not -- the Delta contrib synthesises `_metadata.*` in its own reader, and a contrib's table name may end in `files`/`snapshots` like an Iceberg metadata table. Applying those guards first would decline such a scan before the contrib was ever offered it. So `transformV1Scan` consults the contrib ahead of the metadata-column guard, and the Iceberg metadata-table check moves out of the outer `transformScan` match into `transformV2Scan`, after its hook. Core's per-path metadata handling is otherwise untouched: `main` serves `fileConstantMetadataColumns` natively in V1 and the Iceberg metadata columns in V2, and both keep doing so. Ownership contract. An implementation MUST return `None` for a scan it does not own: contribs are offered a scan one at a time and the first claim wins, so a contrib claiming another format's scan hides it from the contrib that could have read it, with the outcome depending on unspecified ServiceLoader ordering. "Own but cannot handle" is a distinct, expressible case -- claim the scan and terminate it with `withFallbackReason` rather than declining. Core cannot arbitrate competing claims (a claim is opaque; the only way to know a second contrib would also have claimed is to ask it, which is what claiming prevents), so the contract carries it. Tests. `CometScanContribSuite` covers the registry contract on a default build: no contribs registered (asserted against raw ServiceLoader discovery, not just the registry -- `contribs` swallows a ServiceConfigurationError, so "empty" alone is ambiguous), a stub discovered through a URLClassLoader whose claim is returned, decline-passes-through, first-claim-wins with later contribs not consulted, throw-is-a-decline, and LinkageError still propagating. `CometScanRuleSuite` gains a V1 case asserting the fallback *reason* for `_metadata.row_index`; verified red with the guard removed. Co-authored-by: Claude Opus 5 <noreply@anthropic.com>

The documentation is generated from this commit.

2026-08-18

fix: preserve Catalyst nullability and field IDs in native Parquet writes (#5369)

2026-08-02

perf: intern QueryContext SQL text into a per-plan pool (up to 20x smaller serialized plans for TPC-DS) (#5204)

2026-08-01

feat: support Iceberg metadata columns _pos, _spec, _file, and _partition (#4752) * feat: [iceberg] support iceberg metadata columns _pos, _spec, _file, _partition

2026-07-31

feat: native uuid() implementation compatible with Spark (#5034) * feat: native uuid() implementation compatible with Spark * refactor: share MersenneTwister, use uuid crate for formatting Extract SparkMersenneTwister into internal/mersenne.rs so shuffle and uuid both depend on it rather than uuid reaching into the shuffle module. Format UUIDs via the uuid crate encoded into a pre-sized StringBuilder, removing per-row String allocation and the hand-rolled hex formatting. * test: address review feedback on uuid tests Fills the five coverage gaps flagged on #5034: - Multi-partition bit-for-bit: uuid(42) FROM ... DISTRIBUTE BY id exercises partitionIndex != 0, which a single-partition test cannot catch (both engines would silently agree even if partitionIndex were ignored). - Aliased projections: uuid(0) = uuid(0) (seeded) and uuid() = uuid() (unseeded, spark_answer_only) pin the stateful-alias semantics that freshCopyIfContainsStatefulExpression relies on. - Empty batch: a Rust test that evaluate() on a zero-row batch does not advance RNG state, guarding against a stray next_uuid() outside the row loop. - Golden fixture: five seeds x five UUIDs captured from Commons Math3's MersenneTwister (the exact RNG behind Spark's RandomUUIDGenerator) lock all 128 bits and cover negative / MIN / MAX seeds; catches next_long regressions that shuffle's next_int tests would miss. - No-scan path: length(uuid(0)) pins the OneRowRelation shape with a deterministic assertion alongside the existing SELECT uuid(0). * test: make uuid partition and seed coverage non-vacuous The DISTRIBUTE BY query never exercised partitionIndex != 0. `... FROM t DISTRIBUTE BY id` parses as RepartitionByExpression on top of the Project, so uuid ran on the scan's partitions, and five tiny files pack into a single FilePartition. Moved the projection above the exchange: SELECT uuid(42) FROM (SELECT id FROM test_uuid_parts DISTRIBUTE BY id) Add CometUuidExpressionSuite, which builds `Uuid(Some(seed))` directly through the existing version-shimmed `getColumnFromExpression`. This gives 3.4 and 3.5 a real cross-engine assertion for the first place: the SQL `uuid(seed)` form is 4.0+, so uuid_with_seed.sql is skipped there and the Rust golden constants were the only guard, with no way to confirm they came from Commons Math3 rather than from the Rust code they are meant to test. One test covers five seeds including negative/MIN/MAX; the other repartitions and asserts getNumPartitions > 1 before comparing, so the partition-offset coverage is checked rather than assumed. Verified by mutation: dropping `expr.seed.wrapping_add(planner.partition())` in UuidBuilder::build fails both new Scala tests. Restored, all pass on spark-3.5 and spark-4.1. Also: - Drop `spark_answer_only` from `uuid() = uuid()` in uuid.sql. The values are random but the comparison is not -- distinct fresh seeds mean false on every row -- so default query mode keeps the value check and adds the nativeness assertion that the guard depends on. - Record the real reason typeof(uuid(0)) cannot be used: Spark's TypeOf.doGenCode interpolates `child.dataType.catalogString` unquoted into generated Java, emitting `UTF8String.fromString(string)`. That code is byte-identical on 3.5, 4.0 and master, so it is not a 4.1 quirk as previously recorded. It is normally hidden because TypeOf.foldable is true and ConstantFolding removes the node; this suite excludes ConstantFolding, which exposes it. Captured as a harness constraint: no SQL fixture here can use typeof on a non-foldable input. - Note on the Rust golden test that CometUuidExpressionSuite is now its backstop, and that regenerating the constants from the Rust side would make it circular. No partition-count assertion is added to the SQL fixture: that suite compares Comet against Spark, so if the plan collapsed to one partition both engines would collapse identically and any such check would still pass. It is only assertable from Scala, which is where it now lives. * ci: register CometUuidExpressionSuite in PR build workflows The check-suites.py preflight requires every *Suite.scala to be listed in both pr_build_linux.yml and pr_build_macos.yml. Add the new uuid suite to the expressions bucket.

fix: disambiguate Iceberg scans that share a metadata_location (#5180)

2026-07-30

fix: throw ARITHMETIC_OVERFLOW for Long.MinValue div -1 under ANSI mode (#5084) (#5146) * fix: throw ARITHMETIC_OVERFLOW for Long.MinValue div -1 under ANSI mode * refactor: reuse Spark's checkDivideOverflow and dedupe overflow check * test: use ARITHMETIC_OVERFLOW condition name in integral divide expect_error Matches the convention used by the other expect_error queries in arithmetic_ansi.sql. Verified against the spark-3.4, spark-3.5, spark-4.0, spark-4.1, and spark-4.2 profiles. (cherry picked from commit 38c3f8bce35642479117ba49f44a02fee31e6158)

feat: support SampleExec natively for sampling without replacement (#5110) (#5144) * feat: support SampleExec natively for sampling without replacement Adds a native Comet operator for Spark's SampleExec when sampling is performed without replacement, covering DataFrame.sample, SQL TABLESAMPLE, and DataFrame.randomSplit. The native operator ports Spark's BernoulliCellSampler on top of the existing XorShiftRandom, drawing one value per row and seeding per partition with seed + partitionIndex, so it selects the same rows as Spark for a given seed. Sampling with replacement reports Unsupported and falls back to Spark. * refactor: address review feedback on native sample operator - Build the selection mask with BooleanBuffer::collect_bool, which packs bits without allocating an all-ones null buffer - Precompute the empty-range check in BernoulliCellSampler instead of repeating it per row, and mark sample() inline - Drop the redundant schema() override and the single-use sample_batch helper - Use checkSparkAnswerAndFallbackReason in the fallback tests so they assert the reason, not just the absence of the Comet operator - Trim documentation that restated the same paragraph in four places - Point the contributor guide at the directory operator serdes actually live in * fix: shim the SampleExec seed for Spark 4.2 Spark 4.2 changed SampleExec.seed from Long to Option[Long] and resolves an absent seed into a resolvedSeed field, which is what the operator samples with. Read the seed through CometSampleShim, following the CometCollectShim precedent for the same 4.2 divergence. * test: expand sample operator test coverage and add benchmark Add tests for ordering preservation above a sort, sampling above an aggregate, SQL TABLESAMPLE, and empty batch interleaving in the native operator. Add a Sample case to CometExecBenchmark comparing against Spark. (cherry picked from commit 4b091acc9acc442c301754876348480f49b7321d)

feat: support SampleExec natively for sampling without replacement (#5110) * feat: support SampleExec natively for sampling without replacement Adds a native Comet operator for Spark's SampleExec when sampling is performed without replacement, covering DataFrame.sample, SQL TABLESAMPLE, and DataFrame.randomSplit. The native operator ports Spark's BernoulliCellSampler on top of the existing XorShiftRandom, drawing one value per row and seeding per partition with seed + partitionIndex, so it selects the same rows as Spark for a given seed. Sampling with replacement reports Unsupported and falls back to Spark. * refactor: address review feedback on native sample operator - Build the selection mask with BooleanBuffer::collect_bool, which packs bits without allocating an all-ones null buffer - Precompute the empty-range check in BernoulliCellSampler instead of repeating it per row, and mark sample() inline - Drop the redundant schema() override and the single-use sample_batch helper - Use checkSparkAnswerAndFallbackReason in the fallback tests so they assert the reason, not just the absence of the Comet operator - Trim documentation that restated the same paragraph in four places - Point the contributor guide at the directory operator serdes actually live in * fix: shim the SampleExec seed for Spark 4.2 Spark 4.2 changed SampleExec.seed from Long to Option[Long] and resolves an absent seed into a resolvedSeed field, which is what the operator samples with. Read the seed through CometSampleShim, following the CometCollectShim precedent for the same 4.2 divergence. * test: expand sample operator test coverage and add benchmark Add tests for ordering preservation above a sort, sampling above an aggregate, SQL TABLESAMPLE, and empty batch interleaving in the native operator. Add a Sample case to CometExecBenchmark comparing against Spark.

2026-07-29

fix: throw ARITHMETIC_OVERFLOW for Long.MinValue div -1 under ANSI mode (#5084) * fix: throw ARITHMETIC_OVERFLOW for Long.MinValue div -1 under ANSI mode * refactor: reuse Spark's checkDivideOverflow and dedupe overflow check * test: use ARITHMETIC_OVERFLOW condition name in integral divide expect_error Matches the convention used by the other expect_error queries in arithmetic_ansi.sql. Verified against the spark-3.4, spark-3.5, spark-4.0, spark-4.1, and spark-4.2 profiles.

2026-07-26

feat: native randstr implementation compatible with Spark (#5035) * feat: native randstr implementation compatible with Spark * refactor: gate randstr shape restrictions in getSupportLevel, avoid per-row UTF-8 scan Move the literal-length/literal-seed/non-negative-length restrictions from convert into getSupportLevel + getUnsupportedReasons so they surface in EXPLAIN and the compatibility docs, matching the rand/randn serde pattern. Build each string in a reused String buffer instead of validating a byte buffer per row. * test: add golden-value, partition-index, and filter coverage for randstr Address review feedback on the randstr PR: - Rust golden-value test asserting bit-for-bit equality with Spark 4.1.1 across positive, zero, and negative seeds, independent of the SQL tests. - Rust partition-index test exercising the seed + partition_index arithmetic. - Broaden the zero-length test across a wider seed set (seed is irrelevant when length is zero). - Rust large-length smoke test guarding the builder capacity math, and saturating_mul on the capacity hint to avoid overflow on huge lengths. - SQL test asserting an Int seed and its Long literal produce identical output (serde toLong sign extension). - SQL test running randstr through a native projection feeding a filter.

2026-07-25

feat: add CalendarIntervalType support (#4898) * Add CalendarIntervalType Arrow support * Support interval vectors in UDF and shuffle codegen * remove stale test * spotless apply

2026-07-24

feat: native collect_list / array_agg aggregate (#4720) * feat: native collect_list / array_agg aggregate Wires Spark's CollectList aggregate to datafusion-spark's SparkCollectList. array_agg, registered as a SQL alias of CollectList in FunctionRegistry, is also covered. Closes #2524. * fix: fall back collect_list/collect_set in multi-stage distinct aggregates A distinct aggregate combined with collect_list/collect_set produces a multi-stage plan (Partial -> PartialMerge -> Final). CollectList/CollectSet declare a BinaryType buffer in Spark but produce a native ArrayType state, so Comet cannot read a Spark-produced Binary buffer, nor round-trip its own ArrayType buffer across the intermediate PartialMerge stages. Both led to native crashes ("could not cast Binary to List" / "cast List to Binary"). Force these multi-stage aggregates to fall back to Spark consistently: - tag the feeding Partial when a PartialMerge stage of CollectList/CollectSet is present (CometExecRule.tagUnsafePartialAggregates), and - fall back a PartialMerge stage whose buffer was produced by a Spark partial (CometBaseAggregate.doConvert). Two-stage collect_list/collect_set continue to run natively. Patch the upstream SPARK-22223 plan-shape test to disable Comet, since native collect_list removes the ObjectHashAggregateExec it asserts on. Enabling fully-native multi-stage execution is tracked in #4724. * feat: fall back collect_list/collect_set on Spark 4.2 RESPECT NULLS Spark 4.2 adds an ignoreNulls field to CollectList and CollectSet, and collect_list(x) RESPECT NULLS sets it to false, keeping null elements. The native path delegates to SparkCollectList/SparkCollectSet, which always drop nulls, so it would silently return a different result from Spark. Add a per-version CometCollectShim that reads ignoreNulls (always true on Spark 3.4 through 4.1, where the field is absent) and fall back to Spark in getSupportLevel when it is false. Also rename QueryPlanSerde.hasIncompatibleBufferAgg to hasNativeArrayBufferAgg to describe what it detects: an aggregate whose native ArrayType state cannot round-trip Spark's declared BinaryType buffer. * test: accept Comet operators in SPARK-22223 instead of disabling Comet The SPARK-22223 ObjectHashAggregate test asserts on the executed plan. With Comet enabled, collect_list runs natively as CometHashAggregateExec, so ObjectHashAggregateExec is no longer present. Rather than disabling Comet, update the operator assertion to also accept CometHashAggregateExec, matching the pattern already used elsewhere in the diffs. The exchange assertion already matches ShuffleExchangeLike, which CometShuffleExchangeExec implements, so the single-shuffle check still holds. * style: reflow adjustOutputForNativeState doc comment Adding CollectList to the comment pushed a line past the column limit during the apache/main merge; reflow to satisfy spotless. * fix: coerce collect_list/collect_set nested field nullability collect_list and collect_set build their result list with all element fields marked nullable, but SparkCollectList/SparkCollectSet derive the return type from the child, preserving non-nullable nested fields. When the child is a nested type with a non-nullable inner field (e.g. a struct field built from non-nullable columns), the declared aggregate output disagrees with the array the accumulator produces, and the grouped native AggregateExec fails validating its output batch with "column types must match schema types". Cast the collect child to the all-nullable variant of its type so the declared and produced types stay consistent. * review: drop unreachable buffer-source block; consolidate 3.4/3.5 shim; add FILTER + map tests operators.scala: drop the CollectList/CollectSet PartialMerge fallback block. It is unreachable now that the general missingCometProducer + aggsNotSupportingMixedExecution guard just above returns None for the same case. CometExecRule.scala: add a comment on the collect-specific tagging block explaining it is separate from the tagging block just above because canAggregateBeConverted skips the child-native check, so an all-native distinct collect chain would otherwise slip past. QueryPlanSerde.scala: narrow the hasNativeArrayBufferAgg doc comment to describe what the code actually matches and note that Percentile has the same shape but is not matched here. Consolidate CometCollectShim: move the identical spark-3.4 and spark-3.5 copies to spark-3.x. Add collect_list FILTER (WHERE ...) test and a map-input test hitting make_all_fields_nullable's Map arm (size-only via spark_answer_only, since sort_array cannot order a MapType). * test: cover collect_list/collect_set RESPECT NULLS fallback on Spark 4.2 Add a Spark 4.2-gated SQL file test asserting that collect_list and collect_set with RESPECT NULLS fall back to Spark with Spark-identical results, and expand the getSupportLevel comments to explain that the fallback branch is only reachable on 4.2+ (a no-op on 3.4 through 4.1).

2026-07-23

feat: Iceberg table format V3: native table decryption, fall back for other V3 features (#4991)

perf: dedupe Iceberg residuals and delete files in native scan serde (#4982)

feat: support approx_count_distinct aggregate expression (#4819) * feat: support approx_count_distinct aggregate expression Add native support for Spark's approx_count_distinct, a faithful port of Spark's HyperLogLogPlusPlus / HyperLogLogPlusPlusHelper. Each non-null input is hashed with Comet's Spark-compatible XxHash64 (seed 42, floats normalized first), and the HyperLogLog++ registers are stored in Spark's exact packed-Long buffer layout (10 six-bit registers per word). The cardinality is estimated with the same linear-counting and bias-correction tables Spark uses, so results are bit-identical to Spark and the partial-aggregation state matches Spark's aggBufferSchema. Includes a vectorized GroupsAccumulator, SQL file tests comparing against Spark across a range of cardinalities, native unit tests, benchmark coverage, and documentation updates. * feat: address review feedback on approx_count_distinct - restrict decimal inputs to precision <= 18 (Spark hashes wider decimals through BigDecimal, which the native i128 path does not match) - set supportsMixedPartialFinal=true; the register buffer matches Spark's aggBufferSchema, enabling mixed Comet/Spark partial and final aggregation - add getUnsupportedReasons so the Compatibility Guide reflects the type limits - derive Hash, add assert/debug_assert invariants and a float-order comment, reuse the shared normalize_float, drop a redundant field, make bias correction lazy, and reuse a per-accumulator hash scratch buffer - compact BIAS_DATA layout with rustfmt::skip (values unchanged) - add tests: decimal boundary and wide-decimal fallback, non-UTC timestamp, collated-string fallback, and mixed partial/final plan shape

2026-07-22

feat: add spark.comet.shuffle.maxBufferBytes to cap native shuffle writer memory (#4989)

2026-07-17

feat: support gzip compression in native Parquet writes (#4930) * feat: support gzip compression in the native Parquet writer Add Gzip to the CompressionCodec proto enum and introduce a Parquet-specific ParquetCompression enum in parquet_writer.rs so the native writer can honor gzip without affecting the shuffle codec. The planner maps the new proto variant to ParquetCompression::Gzip, which writes with parquet's default GzipLevel (6), matching parquet-mr's zlib default. The shuffle writer path is unchanged and still rejects Gzip via its catch-all error arm. * feat: write Parquet natively when the compression codec is gzip * fix: request parquet-rs codec features explicitly for native Parquet writer The native writer's gzip, snappy, lz4, and zstd support only worked because Cargo feature unification happened to pull in parquet-rs's flate2/snap/lz4/zstd features via other workspace dependencies. Declare these features directly on the parquet dependency so the native writer does not silently lose codec support if that unification changes. * fix: honor parquet.compression precedence and uncompressed codec alias parseCompressionCodec only checked the compression option and the SQLConf default, skipping the parquet.compression option that Spark's own ParquetOptions treats as the middle rung of precedence. Fix the lookup to match Spark's compression, parquet.compression, SQLConf order, and accept "uncompressed" as an alias for "none" since Spark does the same. Add coverage for the parquet.compression option taking precedence over the SQLConf default, for uncompressed as a none alias, and for an unsupported codec (brotli) causing the write to fall back to Spark's own writer instead of CometNativeWriteExec. * refactor: inline single-call-site compression_to_parquet delegate compression_to_parquet was a one-line wrapper around ParquetCompression::to_parquet with a single caller; call it directly instead. * test: replace brittle brotli fallback test with lz4_raw round trip The unsupported-codec fallback test relied on Spark's write failing with ClassNotFoundException for BrotliCodec, so it only passed because the environment lacks a Brotli codec class rather than because Comet routed the write through the fallback path. Use lz4_raw instead, which Spark can write directly via parquet-mr and Comet does not support, so the test can assert a real successful round trip and the correct footer codec instead of intercepting an unrelated failure. Revert the allowFailure plumbing added to captureWritePlan for that test since it is no longer needed. * test: skip lz4_raw fallback test on Spark 3.4 Spark 3.4's PARQUET_COMPRESSION config validates against a fixed set of codecs (brotli, uncompressed, lz4, gzip, lzo, snappy, none, zstd) that does not include lz4_raw, so withSQLConf threw before the fallback path ran. Guard the test with assume(isSpark35Plus, ...) so it runs on versions where lz4_raw is a valid codec value. * test: cover full three-tier codec precedence for Parquet writes Adds a test that pins the `compression` > `parquet.compression` > `spark.sql.parquet.compression.codec` precedence: SQLConf is `zstd`, `parquet.compression` is `snappy`, and `compression` is `gzip`; the written file must report GZIP. A leak from either lower layer would surface as a codec mismatch. The existing test still covers `parquet.compression` beating the SQLConf default when `compression` is absent.

2026-07-15

chore: use Datafusion `substring` (#4161)

2026-07-09

feat: support approx_percentile / percentile_approx aggregate (#4801)

2026-07-06

chore: [branch-17] backport #4760 (#4829) * fix: size Iceberg delete files in the native scan to avoid dropping deletes (#4760) (cherry picked from commit b70e529ae945393ac24cb2647d560de1cd747f2a) * run prettier

2026-07-03

feat: support shuffle array expression (#4797) * feat: support shuffle array expression Add Comet support for Spark's `shuffle` array expression with full Spark compatibility. Comet reproduces Spark's exact random permutation by porting Apache Commons Math3's MersenneTwister and the inside-out Fisher-Yates algorithm from RandomIndicesGenerator, combining the resolved random seed with the partition index like Spark does. The native ShuffleExpr is a stateful PhysicalExpr that carries the RNG state across batches within a partition, mirroring the existing Rand/Randn implementation. * review: address feedback on shuffle expression - rename RNG to PRNG in doc comments - drop unreachable DataType::Null arm from ShuffleExpr::evaluate

2026-07-01

feat: support PreciseTimestampConversion for native batch time-window grouping (#4784) * feat: support PreciseTimestampConversion for native batch time-window grouping Wire Spark's internal PreciseTimestampConversion expression, which the analyzer emits when resolving window()/session_window() grouping. It is a pure reinterpret between the timestamp types and Long, so it maps to an Arrow cast between microsecond Timestamp and Int64. Also support the KnownNullable tagging expression that the window resolution wraps around window bounds. Together these let batch tumbling and sliding time-window aggregations run natively. * test: convert time-window tests to a Comet SQL file test; simplify serde Convert the window() coverage from Scala tests to a SQL file test at expressions/datetime/window.sql. The default query mode still asserts native execution (checkSparkAnswerAndOperator), and a ConfigMatrix over session timezones confirms timestamp-window results match Spark in every zone. Simplify CometPreciseTimestampConversion.convert to use the existing optExprWithFallbackReason helper instead of a hand-rolled if/else. * docs: point session_window support entry at dedicated issue #4785

feat: support interval types and make_ym_interval / make_dt_interval (#4541) * feat: support interval types and make_ym_interval / make_dt_interval [skip ci] Implements the type-support prerequisite from issue #4540: add Spark YearMonthIntervalType and DayTimeIntervalType as physical types that round-trip through Comet's Arrow FFI, and route the make_ym_interval / make_dt_interval constructors through the JVM codegen dispatcher so they execute natively and match Spark exactly. Type plumbing: - proto: add YEAR_MONTH_INTERVAL (18) and DAY_TIME_INTERVAL (19) to DataTypeId. - native serde.rs: map them to Arrow Interval(YearMonth) and Duration(Microsecond) respectively. DayTime stores microseconds in an int64, which matches Duration(Microsecond) rather than the lossy Interval(DayTime) {days, millis}. - Utils.toArrowType / fromArrowType: same mapping on the JVM side. - QueryPlanSerde.serializeDataType: emit the new type ids. - CometBatchKernelCodegen: accept the two interval types in isSupportedDataType and resolve IntervalYearVector / DurationVector; emit primitive set() writes. Expressions: - make_ym_interval -> CometMakeYMInterval, make_dt_interval -> CometMakeDTInterval, both via CometCodegenDispatch, registered in temporalExpressions. CalendarIntervalType and interval arithmetic remain follow-ups under #4540. Tests: SQL file tests for both constructors assert answer parity and native execution (checkSparkAnswerAndOperator), covering column and literal inputs, defaults, negatives, and nulls. * test: add overflow cases for make_ym_interval and make_dt_interval Confirm the codegen-dispatch path propagates Spark's arithmetic-overflow exception identically. The expect_error pattern uses the lowercase word overflow so it matches every Spark version: 4.x raises INTERVAL_ARITHMETIC_OVERFLOW while 3.x raises a raw ArithmeticException.

2026-06-30

feat: support exact percentile and median aggregates natively (#4542) * feat: support exact percentile aggregate natively [skip ci] Wire Spark's exact `Percentile` aggregate (and the `percentile_cont` ANSI form, which Spark rewrites to `Percentile`) to DataFusion's `percentile_cont` aggregate. DataFusion uses the same `index = p * (n - 1)` linear interpolation as Spark, so results match for the common single-percentage form. - proto: add `Percentile` AggExpr message (child, percentage, datatype). - native planner: map it to `percentile_cont_udaf()` with [child, percentile]. - CometPercentile serde: Compatible for a single literal double percentage, default frequency, and numeric input; the child is cast to double so the native result is DoubleType. Array-of-percentages, a non-default frequency argument, and interval inputs fall back to Spark. - operators.adjustOutputForNativeState: map Percentile's TypedImperativeAggregate Binary partial buffer to the native List<Float64> state (ArrayType(DoubleType)), mirroring CollectSet, so the partial/shuffle/final exchange schema is correct. Codegen dispatch is not applicable: aggregates (TypedImperativeAggregate) cannot run in the per-row scalar kernel, so native is the only path. Tests: SQL file test covering global, grouped, integer-input, all-null, exact and interpolated percentiles, plus fallback assertions for the array and frequency forms. No new regressions in the SQL suite. * bench: add percentile cases to CometAggregateExpressionBenchmark [skip ci] * feat: guard percentile DESC fallback, broaden tests and docs Address audit findings on the native percentile aggregate: - getSupportLevel now falls back for the descending WITHIN GROUP form (percentile_cont/disc WITHIN GROUP ... ORDER BY ... DESC on Spark 4.0+), where Percentile.reverse=true. The native percentile_cont always interpolates ascending, so the descending form would return a wrong answer. - Extract the fallback reason strings into shared private vals and add a getUnsupportedReasons override so they reach the compatibility guide. - Expand percentile.sql with long/float/decimal/smallint/tinyint inputs, negative values, and median() coverage (median rewrites to percentile). - Add percentile_within_group.sql (Spark 4.0+) covering the ascending native path and the descending fallback. - Mark percentile, percentile_cont, and median as supported in the expression reference and record the cross-version audit in agg_funcs.md. - Document the DataFusion 6-decimal interpolation quantization (#4719). * fix: mark percentile aggregates Incompatible by default (#4719) DataFusion's percentile_cont quantizes the linear interpolation weight to 6 decimal places, so a deeply-interpolated percentile can differ from Spark by up to roughly (upper - lower) * 1e-6. Gate the otherwise-supported percentile / median / percentile_cont form as Incompatible so it falls back to Spark by default and is opt-in via spark.comet.expression.Percentile.allowIncompatible=true. Add the allowIncompatible config to the percentile SQL file tests so the native path stays covered, and update the expression reference and audit doc to reflect the opt-in status. * style: wrap percentile precision comment under 100 chars

fix: size Iceberg delete files in the native scan to avoid dropping deletes (#4760)

2026-06-24

feat: extend native windows support (#4209)

2026-06-22

fix: reject Parquet INT96 as TimestampNTZ on Spark 3.x (#4357) * fix: reject Parquet TimestampLTZ as TimestampNTZ on Spark 3.x for native_datafusion Pre-Spark-4 (SPARK-36182) rejects reading a Parquet TimestampLTZ column as TimestampNTZ; native_datafusion previously did not, and silently returned the UTC instant. Plumb a per-Spark-version flag from ShimCometConf through the NativeScan proto into SparkParquetOptions, and gate a new rejection arm in the schema adapter on it. INT96 remains a gap because DataFusion's coerce_int96 strips the source timezone before the schema adapter runs, so it is indistinguishable from a true TIMESTAMP_NTZ source. Compatibility guide updated to describe the correctness implications. * style: cargo fmt * fix: reject Parquet INT96 as TimestampNTZ on Spark 3.x for native_datafusion Closes the INT96 gap left by the parent commit. INT96 columns previously surfaced as Timestamp(us, None) on the Rust side because DataFusion's coerce_int96 stripped the timezone, making them indistinguishable from a true TimestampNTZ source. With the new coerce_int96_tz option (DataFusion PR apache/datafusion#22318) we ask DataFusion to coerce INT96 to Timestamp(us, Some("UTC")), restoring the LTZ signal the schema adapter already pattern-matches against. Comet-side change is small: set coerce_int96_tz = "UTC" alongside the existing coerce_int96 = "us"; unskip the INT96 + native_datafusion variant of ParquetTimestampLtzAsNtzSuite. The schema adapter's existing Timestamp(_, Some(_)) -> Timestamp(_, None) rejection now fires for INT96 reads as well. EXPERIMENTAL: pinned via [patch.crates-io] to an andygrove/datafusion fork branch. Cannot be merged until apache/datafusion#22318 ships in a release.

2026-06-12

Revert "feat: Native Broadcast nested loop join support (#4429)" This reverts commit 4c88f5d4863f55c370b05cc104dcd0950b3ad2bd.

2026-06-11

feat: Native Broadcast nested loop join support (#4429) * native_support_broadcast_nested_loop_join

perf: replace CometBatchIterator FFI input path with the Arrow C Stream Interface (#4572)

2026-06-10

perf: avoid FFI import/export between native subtree and ShuffleWriter (#4507)

2026-05-28

feat: vendor-pluggable S3 credentials for native scans (#4309)

2026-05-20

feat: implement make_time and to_time (#4256)

2026-05-18

feat: add support for `posexplode` and `posexplode_outer` (#4270)

2026-05-17

fix: complete native_datafusion Parquet schema-mismatch rejections (#4229)

2026-05-08

feat: add JVM UDF framework for native execution (#4232)

2026-05-06

feat: support Parquet field ID matching in native_datafusion scan (#4216)

2026-05-05

fix: support Spark 4.1 BloomFilter V2 format and bit-scattering (#4196)

fix: [Spark 4.1.1] preserve parent struct nullness when all requested fields missing in Parquet (#4190)

2026-05-04

feat: support `PartialMerge` aggregation mode (#4003)

2026-05-03

feat: support regular BuildRight+LeftAnti hash join (#4073)

2026-04-22

feat: Support Spark expression: arrays_zip (#3643) * Define ArraysZip expr proto * Create ArraysZip SerDe * Register SerDe to arrayExpressions * Add SQL test * Register expression to planner * Rust wrapper around DF's arrays_zip * Null checks * Update supported Spark expressions doc

2026-04-20

feat: support `collect_set` (#3954)

2026-04-13

feat: Support Spark expression hours (#3804) * feat: Add Spark V2 partition transform `Hours` to calculate hours since epoch from timestamps.

2026-04-08

fix: Make cast string to timestamp compatible with Spark (#3884) * fix: Make cast string to timestamp compatible with Spark Add addtional formats and handle edge cases. Update compatibility guide Spark version specific behaviour for cast string to timestamp

2026-04-06

feat: support LEAD and LAG window functions with IGNORE NULLS (#3876) * feat: support LAG window function with IGNORE NULLS - Add ignore_nulls field to WindowExpr proto message - Serialize Lag window function with its ignoreNulls flag in CometWindowExec - Extend find_df_window_function to also look up WindowUDFs (not just AggregateUDFs) - Pass ignore_nulls to DataFusion's create_window_expr - Enable previously-ignored LAG tests Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: make LAG window tests deterministic by adding secondary sort key ORDER BY b alone has ties, causing Spark and DataFusion to produce different but both-valid row orderings. Add c as a secondary sort key so tie-breaking is deterministic and results are comparable. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: enable WindowExec allowIncompatible for LAG tests CometWindowExec is marked Incompatible by default. Add allowIncompatible=true config so LAG tests actually run via Comet and checkSparkAnswerAndOperator can verify native execution. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: skip partition/order validation for offset window functions LAG/LEAD (FrameLessOffsetWindowFunction) support arbitrary partition and order specs. The existing validatePartitionAndSortSpecsForWindowFunc check (which requires partition columns == order columns) is only needed for aggregate window functions, not offset functions. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * feat: add LEAD support and IGNORE NULLS test for LAG window function - Handle Lead case in windowExprToProto alongside Lag - Add comment explaining hasOnlyOffsetFunctions guard - Add test for LAG IGNORE NULLS - Enable LEAD tests with allowIncompatible config and deterministic ORDER BY Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * test: add LEAD with IGNORE NULLS test case Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * test: add SQL tests for LAG/LEAD window functions with IGNORE NULLS Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>

2026-03-30

feat: support SQL aggregate FILTER (WHERE ...) clause in native execution (#3835)

2026-03-21

perf: stop using FFI in native shuffle read path (#3731)

2026-03-07

feat: [ANSI] Ansi sql error messages (#3580) * feat: [ANSI] Ansi sql error messages

2026-02-27

chore: bump iceberg-rust dependency to latest [iceberg] (#3606)

2026-02-25

perf: Add Comet config for native Iceberg reader's data file concurrency (#3584)

2026-02-13

feat: CometNativeScan per-partition plan serde (#3511)

2026-02-08

feat: CometExecRDD supports per-partition plan data, reduce Iceberg native scan serialization, add DPP for Iceberg scans (#3349)

2026-02-07

Feat: to_csv (#3004)

2026-01-25

feat: Add support for round-robin partitioning in native shuffle (#3076)

2026-01-23

perf: [iceberg] Use protobuf instead of JSON to serialize Iceberg partition values (#3247) * perf: Use protobuf instead of JSON to serialize Iceberg partition values

Experimental: Native CSV files read (#3044)

2026-01-21

feat: Add support for `unix_timestamp` function (#2936)

2026-01-15

feat: Comet Writer should respect object store settings (#3042)

2025-12-29

feat: Support ANSI mode avg expr (int inputs) (#2817)

2025-12-20

feat: Add partial support for `from_json` (#2934)

2025-12-18

perf: [iceberg] Deduplicate serialized metadata for Iceberg native scan (#2933)

2025-12-16

feat: Make shuffle writer buffer size configurable (#2899)

2025-12-05

feat: Support ANSI mode SUM (Decimal types) (#2826)

feat: Add support for `explode` and `explode_outer` for array inputs (#2836)

2025-12-02

feat: Partially implement file commit protocol for native Parquet writes (#2828)

2025-11-26

feat: Add experimental support for native Parquet writes (#2812)

2025-11-20

feat: [iceberg] Native scan by serializing FileScanTasks to iceberg-rust (#2528)

2025-11-06

feat: Add support for `abs` (#2689)

2025-10-29

chore: Remove code for unpacking dictionaries prior to FilterExec (#2659)

2025-10-13

feat:support ansi mode rounding function (#2542)

2025-10-07

feat: Parquet Modular Encryption with Spark KMS for native readers (#2447)

2025-09-24

fix: distributed RangePartitioning bounds calculation with native shuffle (#2258)

2025-09-20

fix: regressions in `CometToPrettyStringSuite` (#2384) * Introduce BinaryOutputStyle from Spark 4.0 * Allow casting from binary to string * Pass binaryOutputStyle to query plan serde * Take binaryOutputStyle in planner * Implement Spark-style ToPrettyString * Match file name w/ test name * Test all 5 BinaryOutputStyle in Spark 4.0 * Fix package: 'org.apache.sql' -> 'org.apache.spark.sql' * Add CometToPrettyStringSuite back to CI * Specify binaryOutputStyle for Spark 3.4 * Let Comet deal with non pretty string casting * Enable binary to string casting test * Attempt to fix the build; ToPrettyString is Spark 3.5+ * Removed resolved issues * Type casting only function * Extract test setup logic to CometFuzzTestBase * Move binary_output_style proto <-> enum mapping to core * Move BinaryOutputStyle from cast.rs to lib.rs * Remove incorrect comments

2025-09-02

perf: Avoid FFI copy in `ScanExec` when reading data from exchanges (#2268)

2025-08-30

fix: split expr.proto file (new) (#2267) * split expr.proto file

2025-08-21

chore: Pass Spark configs to native `createPlan` (#2180)

2025-08-19

fix: [branch-0.9] Backport FFI fix (#2164)

2025-08-15

chore: Simplify approach to avoiding memory corruption due to buffer reuse (#2156)

2025-08-13

feat: limit with offset support (#2070) * feat: positive offset support for queries with limit * fix stability suite and spark tests * fix stability suite and spark tests * rolled back accidentally changed logic in a test * replace assert with explicit Err return * fix clippy warnings * Update native/core/src/execution/planner.rs Co-authored-by: Oleks V <comphead@users.noreply.github.com> * Update spark/src/main/scala/org/apache/spark/sql/comet/CometCollectLimitExec.scala Co-authored-by: Oleks V <comphead@users.noreply.github.com> * Update native/core/src/execution/planner.rs Co-authored-by: Oleks V <comphead@users.noreply.github.com> * fix compile errors and format warnings * zero offset and limit test --------- Co-authored-by: Oleks V <comphead@users.noreply.github.com>

2025-08-11

feat: Support Array Literal (#2057) * feat: support literal for ARRAY top level

2025-08-04

Chore: implement string_space as ScalarUDFImpl (#2041)

2025-07-30

feat: monotonically_increasing_id and spark_partition_id implementation (#2037)

2025-07-21

fix: Refactor arithmetic serde and fix correctness issues with EvalMode::TRY (#2018)

2025-07-18

feat: randn expression support (#2010)

2025-07-10

refactor: remove RightSemi and RightAnti join types from planner and proto (#1935) Co-authored-by: Andy Grove <agrove@apache.org>

2025-07-09

chore: use DF scalar functions for StartsWith, EndsWith, Contains, DF LikeExpr (#1887)

2025-06-25

feat: rand expression support (#1199)

2025-06-24

feat: Implement ToPrettyString (#1921)

2025-06-18

feat: support RangePartitioning with native shuffle (#1862)

2025-06-16

chore: Implement date_trunc as ScalarUDFImpl (#1880)

2025-06-03

feat: Translate Hadoop S3A configurations to object_store configurations (#1817)

2025-06-02

Chore: implement bit_not as ScalarUDFImpl (#1825) * implement bit_not as ScalarUDFImpl * Revert expr.proto --------- Co-authored-by: Kazantsev Maksim <mn.kazantsev@gmail.com>

2025-05-27

fix: [native_scans] Support `CASE_SENSITIVE` when reading Parquet (#1782)

2025-05-23

fix: default values for native_datafusion scan (#1756)

2025-05-09

perf: Add performance tracing capability (#1706)

2025-05-02

chore: Remove fast encoding option (#1703)

2025-03-25

Chore: simplify array related functions impl (#1490) ## Which issue does this PR close? Related to issue: https://github.com/apache/datafusion-comet/issues/1459 ## Rationale for this change Defined under Issue: https://github.com/apache/datafusion-comet/issues/1459 ## What changes are included in this PR? In functions related to arrays, scalarExprToProtoWithReturnType or scalarExprToProto is used instead of creating a separate proto for each function. ## How are these changes tested? Regression with available unit tests

2025-02-28

feat: Support IntegralDivide function (#1428) ## Which issue does this PR close? Closes #1422. ## Rationale for this change Support IntegralDivide function ## What changes are included in this PR? Since datafusion div operator conforms to the logic of intergal div, we only need to convert `IntegralDivide(...)` to `Cast(Divide(...), LongType)` and then convert it to native. ## How are these changes tested? added unit test