These commits are when the Protocol Buffers files have changed: (only the last 100 relevant commits are shown)
| Commit: | e85f42f | |
|---|---|---|
| Author: | Andy Grove | |
| Committer: | GitHub | |
feat: run filter and exists lambdas natively with DataFusion's higher-order functions (#7322) * feat: run filter lambdas natively with DataFusion's array_filter A filter lambda other than `x IS NOT NULL` went through the JVM codegen dispatcher, which evaluates it one element at a time like Spark does, plus the transfer of each batch. A body made of natively supported expressions now runs once over all the elements of a batch through DataFusion's array_filter. The native path applies to a one-parameter lambda over an array of a primitive, decimal, string, binary, date or timestamp element type. These stay on the dispatcher: nested lambdas, nondeterministic bodies, bodies that call out to the JVM, non-default string collations, and under ANSI mode a body that short-circuits (AND, OR, IN) around an operation that can raise, since the native operators evaluate every operand. The new spark.comet.exec.nativeLambda.enabled turns the native path off. Adds a lambda tuning section to the user guide and a benchmark. Closes #7321. * fix: keep a lambda parameter readable by short-circuited array function arguments NullShortCircuit evaluates the later arguments of a function such as array_contains on a projection of just the columns they read. It re-indexed only Column, so a lambda parameter passed as such an argument, as in filter(ints, x -> array_contains(other, x)), was dropped from the projection and the query failed. Treat LambdaVariable like Column when projecting, and like Column in case_when's infallible check. The JVM codegen dispatcher now declines a tree that reads a lambda parameter bound outside it. Its kernel is fed only columns, so the parameter read as NULL. The filter serde no longer walks the serialized body to find such a call. Also simplify the planner arm test, share assertNoCodegen between the codegen suites, and tidy the benchmark. * fix: decline the native filter lambda for wide captures and for bodies that can raise in any mode DataFusion copies each column a lambda captures once per element of the array, so a captured string, binary or nested column is copied as many times as the array has elements. A body that reads one now stays on the dispatcher. The check for a body that short-circuits (AND, OR, IN) no longer depends on the ANSI setting. It classifies each expression: arithmetic and casts by the eval mode they carry, and anything not known to be safe, such as element_at, which raises for the index 0 in any mode, as able to raise. The SQL tests pin the path of each query with expect_native and expect_dispatch, replacing the ConfigMatrix over the native lambda flag and the codegen suite tests, and cover tinyint and smallint elements. * feat: run nvl(x, '') != '' on a numeric array as array_compact On a numeric, date or timestamp array Spark plans the customer's rule filter(arr, x -> nvl(x, '') != '') as NOT (coalesce(CAST(x AS STRING), '') = ''), in either operand order. No such cast is empty, so the predicate holds exactly when x is not NULL, and the lambda is x IS NOT NULL. Send it down the array_compact fast path instead of casting every element to a string. On 2M rows of ten-element int arrays (every fifth element NULL), Spark 3.5.5 local[4], best of 5: 0.453 s on the dispatcher, 0.281 s with the native lambda, 0.086 s with this change, against 0.220 s for Spark. * feat: run exists lambdas natively with DataFusion's array_any_match CometArrayExists now serializes a lambda whose body Comet supports natively as a HigherOrderFunc under array_any_match, as filter does with array_filter, when three-valued logic is on. The gate and the serialization move into a trait both serdes share. exists stays on the dispatcher with spark.sql.legacy.followThreeValuedLogicInArrayExists=false, since Spark returns false where array_any_match returns NULL, and when the array argument contains a cast to a map, whose TRY cast can produce a null key that Arrow cannot hold (#6172). Also fixes a higher-order function inside an expression that is evaluated only for some rows: as the right operand of AND or OR, or inside an argument of a function like array_contains. These failed with 'project index N out of bounds', because project() in null_short_circuit.rs collected the function's own lambda parameter as a column of the outer batch. It now ignores variables bound by a lambda inside the expression it projects. This affected filter as well as exists. * bench: add exists cases to CometHigherOrderFunctionBenchmark and the tuning guide * fix: keep an exists lambda that can raise on the dispatcher Spark's exists stops at the first element that matches, so it never evaluates the lambda on the elements after it, while array_any_match evaluates the lambda on every element. With ANSI on, exists(ints, x -> 100 / x > 1) over array(1, 0) raised DIVIDE_BY_ZERO natively where Spark returns true, and with ANSI off element_at with the index 0 raised in the same way. exists now also requires that no expression in the lambda can raise, whatever operators it uses. Also from review: - Reuse CometCast.canProduceNullMapKey for the map-cast check, so a widening key cast in the array argument runs natively. - Make CometArrayExists a plain serde that calls the dispatcher itself, like CometArrayFilter, so the generated docs say Native, and give its row the same notes as filter. - Collect the projection in one walk in NullShortCircuit's project(). - Drop tests that repeat the SQL files: the exists test in CometCodegenHOFSuite, the array_any_match half of the planner test and the numeric nvl query in array_filter_native_lambda.sql. * fix: keep a native lambda body with a null map key or a date/time cast on the dispatcher A body that holds a TRY cast making a null map key cannot be built natively, as an argument with one cannot (#6172), and a cast between a date or timestamp and a string, date or timestamp fails natively for years beyond chrono's range, which Spark supports (#7471). Both now stay on the dispatcher, with a fixture that pins the path and the results. * fix: keep date functions out of native lambdas and let AND and OR stay native - A lambda body that applies a function to a date or timestamp stays on the dispatcher, since native date functions return NULL or fail beyond chrono's range (#6843, #7471). Comparisons, NULL tests, CASE/IF/COALESCE and casts to a number stay native. - Only IN needs the short-circuit check. Native AND and OR are LazyLogical and evaluate a right operand that can fail only on the rows the left leaves open. - mayRaise uses the same rules as #7529: numeric-only arithmetic, casts only between plain types, and GetStructField. - is_deterministic in lazy_logical.rs accepts a LambdaVariable. - The planner resolves the higher-order function through the session. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
The documentation is generated from this commit.
| Commit: | 2b6956a | |
|---|---|---|
| Author: | Ping Zhang | |
| Committer: | GitHub | |
feat: integrate MIN/MAX reader filtering with Spark 2/n (#7657) * feat: integrate MIN/MAX reader filtering with Spark * fix: exempt raw text fixture from archive license check * fix: scope MIN/MAX filter flags and metrics to partial aggregates * test: cover MIN/MAX pruning with nulls and integer limits
| Commit: | fb1d5ca | |
|---|---|---|
| Author: | Andy Grove | |
| Committer: | GitHub | |
fix: [branch-1.2] pick the signed zero Spark ranks first in native percentile (#6928) (#7513) * fix: pick the signed zero Spark ranks first in native percentile (#6928) * fix: pick the signed zero Spark ranks first in native percentile Spark keeps 0.0 and -0.0 as separate keys in the percentile buffer and ranks first the one its OpenHashMap iterates first, which depends on the table's capacity and collisions. When a group holds both zeros, replay Spark's partial, deserialized and final tables to pick the same sign. Pass whether the input was FLOAT so the replay hashes values as Spark does. Closes #6799. * fix: replay each partial buffer separately in native percentile Spark builds, serializes and deserializes one map per partition before merging them into the final buffer, so replaying all merged values as one partial can rank the other signed zero first. Keep the boundaries of the merged partial states and replay each one in turn. Add a two-partition regression test, and write the zero-tie fixtures as one file per insert. * fix: keep percentile partial boundaries in state and gate zero replay on Spark version - Store where each partial ends in the percentile state so a spilled and restored final aggregate replays the same partials. - Replay Spark's tables only when a requested rank lands on a zero. - Replay only on Spark 3.5.2+, which keeps 0.0 and -0.0 as separate keys (SPARK-45599); earlier releases keep the previous behavior, documented as a compatibility note. * fix: nest percentile partials per merge stage and track them only for signed zeros Spark's PartialMerge stages merge each incoming state into a fresh OpenHashMap, so the iteration order of signed-zero keys depends on how many merge stages a row passed through. Record the merge stage in the state (partial_depths) so the replay can nest partials the same way. Track partial boundaries only when the input is FLOAT/DOUBLE on a Spark that keeps both signed zeros as keys; otherwise the lists stay empty. Store the key set as packed bits and drop each partial's table once it has been merged. Document the two unmodeled paths (Spark's sort-based fallback and DataFusion early emit) in the code and the compat note. Remove the redundant unit tests and fix the stale fixture comment. * fix: attach the zero-order note to percentile's support level on Spark 3.5.2+ The serde returned a plain Compatible() for FLOAT and DOUBLE input on Spark versions that keep both signed zeros as keys, even though the replay does not model Spark's sort-based fallback or its aggregates emitting under memory pressure. Return the note with the support level, as the serde already does with signedZeroKeysNote on older Spark, so the limit is visible where the plan shape is accepted. * docs: name Comet's spill in percentile's zero-order note The note said "its own aggregates emitting or spilling", which reads as Spark's aggregates on the compatibility page. Name Comet explicitly and state that a spill inside a merge stage nests that stage's partials one level deeper than Spark does, which is the case where only Comet's answer can move. --------- Co-authored-by: Andy Grove <agrove@apache.org> (cherry picked from commit ca9f52ad5351967d5bbf51122312c86b91879ac1) * docs: regenerate the Percentile compatibility notes for 1.2 * docs: add the percentile compatibility notes to the Spark 4.2 page --------- Co-authored-by: Feng Zhang <fengzhang@apache.org> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
| Commit: | 4d98e5d | |
|---|---|---|
| Author: | Manu Zhang | |
| Committer: | GitHub | |
chore: remove remaining Spark 3.4 compatibility code (#7625) * chore: remove remaining Spark 3.4 compatibility code Co-authored-by: Codex <codex@openai.com> * fix: repair CI after Spark 3.4 cleanup Remove the unused scan-rule import, preserve table-schema metadata in the supported-version test fixture, and correct the source-scoped Scala 2.12 nowarn exception. Normalize the remaining Delta test import. Co-authored-by: Codex <codex@openai.com> * docs: align version guidance after Spark 3.4 removal Match Iceberg reflection documentation to the tested version matrix and drop Spark 3.4.3 from expression audit skills and their callers. Co-authored-by: Codex <codex@openai.com> * fix: finish Spark 3.4 cleanup after review Remove the unreachable JSON compatibility function and redundant version selector while retaining Jackson limits for supported Spark versions. Drop the redundant expected error-class alias and stale compatibility references. Co-authored-by: Codex <codex@openai.com> --------- Co-authored-by: Codex <codex@openai.com>
| Commit: | 24c40d6 | |
|---|---|---|
| Author: | Andy Grove | |
| Committer: | GitHub | |
fix: [branch-1.2] fall back when an outer hash join's streamed order cannot be checked (#7126), declare the input order of a native window over a sorted cache (#7256) and sort a Spark window's doubted input by the whole order it reports (#7555) (#7642) * fix: fall back when an outer hash join's streamed order cannot be checked (#7126) * fix: fall back when an outer hash join's streamed order cannot be checked An outer hash join that runs as DataFusion's right outer join keeps its unmatched probe rows in place only when the probe input declares an order. When the streamed side's reported order could not be checked, the serde declared none but still converted the join, so a sort Spark had already dropped above it was lost. This happened over a sorted localCheckpoint and over some caches written by an outer hash join. The join now falls back to Spark when that order reaches it through an input from the JVM or through a nested loop join. Over native operators in the same plan it stays native, since DataFusion keeps the order it can see there. * test: cover a right outer join over a cache of a right outer join The cache test now runs both shapes that run as DataFusion's right outer join, LeftOuter built on the right and RightOuter built on the left, in both cache formats. The nested loop join case notes that it only gives up the native join for nothing until #6898 is fixed. * fix: say that an outer hash join with an unchecked order stays in Spark The fallback reason still described the old behavior, where the join stayed native and declared no order. It now says the join stays in Spark, and the tests share it through one constant. The compatibility guide leads with when the join falls back and notes that a checkpoint reaches a native join only with spark.comet.convert.rdd.enabled. The two checkpoint tests share their setup. * docs: keep only the comments that explain when an outer join's order is trusted Drop comments that restate the code or a test's name, and shorten the rest to why an order from the JVM or a nested loop join can't be used. (cherry picked from commit c56f6a0209dba0600103a37fac7ca9b006639857) * fix: declare the input order of a native window over a sorted cache (#7256) * fix: declare the input order of a native window over a sorted cache Spark puts no sort below a window when its input reports the order the window needs, a relation sorted before it was cached for instance. When those rows enter the native plan from the JVM, the ScanExec they arrive through declares no order, and the native window failed the task with "Expects PARTITION BY expression to be ordered". Send the window's required ordering in a new input_sort_orders field on the Window proto when the child reports an order that satisfies it and mayReportFalseOrder trusts that order. The planner declares it on the window's input with with_declared_order, as the hash join does for its probe input. A key that Comet cannot express, or that holds a collated string, sends the window back to Spark. Closes #7223. * refactor: build the window's input order only where it is used - Send no input order when a native sort below the window declares its own. - Build the order only once the window's own expressions have serialized, so a window that falls back for another reason does not walk the plan or record an extra reason. - Map a collated key straight to None instead of signalling it with an empty list. - Drop the empty-list branch in WindowBuilder, as with_declared_order already leaves the plan unchanged for no orders. * chore: trim comments and drop a window test the hash join test already covers * test: cover WindowAggExec, nulls, descending keys and the collated fallback From review: - The cache test also runs a frame up to UNBOUNDED FOLLOWING, which runs in WindowAggExec, over a nullable partition key and a descending order key, so the declared order has to match the cache's. - The planner test covers both window operators through the partition order lookup they share, since WindowAggExec fails only when its stream starts, not in partition_by_sort_keys. - A Spark 4.x test checks that a window over a cache sorted by a collated key stays in Spark for the new reason and returns Spark's rows. - The debugging guide's window snippet builds the operators over `input`. * fix: keep a stacked window's longer input order and declare it on empty buckets The window now sends the rest of the order its child reports past the required keys, up to the first that cannot be expressed, so a window above it in the same native plan sees the longer order. An empty bucket of a sorted bucketed scan is planned as an EmptyExec, which now counts as a source whose order holds. The cache test checks the declared columns, directions and null placements. * fix: declare a window's order next to a partial one its input declares Two aliases of one sorted column share an order in Spark, so a window over the window for one alias may need the other alias's order with no sort in between. with_declared_order now returns early only when the input's declared order already satisfies the one sent, and otherwise declares it next to the input's when keeps_source_order trusts the input. The serde checks only a suffix key's child against the child's output, since its same-order expressions are not sent and may name pruned columns. * refactor: send only a window's required order and merge overlapping tests with_declared_order extends a partial order at the upper window, so the serde no longer sends the keys a window's child reports past the ones the window requires. The two stacked-window tests become one over a single cache, the EmptyExec unit test folds into the extension test, and redundant assertions and comments are trimmed. * fix: declare only a window's partition keys and fall back on an unchecked order The native window checks only its partition keys' order, so the serde sends only those, and a float order key that Spark sorts with -0.0 and 0.0 as equal is never declared. When the order the child reports may be false past the native plan, the window stays in Spark, as the outer hash join does, instead of failing the task. The SortedInputExec doc and the operator guide now cover the window. * fix: keep a window's doubted order native, fall back on collated keys Falling back when the window's input order may be false handed the same rows to Spark's window, which trusts that order too and grouped them wrongly. The window again sends no order there, so the native window rejects its input as on main. A window whose partition or order key holds a collated string now stays in Spark, as WindowGroupLimit does, since the native window compares keys byte by byte. keeps_source_order accepts a right outer hash join whose probe declares an order, which is when DataFusion keeps its unmatched probe rows in place. * fix: fall back on a collated window key only over a trusted order Falling back hands the rows to Spark's window, which trusts the order the input reports. Over an order mayReportFalseOrder doubts, such as a native nested loop join's, that grouped the rows wrongly, so the window now stays native there, as on main, and the native window rejects the input. The outer join regression test partitions by k, which Spark reports through the alias a, so it needs the order declared next to the join's. * fix: sort the input of a Spark window over a doubted order Spark plans no sort below a window whose child reports the order it needs, and its window trusts that order. Over a Comet operator that may not keep it, such as a native nested loop join (#6898), a window that stays in Spark grouped rows out of order. CometExecRule now sorts that input again. With that, a window with a collated key falls back in every case, and so does a window with ORDER BY but no PARTITION BY over a doubted order, since the native window checks no order there. The compatibility guide lists collated keys among the window's fallbacks. * address review: run the nested loop join window test with AQE on, sort Python windows too (cherry picked from commit cd19831a7ea9130aae7b95882810e9b050a13aa7) * fix: sort a Spark window's doubted input by the whole order it reports (#7555) sortDoubtedWindowInputs sorted the input of a window that stays in Spark by the window's own keys only. Spark had already planned the operators above around the longer order the input reported, so a second window ordered by more keys, or a sortWithinPartitions it had removed, saw rows out of order. The sort now uses the whole reported order when it covers the window's keys. Closes #7553. (cherry picked from commit fd82542adf69b64ed20ecb82c73493b36273c8e4) --------- Co-authored-by: dustin <dwsmith1983@users.noreply.github.com> Co-authored-by: Oleks V <comphead@users.noreply.github.com>
| Commit: | 1cb614b | |
|---|---|---|
| Author: | Parth Chandra | |
| Committer: | GitHub | |
fix: ship Iceberg scan pool entries per partition so task memory does not grow with the table (#7491) * fix: ship Iceberg scan pool entries per partition so task memory does not grow with the table Native Iceberg scans put the whole scan's delete files, delete-file paths and sets, partition values and residuals in the common block. Every task deserialized it from the task binary, and the injector copied it into each partition's native plan, so tasks on large merge-on-read tables ran out of heap. Each partition now carries only the pool entries its tasks use, in IcebergScan.own_pools (a new IcebergFilePools message), and the common block keeps only the small pools and settings. The driver builds each partition's message lazily, while Spark serializes its task (PartitionPlanData / PlanDataSlice), so it never holds every partition's message at once. The executor merges the partition's pools into the common block before planning, and native rejects a plan that still carries own_pools. A coalesce without a shuffle repeats shared delete files once per input partition; this is documented and pinned by a test. * fix: keep Iceberg pool entries every partition uses in the common block, and hold driver pools as bytes Entries that every non-empty partition uses, such as an unpartitioned equality delete, now ship once in the common block instead of in every task. Every task needs them anyway, so executors pay nothing extra, and tasks no longer grow past spark.rpc.message.maxSize when many shared delete files exist. Partition messages index the common entries first, then their own. After planning, the driver keeps the full pools as the serialized bytes plus an offset per entry, and parses only the entries a task uses when that task is serialized. The parsed message took about 3.9 times its serialized size; bytes plus the index take about 1.1 times. Adds coalesce queries to the merge-on-read SQL test. * fix: keep Iceberg pool entries that at least half of the partitions use in the common block An equality delete applies only to older data files, so newer appends, or a bucket with no deletes, leave some partitions without it. Requiring every partition to use an entry sent such deletes back into every task. Entries used by at least half of the non-empty partitions now ship once in the common block. Every entry there is needed by at least half of the tasks, so the common block is at most twice what an average partition's tasks need. * style: drop an interpolator with no placeholders * refactor: intern Iceberg per-file pools per partition in the existing serde pass Review feedback: build each partition's pools in the pass that already interns them, and ship them in the partition's existing IcebergScan.common, so the injector merges the two with mergeFrom. This drops the global-then-split step (IcebergScanPools, Layout, SerializedPools), the IcebergFilePools message and own_pools field, and the half-of-partitions threshold: the common block now holds none of the per-file pools. The lazy driver-side build (PartitionPlanData, PlanDataSlice) moves out of this PR, so CometExecRDD, CometNativeShuffleInputRDD, NativeExecContext, CometNativeScanExec and the Delta contrib scan are unchanged from main. * docs: note that the driver keeps a copy of a shared delete file per partition * fix: share Iceberg delete files between partitions so a coalesced task carries each once Review feedback: a coalesce without a shuffle writes several input partitions into one task, and each partition held its own copy of a shared delete file, so the task carried one copy per input partition where the latest release carried one. Per-partition plan data is now a PartitionPlanData object instead of bytes. An Iceberg partition holds its own message plus references to delete files serialized once for the whole scan, so Java serialization writes a shared delete file once per task stream and the driver keeps one copy. The executor sets each delete file's path index when it builds the partition's message, since the path pool order differs per partition. Native Parquet and contrib scans wrap their bytes, so the CometScanWithPlanData trait is unchanged. Move the Iceberg pool test helpers into IcebergScanPlanData, with distinctResiduals for tests that read the residual pool. * style: drop an interpolator with no placeholders * fix: share Iceberg residuals between partitions, and check a real coalesce Review feedback: a residual that applies to every file, such as a large IN list, was copied into each partition's message, so the driver held one copy per partition and a coalesced task carried one per input partition. The release held one, in the common block. Residuals are now serialized once per scan and shared by reference like delete files. The executor appends them to the partition's residual pool, so residual_idx stays partition-local. The global equality-delete test now runs a real coalesce(1) and checks that its one task carries the shared delete file once. * fix: name residuals in the pool comment, and drop an implicit Int to Long widening in a test
| Commit: | 36ab96a | |
|---|---|---|
| Author: | Steve Vaughan | |
| Committer: | GitHub | |
fix: match SPARK-60119 when a string casts to a negative-scale decimal (#7522) Closes #7187. Spark 4.0.5, 4.1.4 and 4.2.1 include SPARK-60119, which makes `Decimal.changePrecision` skip `BigDecimal.setScale` when the value's magnitude already decides the result. With `spark.sql.legacy.allowNegativeScaleOfDecimal=true`, `'1e-2147483647'` cast to `DECIMAL(10,-2)` now returns 0 in every eval mode, where earlier releases throw `ArithmeticException`. The native cast still returned NULL, or raised `CAST_INVALID_INPUT` under ANSI, because `parse_string_to_decimal` rejects a scale difference that does not fit in an int. `SparkFixes.castsTinyStringToDecimalZero` tells by the Spark version whether Spark has the fix, as `castsZeroWithAnyExponentToDecimal` does for SPARK-60118. `CometCast` passes it as the new `decimal_tiny_to_zero` cast option, and `parse_string_to_decimal` returns 0 when it is set and the difference is below the int range. A difference above the int range is a value too large for any precision and stays an error, and so does the cast on earlier releases. The two `CometNativeCastSuite` tests that expect Spark to fail on this input now run only where it does, and a new test covers the releases with the fix in ANSI and non-ANSI mode.
| Commit: | bb336d8 | |
|---|---|---|
| Author: | KUAN-HAO HUANG | |
| Committer: | GitHub | |
feat: support map inputs for explode (#6107) * feat: support map inputs for explode * test: recognize missing Parquet field IDs through nested causes * test: strengthen native map explode coverage and benchmarks Cover JVM-fed maps, nullable Boolean batch boundaries, all-NULL/empty outer batches, native map producers, and stored floating-point keys. Require native generators and matching Spark results and schemas. Benchmark map variants and nested map/array inputs with sinks that consume keys and positions. Remove the unrelated Parquet field-ID assertion change so it can be reviewed independently. * perf: avoid unused allocations when unnesting collections Keep input arrays as temporary placeholders until generated columns replace them. Build take indices only when a carried column consumes them. Cover recursive unnest with and without carried columns and both drop and outer null handling. Preserve the existing bounded map and array execution path. * perf: repeat carried short strings by input row in explode * fix: bound map expansion before repeating carried values Split individual List rows before materialization and cap repetition so carried buffers and child offsets stay within Arrow's offset limits. Preserve shared entry buffers, positions and outer semantics. * perf: repeat carried values by input runs in explode Fill non-null primitive runs directly for expansions of at least five and build short-string offsets with one Vec extension per run. Retain Arrow take for smaller expansions and nullable primitives. * fix: retain Spark execution for chained map generators Native generators materialize carried columns between expansions and were slower than Spark's pipeline in the nested benchmark. Keep maps followed by another generator on Spark across projections and filters, while retaining native single maps and chains separated by an exchange. * fix: remove redundant interpolation in explode benchmark * fix: bound map explode chunks by visible offset spans Keep normal batches intact and skip offset checks when repetition is already safe. Remove the custom carried-value kernels, retain Arrow take, and avoid unused allocations when no columns are carried. Restore array and map benchmark coverage and consolidate map SQL cases. * fix: trim carried list children before explode take Sliced list and map arrays retain their full child arrays, which inflates Arrow take's allocation estimate. Trim the referenced child span before taking carried columns, preserving nested metadata and validity. * perf: reuse shared list lengths for positional explode * fix: account for actual carried byte repetitions * fix: keep chained map generators on the native path Remove the map-only planner fallback so maps and arrays share the bounded native explode path. Cover both chain directions across all generator variants and AQE, add the reverse-chain benchmark, and update the SQL expectation and support notes. * test: bound explode offset regression allocations Replace the 2 GiB carried-string stream test with real-boundary helper checks and small-limit stream cases for Utf8, Binary and Struct values. Reuse private stream construction and pass its offset limit through the existing chunk guards. Check batching, output values and retained buffers, including skewed rows that should remain in one batch.
| Commit: | f4a6401 | |
|---|---|---|
| Author: | Parth Chandra | |
| Committer: | GitHub | |
feat: txt file support (#6784) * feat: txt file support * address review comments * more comments addressed * address second review round: BOM-vs-lineSep, per-read hadoop options, multi-bucket text gate - text_scan.rs: strip a leading UTF-8 BOM only with the default separator (a custom lineSep leaves the BOM as data), matching Spark; update the unit test - CometTextNativeScanExec / CometCsvNativeScanExec: build the object-store Hadoop conf from the scan's read options (case preserved) so per-read settings like fs.s3a.endpoint reach native - CometScanRule: decline native Text scans whose roots span more than one scheme/bucket (RootPathIssues.crossLocations); Parquet gate behavior unchanged - tests: add .zstd fallback, BOM-vs-lineSep and BOM-only parity tests - docs: fix the non-UTF-8 compatibility link target * fmt * fix: use shared V2 input_file_name fallback message in text scan The Text branch had its own input_file_name/_block guard that emitted a Native Text scan ... message, but the blanket V2 gate at the top of transformV2Scan already handles these expressions and emits Native V2 scan ... first. The Text message was dead, and the test asserted it, so the scans suite failed on all five Spark profiles. Drop the redundant guard and assert the shared message. * fix: fall back for self-overlapping text line separators Spark splits a custom lineSep with Hadoop LineReader.readCustomLine, which backtracks only one byte on a mismatch instead of following the separator's own prefix overlap. For a separator that overlaps itself (e.g. aab, whose aa prefix can restart a match) that reader can split differently from the native reader's exact substring scan. Rather than reproduce Hadoop's buffered matcher, fall back at plan time when the separator self-overlaps (KMP prefix function has a nonzero entry); overlap-free separators stay native. Guarded to line mode since wholetext ignores the separator. Added a test with lineSep=aab. * fix: fall back to the wrapped Spark scan when native text conversion is declined When the Text serializer declines a scan (for example files spread over two ABFS containers), CometExecRule kept the CometBatchScanExec, which advertises columnar output while Spark's text reader produces UnsafeRow, so the query failed with a ClassCastException. Return the wrapped scan like the CSV branch does, and add Text tests for the ABFS decline layouts. * fix: strip the leading BOM from the first record for custom line separators too Hadoop's LineRecordReader reads the first record and then drops a leading UTF-8 BOM from it, whatever the separator. The earlier fix skipped the strip for every custom separator, so a file like BOM a|b with lineSep=| returned BOM+a, and a BOM-only file returned one row instead of none. Split first, then strip the BOM from the first record only. A file that is only the BOM has no rows; BOM followed by a separator leaves an empty row; lineSep equal to the BOM is unaffected because the BOM is then the separator. Add Rust unit tests and Scala tests for BOM a|b, BOM|x, and a BOM-only file with lineSep=|. * fix: build the text gate's Hadoop conf from the case-preserving option map The Text eligibility gate built its Hadoop conf from scan.options.asScala, which lowercases keys. Hadoop keys are case sensitive and Spark's TextScan keeps their case, so a differently cased read option such as ...line.maxLength did not override the Hadoop conf limit in Spark, but the gate saw the option as the lowercase key, read an unlimited length, and admitted native execution, which returned the long line Spark skips. Use asCaseSensitiveMap, matching TextScan.createReaderFactory and the serializer, and add a test. * fix: address text scan review findings on file selection, gates, metrics - Resolve the codec and check files on the selected partitions' decoded paths, not fileIndex.inputFiles (URL-encoded, and including pruned files). - Fall back for local files that have a Hadoop .crc checksum file. - Run the scheme and object_store path checks on the selected files, not only the roots. - Fall back for mapreduce.job.input.file.must.* options in line mode. - Read line.maxlength only in line mode, and fall back instead of failing when it is not an integer. - Decline CSV scans whose libhdfs scheme is set only by a read option. - Canonicalize the Text scan's output and wrapped scan so repeated lookups compare equal. - Report bytes_scanned from the native opener and expose it with output_rows so the task input metrics are filled in. * style: drop the redundant Eventually import from the text suite * read text lines on demand and build the codec factory once The native reader split the whole file into a Vec of slices before returning the first batch, one 16-byte descriptor per line. A file of newlines needs a descriptor per byte, so a 64 MiB file reached 1 GiB, including for count(*). Find lines by scanning forward from a read offset instead, and build each batch when the stream asks for it, so only the file bytes and the current batch are held. The BOM, byte-range and wholetext rules apply as lines are found. The compression check built a CompressionCodecFactory for every selected file on Spark 3.4, 3.5 and 4.0. Build the checker once per scan through ShimFileFormat.compressedFileCheck. * docs: list the native Text scan in Supported Spark Operators * test: pin the self-overlapping lineSep fallback for a separator at an input buffer boundary * refactor: address comphead's review of the native text scan - planner: share the partition lookup, store and file group between the CSV and Text arms (scan_task_files), and build the store through scan_store so a plan made without data needs no storage credentials. Drop the unused partition schema from TextScan (tag 2 is reserved). - text_scan: remove the decompress arm, the byte-range filter and the offset-0 flag, which the Scala gate makes unreachable (it declines compressed and split files), and the tests of that filter. Append decoded values straight into the StringBuilder instead of collecting them first. - gate: build the selected file list once and use it for the split, size, compression, checksum and path checks. The wholetext cap is now min(maxFileSize, Int.MaxValue/3), so lowering maxPartitionBytes no longer disables native wholetext. - serde: move the format-independent part of the CSV and Text convert methods into CometScanUtils.prepareNativeFileScan, and give the CSV node the same canonical form as the Text node so repeated CSV scans compare equal. * test: pin that an empty wholetext file yields no rows, as in Spark Spark's file index (PartitioningAwareFileIndex.listFiles) drops files with length 0 before a scan is planned, so wholetext on an empty file returns no row, not one empty row. The new test compared the two engines and then asserted one row, which failed; assert none instead. Also fix a comment in the wholetext directory test that claimed the empty file gave a row, and add Rust tests that run a file through DataFusion's real scan (the planner's 0..0 range for an empty file included).
| Commit: | e59d937 | |
|---|---|---|
| Author: | Steve Vaughan | |
| Committer: | GitHub | |
fix: raise NUMERIC_OUT_OF_SUPPORTED_RANGE for a string with more than 38 integral digits cast to decimal (#7578) * fix: raise NUMERIC_OUT_OF_SUPPORTED_RANGE for a string with more than 38 integral digits cast to decimal Spark rejects a string whose integral part has more than 38 digits as soon as it has parsed it, before it checks the value against the target type. That is NUMERIC_OUT_OF_SUPPORTED_RANGE under ANSI, whatever the target type, and NULL otherwise. Comet returned the same NULL but reported another error under ANSI, because its parser failed first: CAST_INVALID_INPUT when the digits or the scale adjustment did not fit an i128, and NUMERIC_VALUE_OUT_OF_RANGE for 39 digits that did. parse_decimal_str now applies the limit as soon as it has the scale, before those errors, for strings of any length. The digits are counted only when an upper bound says the string may be over the limit, so the common case costs one comparison. With spark.sql.legacy.allowNegativeScaleOfDecimal Spark skips the limit, for zero too. CometCast sends that in the new decimal_skip_integral_digit_limit cast option, and parse_string_to_decimal skips the limit when it is set. * docs: say which strings still report CAST_INVALID_INPUT with the legacy decimal setting With spark.sql.legacy.allowNegativeScaleOfDecimal Spark skips the 38 integral-digit limit, and the value then fails the target type's precision check. Comet reports that as NUMERIC_VALUE_OUT_OF_RANGE only for a string that fits an i128 with a scale adjustment of at most 38. Under ANSI, a scale adjustment over 38 and more digits than an i128 holds still report CAST_INVALID_INPUT. The doc comment on parse_string_to_decimal, the audit entry and cast.md said more than the code does. They now name the two shapes and the issues that track them (#7576 and #7581), and the tests pin the error Comet gives today.
| Commit: | cd19831 | |
|---|---|---|
| Author: | Oleks V | |
| Committer: | GitHub | |
fix: declare the input order of a native window over a sorted cache (#7256) * fix: declare the input order of a native window over a sorted cache Spark puts no sort below a window when its input reports the order the window needs, a relation sorted before it was cached for instance. When those rows enter the native plan from the JVM, the ScanExec they arrive through declares no order, and the native window failed the task with "Expects PARTITION BY expression to be ordered". Send the window's required ordering in a new input_sort_orders field on the Window proto when the child reports an order that satisfies it and mayReportFalseOrder trusts that order. The planner declares it on the window's input with with_declared_order, as the hash join does for its probe input. A key that Comet cannot express, or that holds a collated string, sends the window back to Spark. Closes #7223. * refactor: build the window's input order only where it is used - Send no input order when a native sort below the window declares its own. - Build the order only once the window's own expressions have serialized, so a window that falls back for another reason does not walk the plan or record an extra reason. - Map a collated key straight to None instead of signalling it with an empty list. - Drop the empty-list branch in WindowBuilder, as with_declared_order already leaves the plan unchanged for no orders. * chore: trim comments and drop a window test the hash join test already covers * test: cover WindowAggExec, nulls, descending keys and the collated fallback From review: - The cache test also runs a frame up to UNBOUNDED FOLLOWING, which runs in WindowAggExec, over a nullable partition key and a descending order key, so the declared order has to match the cache's. - The planner test covers both window operators through the partition order lookup they share, since WindowAggExec fails only when its stream starts, not in partition_by_sort_keys. - A Spark 4.x test checks that a window over a cache sorted by a collated key stays in Spark for the new reason and returns Spark's rows. - The debugging guide's window snippet builds the operators over `input`. * fix: keep a stacked window's longer input order and declare it on empty buckets The window now sends the rest of the order its child reports past the required keys, up to the first that cannot be expressed, so a window above it in the same native plan sees the longer order. An empty bucket of a sorted bucketed scan is planned as an EmptyExec, which now counts as a source whose order holds. The cache test checks the declared columns, directions and null placements. * fix: declare a window's order next to a partial one its input declares Two aliases of one sorted column share an order in Spark, so a window over the window for one alias may need the other alias's order with no sort in between. with_declared_order now returns early only when the input's declared order already satisfies the one sent, and otherwise declares it next to the input's when keeps_source_order trusts the input. The serde checks only a suffix key's child against the child's output, since its same-order expressions are not sent and may name pruned columns. * refactor: send only a window's required order and merge overlapping tests with_declared_order extends a partial order at the upper window, so the serde no longer sends the keys a window's child reports past the ones the window requires. The two stacked-window tests become one over a single cache, the EmptyExec unit test folds into the extension test, and redundant assertions and comments are trimmed. * fix: declare only a window's partition keys and fall back on an unchecked order The native window checks only its partition keys' order, so the serde sends only those, and a float order key that Spark sorts with -0.0 and 0.0 as equal is never declared. When the order the child reports may be false past the native plan, the window stays in Spark, as the outer hash join does, instead of failing the task. The SortedInputExec doc and the operator guide now cover the window. * fix: keep a window's doubted order native, fall back on collated keys Falling back when the window's input order may be false handed the same rows to Spark's window, which trusts that order too and grouped them wrongly. The window again sends no order there, so the native window rejects its input as on main. A window whose partition or order key holds a collated string now stays in Spark, as WindowGroupLimit does, since the native window compares keys byte by byte. keeps_source_order accepts a right outer hash join whose probe declares an order, which is when DataFusion keeps its unmatched probe rows in place. * fix: fall back on a collated window key only over a trusted order Falling back hands the rows to Spark's window, which trusts the order the input reports. Over an order mayReportFalseOrder doubts, such as a native nested loop join's, that grouped the rows wrongly, so the window now stays native there, as on main, and the native window rejects the input. The outer join regression test partitions by k, which Spark reports through the alias a, so it needs the order declared next to the join's. * fix: sort the input of a Spark window over a doubted order Spark plans no sort below a window whose child reports the order it needs, and its window trusts that order. Over a Comet operator that may not keep it, such as a native nested loop join (#6898), a window that stays in Spark grouped rows out of order. CometExecRule now sorts that input again. With that, a window with a collated key falls back in every case, and so does a window with ORDER BY but no PARTITION BY over a doubted order, since the native window checks no order there. The compatibility guide lists collated keys among the window's fallbacks. * address review: run the nested loop join window test with AQE on, sort Python windows too
| Commit: | b82f5ea | |
|---|---|---|
| Author: | Bhargava Vadlamani | |
| Committer: | GitHub | |
feat: add support for comet cartesian join (#6938) * add support for comet cartesian join * address review: make native cartesian product opt-in (default false) DataFusion's CrossJoinExec buffers the whole left input and cannot spill, so a large left can OOM where Spark's spillable CartesianProduct succeeds (sunchao's P2). Default the flag to false until the native path can spill; add a regression test that a cross join stays on Spark by default. Part of #199. * add benchmarks for comet cartesian join * add benchmarks for comet cartesian join move ests to sql file * update documentation per review * update documentation per review * update documentation abd tests per review * update documentation per review
| Commit: | 97ad03b | |
|---|---|---|
| Author: | Andy Grove | |
| Committer: | GitHub | |
fix: write the Spark version into the footer of natively written Parquet files (#7053) * fix: write the Spark version into the footer of natively written Parquet files * docs: note that natively written Parquet files now carry the Spark version footer key The native writer records org.apache.spark.version, so Spark reads the files back without rebasing. Reword the rebase-modes known issue to what is still true: the write rebase modes are ignored and no legacyDateTime key is written.
| Commit: | 79b5ddd | |
|---|---|---|
| Author: | Andy Grove | |
| Committer: | GitHub | |
feat: add comet-scheduler and comet-executor for Comet Native Executors (#7299) * feat: add the Comet Native Executors wire protocol A Spark driver submits a query to a Comet Native Executor cluster as a QueryRequest: a stage graph of Comet native blocks, connected by the exchanges between them, with each stage's Spark output schema and, for stages that read files, each partition's plan. A Fragment carries a stage from the scheduler to its executors. The proto crate also exports a fingerprint of the .proto files, so that a cluster can refuse a query from a client that speaks another plan format. See #7288. * feat: run Comet native blocks as DataFusion operators outside Spark CometFragmentExec runs a stage's native block for each of its partitions, with the block's input leaves fed by other DataFusion plans, such as a distributed engine's shuffle readers, instead of by the JVM. PhysicalPlanner::with_native_inputs is the seam: an input leaf (Scan or ShuffleScan) takes the next native input, cast to the types Spark declared for it, and an expression that calls into the JVM fails to plan. Each partition's output is cast to the stage's declared Spark schema before it leaves the stage, as a Spark task's shuffle writer casts its input, since a hash exchange places rows by the types of their values too. The block runs in a Comet session over the host's memory pool and disk manager. A partition descriptor child, an in-memory source with one row per partition, carries each partition's index in its stage and, for a stage that reads files, the partition's own copy of the block. An engine that splits a stage between tasks slices it with the stage's other leaves. block_schema plans a block with empty inputs, so that a scheduler can check a block before running it. See #7288. * feat: add comet-scheduler and comet-executor built on DataFusion Ballista A new Cargo workspace, native/native-executors, builds the two processes of a Comet Native Executor cluster on the Ballista 55 crates. It is a workspace of its own, so the main Comet build never depends on Ballista. comet-scheduler wraps Ballista's scheduler in an Arrow Flight service. GetFlightInfo takes a QueryRequest, checks that the client runs the same Comet build, plans each block with Comet's planner, turns the stage graph into a Ballista physical plan, runs it, and returns one endpoint per result partition with an opaque ticket for the executor's DoGet. Hash exchanges become RepartitionExec(Hash), and single-partition and broadcast exchanges become CoalescePartitionsExec, which Ballista's planner turns into stage boundaries. A stage that reads exchanges has as many partitions as they deliver. Ballista can't plan a range shuffle for a physical plan yet (apache/datafusion-ballista#2680), so a range exchange is read as one partition, and the stage that sorts it runs in a single task. comet-executor is Ballista's executor with Comet's codec, which carries each CometFragmentExec's Fragment through Ballista's stage and task plans. It needs no JVM. Both binaries take their settings from the environment, with Ballista's defaults otherwise. See #7288. * test: run queries on comet-scheduler and comet-executor through Flight tests/cluster.rs starts the two binaries, submits stage graphs to the scheduler's Flight service as a driver would, and reads every result partition from the executors with DoGet. It covers a hash shuffle, a range exchange, and a query from another Comet build, which the scheduler refuses. * ci: build and test the native-executors workspace native/native-executors is a Cargo workspace of its own, so the Rust jobs don't cover it. The new linux-test-native-executors job runs clippy and the workspace's tests, including the ones that start a scheduler and an executor, and the lint job checks its formatting. * fix: fail instead of loading native UDFs or missing a JVM on a Comet Native Executor A native executor serves every client and has no JVM. JVMClasses::with_env now returns an error when no JVM was started, instead of dereferencing an unset VM, so a JVM path the planner doesn't screen out (such as a scan's S3 credential provider) fails cleanly. A native UDF loads the library path its plan names under a policy taken from the client's own configuration, so planning one is refused on a native executor. * fix: refuse a stage that reads files and a hash exchange Spark places the rows of a bucketed scan with Murmur3 and shuffles only the other side of a join to match. The hash exchange would place that side with DataFusion's hash, so partitions of the two sides would no longer line up and the join would return wrong rows. build_stage now rejects the shape. * build: have dependabot update the native-executors lockfile The workspace has its own Cargo.lock, which the /native entry doesn't cover. Use the same ignore and grouping rules, and document the limits of what a stage can do on a native executor.
| Commit: | 835a616 | |
|---|---|---|
| Author: | Andy Grove | |
| Committer: | GitHub | |
fix: [branch-1.2] match Spark nullness for missing nested Parquet fields (#7085) (#7313) * fix: match Spark nullness for missing nested Parquet fields * fix: fall back for name-mapped nested Iceberg scans * fix: decline Iceberg scans when table properties cannot be read * docs: migrate nested-field notes to consolidated scan guides * docs: describe list and map nulling for all-missing nested leaves (cherry picked from commit c4a0b1f2bce44ffa2bbeab3bbc1f715853734969) Co-authored-by: KUAN-HAO HUANG <101171023+rich7420@users.noreply.github.com>
| Commit: | d8a272d | |
|---|---|---|
| Author: | Chao Sun | |
| Committer: | GitHub | |
perf: reduce repeated integer keys before left-semi sort-merge joins (#7190) * perf: reduce duplicate right-side rows before sorting left-semi joins * fix: satisfy strict Scala checks in the semi-join benchmark * perf: bypass costly semi-join key reduction and preserve shared buffers * refactor: use fixed-size windows for the key sample * Avoid unprofitable semi-join reduction on nullable, skewed, and shared inputs * Use bounded duplicate filtering on freshly shuffled semi-join keys * Limit semi-join reduction to a single integer key and retry larger shuffle batches * Reserve only one output row for constant shuffle keys * Avoid extra spills when reducing semi-join keys * Check semi-join memory admission when native execution starts * Clarify semi-join reduction scope and prefix policy * Test and explain semi-join reduction admission * fix: scope semi-join memory admission to fair pool * Add many-map membership reduction benchmark and clarify scope
| Commit: | cc6535c | |
|---|---|---|
| Author: | Andy Grove | |
| Committer: | GitHub | |
feat: support the S3A magic committer in native Parquet writes (#6940) (#7281)
| Commit: | dea84f9 | |
|---|---|---|
| Author: | Steve Vaughan | |
| Committer: | GitHub | |
fix: match SPARK-58821 and SPARK-60118 in sequence and string to decimal cast (#7171) (#7266) * fix: match SPARK-58821 and SPARK-60118 in sequence and string to decimal cast Spark 4.0.5, 4.1.4 and 4.2.1, now in release candidates (4.0.5-rc3, 4.1.4-rc4 and 4.2.1-rc3), change two results that Comet reproduces natively: - SPARK-58821: when stop - start overflows a long but the length is small, sequence() returns the elements, where earlier releases raise INTERNAL_ERROR("Unreachable code reached."). A new native variant, spark_sequence_recomputed, does the same, and CometSequence.nativeFunction picks it on those releases. - SPARK-60118: casting a string to decimal no longer applies the integral-digit limit to zero, so a zero with a large exponent ("0e79") casts to 0 instead of NULL (or NUMERIC_OUT_OF_SUPPORTED_RANGE under ANSI). CometCast sets a new Cast proto field, decimal_zero_any_exponent, and parse_string_to_decimal skips the zero limit when it is set. SparkFixes holds the version checks. The internal-error case in sequence.sql moves to CometArrayExpressionSuite, because SQL file version directives compare only major.minor. * docs: describe SPARK-60119 and the version-number caveat, and share one version table Review follow-up. - SPARK-60119, which is in the same release candidates, makes Spark return 0.00 for a string far below the target scale's ulp, as the native cast already does. The cast audit entry and the string.rs comments now say that #6966's positive-scale case applies only before 4.0.5, 4.1.4 and 4.2.1, and that its negative-scale case still differs on them. - The cast compatibility page and CometSequence's compatibility notes say that Comet chooses by the Spark version number, and what a vendor build that reports an older version but includes SPARK-60118 or SPARK-58821 returns, as floating-point.md does for SPARK-59602. - recomputed_length_after_overflow keeps only the new element checks. length_limit_and_overflow_edges and the length grid already cover the internal error and both variants' limit errors. - SparkFixesSuite holds one version table for both fixes and CometSequence.nativeFunction, replacing the near-identical tables in CometArrayExpressionSuite and CometNativeCastSuite, which keep their convert checks. It is listed in both PR build workflows. * fix: follow released Spark for '0e2147483647', and check Spark's error in the sequence test Review follow-up. - On the released path, parse_string_to_decimal counts a zero's integral digits as 1 - scale with a wrapping i32 subtraction, as Spark's int arithmetic does, so '0e2147483647' casts to 0 there. The zero test covers it, and two inputs whose counts don't wrap. - The sequence overflow test's released-Spark branch uses checkSparkError, which checks Spark's INTERNAL_ERROR, exception class and SQLSTATE rather than the message. - The cast audit entry describes SPARK-60118's long digit count: released Spark throws for '1e2147483647' and similar strings, the fixed releases return NULL or raise NUMERIC_OUT_OF_SUPPORTED_RANGE, and the native cast's ANSI error is #7188. - SparkFixes' scaladoc names the two fixes it covers and points to the other version checks. Adapted for branch-1.1, which predates parts of main next to #7171's lines: - sequence.rs: branch-1.1's kernel takes the start, stop and step arrays and a conversion closure, and its sequence_length flags the overflow itself, so the OverflowedLength choice is threaded through that code. The new test also checks that a length over the limit is still an error with spark_sequence_recomputed, which main's length grid covers. - string.rs: the zero_any_exponent change and its test. The SPARK-60119 comment rewording is dropped, since branch-1.1 doesn't have the scale-adjustment comments it edits. - planner.rs: branch-1.1's SparkCastOptions::new_with_version still takes allow_incompat. - The contributor-guide audit entries are dropped, since that guide isn't published from release branches and they describe main's code. SparkFixes' scaladoc doesn't mention ArraySetSupport.normalizesFloats, which branch-1.1 doesn't have. - Import lists keep branch-1.1's names plus the ones #7171 adds. (cherry picked from commit 1dec4c058f3f887f6df0bf21181c6450ba6baf05)
| Commit: | 9bcce3a | |
|---|---|---|
| Author: | Steve Vaughan | |
| Committer: | GitHub | |
fix: match SPARK-58821 and SPARK-60118 in sequence and string to decimal cast (#7171) (#7263) * fix: match SPARK-58821 and SPARK-60118 in sequence and string to decimal cast Spark 4.0.5, 4.1.4 and 4.2.1, now in release candidates (4.0.5-rc3, 4.1.4-rc4 and 4.2.1-rc3), change two results that Comet reproduces natively: - SPARK-58821: when stop - start overflows a long but the length is small, sequence() returns the elements, where earlier releases raise INTERNAL_ERROR("Unreachable code reached."). A new native variant, spark_sequence_recomputed, does the same, and CometSequence.nativeFunction picks it on those releases. - SPARK-60118: casting a string to decimal no longer applies the integral-digit limit to zero, so a zero with a large exponent ("0e79") casts to 0 instead of NULL (or NUMERIC_OUT_OF_SUPPORTED_RANGE under ANSI). CometCast sets a new Cast proto field, decimal_zero_any_exponent, and parse_string_to_decimal skips the zero limit when it is set. SparkFixes holds the version checks. The internal-error case in sequence.sql moves to CometArrayExpressionSuite, because SQL file version directives compare only major.minor. * docs: describe SPARK-60119 and the version-number caveat, and share one version table Review follow-up. - SPARK-60119, which is in the same release candidates, makes Spark return 0.00 for a string far below the target scale's ulp, as the native cast already does. The cast audit entry and the string.rs comments now say that #6966's positive-scale case applies only before 4.0.5, 4.1.4 and 4.2.1, and that its negative-scale case still differs on them. - The cast compatibility page and CometSequence's compatibility notes say that Comet chooses by the Spark version number, and what a vendor build that reports an older version but includes SPARK-60118 or SPARK-58821 returns, as floating-point.md does for SPARK-59602. - recomputed_length_after_overflow keeps only the new element checks. length_limit_and_overflow_edges and the length grid already cover the internal error and both variants' limit errors. - SparkFixesSuite holds one version table for both fixes and CometSequence.nativeFunction, replacing the near-identical tables in CometArrayExpressionSuite and CometNativeCastSuite, which keep their convert checks. It is listed in both PR build workflows. * fix: follow released Spark for '0e2147483647', and check Spark's error in the sequence test Review follow-up. - On the released path, parse_string_to_decimal counts a zero's integral digits as 1 - scale with a wrapping i32 subtraction, as Spark's int arithmetic does, so '0e2147483647' casts to 0 there. The zero test covers it, and two inputs whose counts don't wrap. - The sequence overflow test's released-Spark branch uses checkSparkError, which checks Spark's INTERNAL_ERROR, exception class and SQLSTATE rather than the message. - The cast audit entry describes SPARK-60118's long digit count: released Spark throws for '1e2147483647' and similar strings, the fixed releases return NULL or raise NUMERIC_OUT_OF_SUPPORTED_RANGE, and the native cast's ANSI error is #7188. - SparkFixes' scaladoc names the two fixes it covers and points to the other version checks. (cherry picked from commit 1dec4c058f3f887f6df0bf21181c6450ba6baf05)
| Commit: | f1121ca | |
|---|---|---|
| Author: | Andy Grove | |
| Committer: | GitHub | |
feat: support the S3A magic committer in native Parquet writes (#6940) * feat: support the S3A magic committer in native Parquet writes On Spark 4.0+, a task file that is neither local nor on HDFS is created by the JVM through Hadoop's FileSystem API, as Spark's Parquet writer creates it, and the native writer streams the file's bytes into that stream over JNI. S3A therefore creates the magic committer's files itself and records their pending uploads. A failed write aborts the stream. Planning admits an S3 destination when the S3A magic committer commits it, through PathOutputCommitProtocol or BindingParquetOutputCommitter. Other S3 writes still fall back to Spark. * test: make the magic committer's failed-write check isolate the writer's abort S3A's job abort also aborts pending uploads under the destination, through core-default.xml's fs.s3a.committer.abort.pending.uploads=true, which hid a writer that never aborted its file. Turn that off in the test, use one task so job abort cannot race other tasks, and run each committer binding alone. Add a unit test that the output stream writer sends nothing after a failed write. * feat: support the S3A magic committer in Spark 3.4 and 3.5 native writes The Spark 3.x writer runs the session's commit protocol with the write's Hadoop configuration since #4746, so it reaches S3A's committers as Spark's writer does. Both writers now create task files through one helper, NativeWriteUtils.writeFile, which creates any file that is neither local nor on HDFS through Hadoop and closes or aborts it, and both serdes share one file system gate. The magic committer tests run on every supported Spark version; in-memory commit tracking is skipped on Hadoop 3.3.4, which lacks it. * refactor: simplify the native write path through the JVM The native writer now fails a task with the output stream's own error, which keeps the Java exception for the JNI boundary to rethrow as it was, so HadoopFileOutput no longer records failures and the JVM no longer swaps exceptions. Both writers write a task file through one helper, CometWriteFilesExec.writeFile, which sets the file and stream flag, runs the plan, and closes or aborts the file. Also: the pusher and output stream registrations share their rules, the writer takes an optional stream, local writes report their counted bytes, planning skips object store options and Hadoop configurations it does not need, the committer factory keys come from Hadoop's constants, and the tests share their setup and read S3A's success manifest through SuccessData. * address review: drop the unreachable bucket committer fallback and say once why S3A creates the file * address review: enrich write errors as Spark does, test the Hadoop route's failure paths Fail a native write's task with TASK_WRITE_FAILED as Spark's writer does, keeping fetch failures and fast-failing on an existing file. Admit subclasses of PathOutputCommitProtocol at planning. Build each task's plan once and create its iterator through CometExec, register the output stream through register_before_execution directly, and move the supported destinations to the Parquet page's Writing section. Test a file of several MiB, a failure partway through it, a failed close, a stream that cannot abort, a retried task and a path with special characters, and against MinIO, a failure after S3A uploaded a part and failed task and job commits. * test: compare what a failed job commit leaves pending with Spark's writer S3A's magic committer leaves the uploads of committed tasks pending when job commit fails, for Spark's own writer as well, so check that the native write leaves the same and nothing visible. Scope each pending upload check to its table, and abort what a test leaves behind. * address review: test the fast-fail and commit paths, nested and temporal columns, and the sink's guards Cover the branches of the Hadoop route that had no test: a task file that already exists, a file that exists at task commit (with and without fastFailOnFileFormatOutput), nested and temporal columns compared with Spark's writer through chunked writes and the abort path, an INSERT into a table of the V2 file source, and the output stream sink's oversize and null checks. * fix: handle the fallible execution context lookup in setOutputStream * test: write complex and temporal types through the Hadoop route * test: leave the non-ASCII character out of the output path when the JVM cannot encode it
| Commit: | ca9f52a | |
|---|---|---|
| Author: | Feng Zhang | |
| Committer: | GitHub | |
fix: pick the signed zero Spark ranks first in native percentile (#6928) * fix: pick the signed zero Spark ranks first in native percentile Spark keeps 0.0 and -0.0 as separate keys in the percentile buffer and ranks first the one its OpenHashMap iterates first, which depends on the table's capacity and collisions. When a group holds both zeros, replay Spark's partial, deserialized and final tables to pick the same sign. Pass whether the input was FLOAT so the replay hashes values as Spark does. Closes #6799. * fix: replay each partial buffer separately in native percentile Spark builds, serializes and deserializes one map per partition before merging them into the final buffer, so replaying all merged values as one partial can rank the other signed zero first. Keep the boundaries of the merged partial states and replay each one in turn. Add a two-partition regression test, and write the zero-tie fixtures as one file per insert. * fix: keep percentile partial boundaries in state and gate zero replay on Spark version - Store where each partial ends in the percentile state so a spilled and restored final aggregate replays the same partials. - Replay Spark's tables only when a requested rank lands on a zero. - Replay only on Spark 3.5.2+, which keeps 0.0 and -0.0 as separate keys (SPARK-45599); earlier releases keep the previous behavior, documented as a compatibility note. * fix: nest percentile partials per merge stage and track them only for signed zeros Spark's PartialMerge stages merge each incoming state into a fresh OpenHashMap, so the iteration order of signed-zero keys depends on how many merge stages a row passed through. Record the merge stage in the state (partial_depths) so the replay can nest partials the same way. Track partial boundaries only when the input is FLOAT/DOUBLE on a Spark that keeps both signed zeros as keys; otherwise the lists stay empty. Store the key set as packed bits and drop each partial's table once it has been merged. Document the two unmodeled paths (Spark's sort-based fallback and DataFusion early emit) in the code and the compat note. Remove the redundant unit tests and fix the stale fixture comment. * fix: attach the zero-order note to percentile's support level on Spark 3.5.2+ The serde returned a plain Compatible() for FLOAT and DOUBLE input on Spark versions that keep both signed zeros as keys, even though the replay does not model Spark's sort-based fallback or its aggregates emitting under memory pressure. Return the note with the support level, as the serde already does with signedZeroKeysNote on older Spark, so the limit is visible where the plan shape is accepted. * docs: name Comet's spill in percentile's zero-order note The note said "its own aggregates emitting or spilling", which reads as Spark's aggregates on the compatibility page. Name Comet explicitly and state that a spill inside a merge stage nests that stage's partials one level deeper than Spark does, which is the case where only Comet's answer can move. --------- Co-authored-by: Andy Grove <agrove@apache.org>
| Commit: | 1dec4c0 | |
|---|---|---|
| Author: | Steve Vaughan | |
| Committer: | GitHub | |
fix: match SPARK-58821 and SPARK-60118 in sequence and string to decimal cast (#7171) * fix: match SPARK-58821 and SPARK-60118 in sequence and string to decimal cast Spark 4.0.5, 4.1.4 and 4.2.1, now in release candidates (4.0.5-rc3, 4.1.4-rc4 and 4.2.1-rc3), change two results that Comet reproduces natively: - SPARK-58821: when stop - start overflows a long but the length is small, sequence() returns the elements, where earlier releases raise INTERNAL_ERROR("Unreachable code reached."). A new native variant, spark_sequence_recomputed, does the same, and CometSequence.nativeFunction picks it on those releases. - SPARK-60118: casting a string to decimal no longer applies the integral-digit limit to zero, so a zero with a large exponent ("0e79") casts to 0 instead of NULL (or NUMERIC_OUT_OF_SUPPORTED_RANGE under ANSI). CometCast sets a new Cast proto field, decimal_zero_any_exponent, and parse_string_to_decimal skips the zero limit when it is set. SparkFixes holds the version checks. The internal-error case in sequence.sql moves to CometArrayExpressionSuite, because SQL file version directives compare only major.minor. * docs: describe SPARK-60119 and the version-number caveat, and share one version table Review follow-up. - SPARK-60119, which is in the same release candidates, makes Spark return 0.00 for a string far below the target scale's ulp, as the native cast already does. The cast audit entry and the string.rs comments now say that #6966's positive-scale case applies only before 4.0.5, 4.1.4 and 4.2.1, and that its negative-scale case still differs on them. - The cast compatibility page and CometSequence's compatibility notes say that Comet chooses by the Spark version number, and what a vendor build that reports an older version but includes SPARK-60118 or SPARK-58821 returns, as floating-point.md does for SPARK-59602. - recomputed_length_after_overflow keeps only the new element checks. length_limit_and_overflow_edges and the length grid already cover the internal error and both variants' limit errors. - SparkFixesSuite holds one version table for both fixes and CometSequence.nativeFunction, replacing the near-identical tables in CometArrayExpressionSuite and CometNativeCastSuite, which keep their convert checks. It is listed in both PR build workflows. * fix: follow released Spark for '0e2147483647', and check Spark's error in the sequence test Review follow-up. - On the released path, parse_string_to_decimal counts a zero's integral digits as 1 - scale with a wrapping i32 subtraction, as Spark's int arithmetic does, so '0e2147483647' casts to 0 there. The zero test covers it, and two inputs whose counts don't wrap. - The sequence overflow test's released-Spark branch uses checkSparkError, which checks Spark's INTERNAL_ERROR, exception class and SQLSTATE rather than the message. - The cast audit entry describes SPARK-60118's long digit count: released Spark throws for '1e2147483647' and similar strings, the fixed releases return NULL or raise NUMERIC_OUT_OF_SUPPORTED_RANGE, and the native cast's ANSI error is #7188. - SparkFixes' scaladoc names the two fixes it covers and points to the other version checks.
| Commit: | c4a0b1f | |
|---|---|---|
| Author: | KUAN-HAO HUANG | |
| Committer: | GitHub | |
fix: match Spark nullness for missing nested Parquet fields (#7085) * fix: match Spark nullness for missing nested Parquet fields * fix: fall back for name-mapped nested Iceberg scans * fix: decline Iceberg scans when table properties cannot be read * docs: migrate nested-field notes to consolidated scan guides * docs: describe list and map nulling for all-missing nested leaves
| Commit: | cead4b7 | |
|---|---|---|
| Author: | dustin | |
| Committer: | GitHub | |
fix: end the Delta scan's plan-data key with its plan id (#6932) * fix: end the Delta scan's plan-data key with its plan id Two Delta scans of one table with equal schema, projection and data filters got the same key, since common.source is "(unknown)" outside of explain. The key now ends with the op's plan_id through PlanDataInjector.withPlanId, as the core and Iceberg keys do. * test: move the Delta plan-data key test into CometDeltaNativeScanSuite The test now sits next to the self-join test and checks only what the fix changes: two partition-filtered scans with equal common protos get different keys, each ending with its op's plan id, and each scan's planning data holds only its own partition's files. The getKey, rewrap and sameResult checks covered unchanged code and are gone.
| Commit: | 653c795 | |
|---|---|---|
| Author: | KUAN-HAO HUANG | |
| Committer: | GitHub | |
feat: support AtLeastNNonNulls natively (#6180) * feat: support AtLeastNNonNulls natively * refactor: drop AtLeastNNonNulls small batch threshold Keep only the bitmap counter. At 8192 rows, the row counter was slower in 51 of 56 measured cases on a non-idle machine. * fix: fall back when AtLeastNNonNulls would skip a shared subexpression Spark's subexpression elimination evaluates a shared subexpression for every row before the predicate, so an ANSI failure in it surfaces even on rows where AtLeastNNonNulls reaches n first. Decline native conversion when a child after the first n, which native evaluation may skip, holds such a subexpression, reusing the check behind canShortCircuitNulls. * fix: preserve shared expressions across decimal promotion
| Commit: | 06cff60 | |
|---|---|---|
| Author: | Liang-Chi Hsieh | |
| Committer: | GitHub | |
feat: execute scalar Arrow Python UDFs in native Comet (#6130) * feat: execute scalar Arrow Python UDFs in native Comet * docs: clarify scalar Arrow UDF support starts in Spark 4.1 * fix: align native Arrow UDF semantics and CI coverage * fix: avoid Arrow UDF polling spin and narrow schema support * Fix embedded Python symbols and verify Arrow input types * Document embedded Python build and deployment requirements * Fix native Arrow UDF environment and batch semantics * Honor Arrow Python UDF byte batch limit * Skip empty Arrow UDF batches and protect plan metadata * Fix native Arrow UDF subqueries, file fallback, and exchange reuse * Address native Arrow UDF review: archives, results, logging, metrics - Keep queries with Spark-added archives or Python worker logging on Spark's worker path. - Accept spark.executorEnv.* entries in a UDF's environment when the executor process already has them; PYTHONHASHSEED must stay 0. - Check the enable config last, so other ArrowEvalPythonExec types report why they are unsupported instead of suggesting the config. - On Spark 4.2, accept any result pyarrow.RecordBatch.from_arrays accepts, such as a list or NumPy array. - Decode invalid UTF-8 in returned string arrays, as for JVM imports. - Include the Python traceback in UDF errors. - Report output_rows, elapsed_compute and python_time metrics. - Document the task-kill limitation and fix the memory docs. - Test sliced inputs for every type, a multi-row-group Parquet scan, archives, and the new fallbacks. * Fix Arrow UDF tracebacks before Python 3.12 and serialize imports Before Python 3.12, PyO3 keeps an exception's traceback apart from its value, so passing only the value to traceback.format_exception dropped the traceback. Pass the type, value and traceback explicitly. Concurrent first imports of PyArrow from several task threads can hand one thread a partially initialized pyarrow.lib on Python 3.10. Import PyArrow while the interpreter is initialized, before any other thread can run Python, and unpickle callables one at a time under a lock that waits without holding the GIL. * Start embedded Python with unbuffered stdio Spark starts its Python workers with PYTHONUNBUFFERED. The embedded interpreter is never finalized, so print() output still buffered in sys.stdout was lost when the executor exited. Disable stdio buffering and leave the JVM's C streams unconfigured. A regression test runs a UDF that prints in a child process, exits without flushing, and checks that the parent reads the output. * Stop native Arrow UDF stream after its first error After a Python error, polling the stream again evaluated the next slice of the batch, and after a child error flat_map pulled the next child batch, so a consumer that kept polling called Python again while the task was failing. End the stream after the first error it yields. A regression test splits two child batches into four slices with a UDF that raises, drains the stream, and checks that Python ran once. --------- Co-authored-by: Isaac <no-reply@databricks.com>
| Commit: | 39380cc | |
|---|---|---|
| Author: | Chao Sun | |
| Committer: | GitHub | |
feat: opt in to join filter pushdown through deterministic residuals (#6431)
| Commit: | 3355602 | |
|---|---|---|
| Author: | Andy Grove | |
| Committer: | GitHub | |
fix: keep a streamed side sorted outside the native plan in order through outer hash joins (#6835) * fix: keep a streamed side sorted outside the native plan in order through outer hash joins Spark's hash joins report the streamed side's order as their own, so Spark drops a sort above the join that the order satisfies. DataFusion's hash join keeps the unmatched probe rows of a right outer join in place only when the probe input declares an order, and otherwise moves them after the matched rows of each batch. Comet builds every hash join on the build side, so Spark's left outer join built on the right and right outer join built on the left both run as that right outer join, and a streamed side that enters the native plan from the JVM, a relation sorted before it was cached for instance, arrives through a ScanExec that declares no order. The join then returned those rows out of order with no sort left in the plan to fix them. Send the streamed side's sort order with the join for those two shapes, and have the planner declare it on the probe input through a pass-through SortedInputExec when the input declares none. The declaration is placed on that one input rather than on every ScanExec because an order visible to the whole native plan would also change how DataFusion's aggregates group. A sort order that cannot be expressed natively sends the join back to Spark. With the in-memory cache on by default since #5634, the native cache scan reaches this with default settings. * test: cover the right outer shape, an aggregate above the join and the fallback From review of the hash join order declaration: - The join and cache tests also run a right outer join built on the left, which Comet does not swap, and compare rows as multisets, not sets. - A cache test groups by the key above the join, where the declared order puts both aggregates in sorted mode. - A join test covers the fallback when the streamed order cannot be serialized. - Planner tests cover declaring the order on the probe input for both shapes, and leaving a probe that already declares one alone. - SortedInputExec's doc says the declared order is visible above the join. * fix: declare the streamed order only over operators that keep a JVM input's order From review: - with_declared_order walks down from the probe input through operators that report keeping their input's order and declares only when it reaches a ScanExec or ShuffleScanExec. DataFusion reports a right hash join as keeping its probe order even when it moves unmatched rows, so the walk stops at any hash join that is not an inner join. The null-aware anti join stops it, so an order Spark reports above NOT IN no longer reaches the aggregates above the outer join. - doConvert sends the longest prefix of the streamed order that Comet can express, since declaring any order keeps the probe rows in place, and falls back only when it cannot express the first key. - SortedInputExec's doc names the top-k sort as another consumer, and the CometHashJoin comment names the null-aware anti join as the exception. - Tests: dwsmith1983's NOT IN repro, a residual join condition, a descending, nulls last streamed order with null keys, the prefix case, the aggregate test's join and exchange checks, and planner tests for the walk and the planned join type. * fix: send no streamed order when a Comet join that may move rows produces it A cache reports the order of the plan that wrote it, so the output of a native null-aware anti join, an outer hash join whose probe order the native planner did not declare, or a nested loop join can reach an outer hash join as a JVM input whose reported order does not hold. Declaring that order put the aggregates above the join in sorted mode over unsorted rows, which split groups. CometHashJoin.doConvert now walks down the streamed side, through CometScanWrapper and into the plan that wrote any cache it reads, and sends no order when such a join produces it with no sort in between. * fix: walk an adaptive cache's final plan and keep the streamed order through semi joins Review feedback on the streamed order that a native outer hash join declares on its probe input. The serde walk that decides whether a cache or checkpoint can be trusted now: - reads the final plan of an adaptive cached plan, as Spark's cache scan does, since the adaptive plan reports no order itself and the walk stopped there; - follows only the streamed side of a hash join, and counts an outer join as declared only when it sent its order and its probe input is a scan in its own native plan; - no longer trusts a leaf that reports an order unless it is a scan or a range, which covers an RDD checkpoint whose plan is gone. The planner now walks through right semi, anti and mark hash joins as well as inner ones, since DataFusion builds their probe indices in ascending order whether or not the probe declares an order. Only a right outer join depends on the declaration. Tests cover a cached NOT IN with AQE off and on, through Spark's cache scan and through the native one, a checkpointed NOT IN, an outer join over a cache written by a declared outer join and by one over a sample, an outer join over a semi join, a NOT IN below an inner join's build side, and a broadcast outer join over a sorted cache with default settings. The test that compared Comet with itself now compares against the rows. * docs: describe how native hash joins keep the streamed order Add a Hash Join Output Order section to the operator compatibility page: native hash joins keep the streamed side's order Spark relies on, an outer join whose first streamed sort key Comet cannot express falls back to Spark, and where the order might not hold Comet declares nothing. * fix: keep the streamed order to expressible keys and stop CometSampleExec reporting one Spark's SampleExec reports no output ordering, so CometSampleExec must not report its child's: a cache written by a join over a sample carried an order that nothing keeps. The streamed order now stops at a key whose type the native sort does not accept, as for a collated string, SortedInputExec keeps its input's equivalence properties, and the contributor guide says that an operator reporting an ordering must keep it natively. * test: cover the streamed order for decimal, string and timestamp keys Also assert that the aggregate test's join sent the order and emits its groups sorted, and replace the sample test with one for a cache that reports no order. * test: compare the key-type rows with Spark's as a multiset * test: make the aggregate test fail when the join's order is not declared With 40 keys of 50 rows each, a join that moved its unmatched rows still fed the aggregate the keys in first-seen order, so the test passed without the declaration. Add a layout of 1000 keys with two rows each, where a batch holds alternately matched and unmatched keys, and assert that the serde sent the order and that each partition comes back in key order. * fix: declare the order of a native range for the joins above it Spark's RangeExec reports id ascending, or descending for a negative step, and Spark drops a sort above an outer hash join on that basis. The native range is a leaf with no JVM input, so the planner declared no order on the join's probe input and the join moved its unmatched rows. LazyMemoryExec::add_ordering leaves the cached output ordering of the plan properties stale in DataFusion 55, so the planner wraps the range in a SortedInputExec instead, which now reports its input's metrics. * refactor: name the streamed side of a hash join once * test: cover a native range and share the cache test helpers Move partitionsInOrder, keyOrder, checkEachKeyGroupedOnce and cachedPlanOf to CometTestBase so both suites use one copy, and set up the sorted cache the join tests share once. * test: drop the imports the shared test helpers left unused Moving partitionsInOrder and keyOrder to CometTestBase left InterpretedOrdering and SortDirection unused in CometJoinSuite, which scalafix and the strict Scala warnings build reject. * test: cover a streamed double order with signed zeros and NaN An outer hash join over a streamed side sorted by a double key with -0.0, 0.0, NaN and the infinities keeps its order, groups each key once, and returns Spark's rows for a top-k above it. The planner declares the order on the key normalized as the native sort normalizes it, and the top-k runs in a native plan of its own over the join's output. * fix: send a streamed order that starts with an array or a struct The check on the streamed order's keys took only the scalar types that the native sort accepts for a single column, so a join whose streamed order starts with an array or a struct went back to Spark. exprToProto takes those keys, and declines a nested key under a non-default null order itself. The check was there for the collated string, so it now rejects a key that holds one, at the top level or nested, as supportedSortType does for a sort. * fix: declare the order of a sorted bucketed table read by the native scan With spark.sql.legacy.bucketedTableScan.outputOrdering on, Spark reports a sorted bucketed table's sort columns as the scan's order when every bucket has one file. The serde trusted that order and sent it, but the planner declared an order only over a JVM input, so an outer hash join over the native scan moved its unmatched rows after Spark had dropped the sort above it. A scan of files reads the files of its one file group one after the other, so the planner now counts it as a source whose order holds. * docs: qualify where native hash joins keep the streamed order The operator compatibility guide now says the order is kept where Comet can show that it holds, and points to #6922 for the caches it still leaves undeclared. The contributor guide says what a leaf that reports an ordering has to do natively. * fix: drop the native NOT IN order handling that the ordered-side fallback makes unreachable Comet now leaves a null-aware anti join over an ordered streamed side in Spark (#6810), so a native NOT IN never reports an order and its rows cannot reach an outer hash join with a false one. Remove the walk's null-aware case, the tests that needed a native NOT IN to write a cache or checkpoint, and the documentation that named NOT IN as a source of a false order.
| Commit: | d443662 | |
|---|---|---|
| Author: | Andy Grove | |
| Committer: | GitHub | |
feat: support PivotFirst aggregate for the optimized PIVOT fast path (#4798) * feat: support PivotFirst aggregate for the optimized PIVOT fast path Wire the Spark-internal PivotFirst aggregate through Comet so that PIVOT queries which trigger Spark's two-phase fast path (12+ pivot values with an aggregate whose output type is in PivotFirst.supportsDataType) run natively instead of falling back. Scaffolded with the implement-comet-expression skill. * simplify: address /simplify review of PivotFirst - Delegate value-type gate to Spark's `PivotFirst.supportsDataType` instead of maintaining a parallel list in the serde. - Wrap the plan-time pivot map/vector in `Arc` so per-group accumulator init bumps a refcount instead of deep-cloning. - Drop the redundant `pivot_values` field from the accumulator (only the map and `slots.len()` are read at runtime); the accumulator now derives its slot count from the shared map. - Use the shared `format_state_name` helper for state field names. - Delegate `evaluate()` to `state()` so the two paths share their slot materialization. - Walk each state column backwards in `merge_batch` and stop at the first non-null so we build at most one `ScalarValue` per slot per merge. - Drop the "pad every IN clause to 12 values" workaround from the SQL tests and the CometAggregateSuite test - the fast-path gate is supported-data-type only, no minimum count. * fix: match Spark's float pivot key semantics in PivotFirst * fix: fall back for PivotFirst pivot columns and lists Comet cannot match Spark finds a pivot value's slot in a Scala HashMap only for atomic pivot columns, and even there a binary value matches by reference, so no row ever matches before Spark 4.2. Arrays and structs (including the struct that a pivot on several columns builds) are matched through an interpreted ordering, and from 4.2 so are binary values and non-default collations. The native lookup reproduces none of that, so those pivot columns now fall back. Spark also sizes the aggregate buffer from the distinct pivot values, so a list with duplicates (including 0.0 next to -0.0) made the native partial aggregate emit more columns than the shuffle schema declares. Such lists fall back too. The SQL fixture now asserts native execution for every query that should run natively, covers each pivot column type that does, and checks each fallback. * perf: add a GroupsAccumulator for PivotFirst PivotFirst only had a row Accumulator, so grouped aggregation went through DataFusion's GroupsAccumulatorAdapter, which visits every group's state on each batch, and the accumulator built a ScalarValue for every input row. The grouped accumulator keeps every group's slots in one flat, group-major buffer with a validity bitmap, and returns them as the list without copying. Pivot values are looked up in a typed index built once per aggregate, with Spark's float rules in a new float_semantics::ieee_equality_key helper. The ungrouped accumulator shares the index. A pivot list with a repeated value is rejected at construction, since the serde falls back for it. * test: add a benchmark for the optimized PIVOT plan CometPivotBenchmark pivots sum(v) over 4M rows on a string column with 4, 32 or 128 values, grouped into 1K, 100K or 1M groups, and compares Spark, Comet with PivotFirst disabled (so the aggregates using it fall back while the rest of the plan stays native) and Comet with PivotFirst native. The query selects every pivot column into the noop sink, and before timing each Comet run is checked against Spark's answer and for the plan its name claims. * perf: copy only the emitted groups out of PivotFirst's state `EmitTo::First(n)` split the cell buffer with `split_off`, which leaves the original allocation with the first part and copies the rest. The emitted batch therefore held the buffer of every group, and the remaining groups were copied, nearly doubling live memory for a partial emit of a few groups. `EmitTo::take_needed` copies out whichever part is smaller. * refactor: run ungrouped PivotFirst through the grouped accumulator The ungrouped accumulator kept a `ScalarValue` per slot, built one with `ScalarValue::try_from_array` for every matching input row, and repeated the grouped accumulator's update, merge and state logic. It now wraps a `PivotFirstGroupsAccumulator` holding one group. `evaluate` reads that group's slots without taking them, because DataFusion requires that an accumulator can be evaluated more than once. That leaves native code no row-based path for a value type the grouped accumulator does not store. The serde therefore checks the value type against the types the native aggregate stores rather than calling `PivotFirst.supportsDataType`, so a type that a later Spark version adds falls back instead of failing the query. * test: cover ungrouped, empty, multi-batch and collated PivotFirst inputs PivotFirst.sql gains a pivot with no grouping column, the same pivot over no rows, and a pivot over 20000 groups whose input is repartitioned by the pivot column, so that each group has several partial states to merge. PivotFirst_collation.sql checks on Spark 4.0+ that a UTF8_LCASE pivot column falls back. Every native query now also asserts that PivotFirst itself ran natively. That was the only thing the two CometAggregateSuite tests checked that the fixture did not, so they are removed. The fixture's comments no longer name a nonexistent PivotTransformer or threshold, or a cast the float query does not make.
| Commit: | d64915e | |
|---|---|---|
| Author: | KAZUYUKI TANIMURA | |
| Committer: | GitHub | |
fix: raise DIVIDE_BY_ZERO from native corr on a zero denominator in ANSI mode (#6830) * fix: match Spark variance updates for large nearby values * fix: match Spark statistical aggregate merges * fix: preserve Spark semantics when merging empty partials * test: cover regression aggregates merging fractional constants * fix: align scalar correlation evaluation with Spark * fix: preserve correlation underflow and NaN semantics * test: normalize NaN in correlation extreme-value assertions * fix: restore native regression aggregates after merge correction * fix: preserve Spark regr_r2 divide-by-zero semantics * fix: pass evaluation mode in regression benchmark * fix: Native corr, covariance, variance and stddev return wrong values for a constant fractional column merged from several partitions * test: compare corr's query context inside withSQLConf for Spark 3.x withSQLConf returns Unit on Spark 3.4 and 3.5, so the expected error could not be returned from the block. * fix: attach query context to regr_r2's divide-by-zero error Move ZeroDivisor into welford.rs so regr_r2 shares it with corr, and pass the aggregate's expr_id and the query context registry to Regr. Both ANSI tests now compare the error's query-context fragment with Spark's. * test: tighten corr ANSI coverage per review Use expect_error_class in corr.sql so the ANSI queries also check native operators, exception class and SQLSTATE. Keep only the ANSI branch of the Scala test, whose LEGACY checks repeat the fractional-constants test, and drop a Rust test loop over null_on_divide_by_zero that never reaches the count == 1 branch. --------- Co-authored-by: rich7420 <rc910420@gmail.com>
| Commit: | e4babc4 | |
|---|---|---|
| Author: | Andy Grove | |
| Committer: | GitHub | |
perf: store shuffle offsets as lengths with zstd (#6943) * perf: store shuffle offsets as lengths with zstd Arrow's offsets buffers are running sums, which compress poorly. With zstd, the shuffle block writer now rewrites every offsets buffer of the record-batch body as lengths split into byte planes before compressing, and flags the block in its codec tag. The reader rebuilds the offsets before Arrow decodes the body. Shuffle files with string, binary, array and map columns get 9% to 40% smaller, and zstd gets no slower. LZ4 and snappy blocks are unchanged: lz4_flex decodes the long runs that lengths compress into slowly (PSeitz/lz4_flex#227). spark.comet.shuffle.compression.offsetsAsLengths.enabled (default true) turns it off. * refactor: decide offsets as lengths once per shuffle block writer ShuffleBlockWriter::try_new_with_offsets_as_lengths replaces the builder. The writer now decides when it is built: it checks once, on an empty batch, that arrow-ipc writes the buffers the mapped roles describe, keeps plain offsets if not, and sets the codec-tag flag in its header. That removes the per-block flag plumbing through every codec arm and the header rewrite. The writer and reader share one buffer-range helper and one rewrite loop. The reader rebuilds offsets before freezing the body. The two Scala tests share a cometShuffleMetric helper. * docs: say where shuffle offsets are stored as lengths The tuning guide and the config description now say that native shuffle and JVM shuffle blocks without dictionary-encoded columns store offsets as lengths with zstd, and that Celeborn shuffle blocks do not. native_shuffle.md describes the codec-tag flag, and the native and JVM shuffle config tables list spark.comet.shuffle.compression.offsetsAsLengths.enabled. * docs: list offsets_as_lengths.rs in the native shuffle guide * test: cover nulls, empty values, binary and struct columns and spilling in the offsets-as-lengths tests
| Commit: | 5c7b3ea | |
|---|---|---|
| Author: | KUAN-HAO HUANG | |
| Committer: | GitHub | |
fix: match Spark statistical aggregate updates and merges (#6076) * fix: match Spark variance updates for large nearby values * fix: match Spark statistical aggregate merges * fix: preserve Spark semantics when merging empty partials * test: cover regression aggregates merging fractional constants * fix: align scalar correlation evaluation with Spark * fix: preserve correlation underflow and NaN semantics * test: normalize NaN in correlation extreme-value assertions * fix: restore native regression aggregates after merge correction * fix: preserve Spark regr_r2 divide-by-zero semantics * fix: pass evaluation mode in regression benchmark
| Commit: | 2b45c9d | |
|---|---|---|
| Author: | dustin | |
| Committer: | GitHub | |
feat: add native Delta Lake scan contrib module (page/row-group pruning) (#5365) * feat: add JVM-planned native Delta Lake scan contrib module Adds an optional contrib/delta-spark module that claims delta-spark DSv1 scans through CometScanContrib and runs them on Comet's shared native parquet path, including main's JVM-exact field-name folding for case-insensitive footer matching. Deletion vectors are decoded natively into per-file ParquetAccessPlans that DataFusion intersects with row-group and page-index pruning, so DV skips and page skips compose in a single scan. Scans the native path cannot serve safely (DML row-index reads, unsupported filesystem schemes, userinfo-bearing authorities, credential-provider-only auth, S3 config divergence, multi-store shapes) fall back to Spark with an explained reason. Co-authored-by: Scott Schenkein <schenksj@yahoo.com> Co-authored-by: Aditya Vaish <adivaish@microsoft.com> * fix: contain provider link failures, gate Hadoop-only S3 endpoints, and check inline DV sizes Review round: discovery skips a provider whose class cannot link, the S3 claim gate declines a scheme-less endpoint with SSL off and any assumed-role STS endpoint, inline deletion vectors are checked against their descriptor size, row-group offsets use checked arithmetic, and the rebase arms return the input batch untouched when nothing predates the cutover. Build and CI: one delta.version, the spark test-jar only under the delta profile, a plain-runner job for the MinIO suite, a tighter change filter, and docs that name every Spark line the workflow runs. * fix: keep every leaf requested when the name fold fails during rebase planning * ci: activate the delta profile for the prerequisite installs and set up the MinIO runner with sudo * perf: pass null-free modern batches through the rebase untouched with one vectorised minimum * chore: drop redundant interpolators flagged by scalafix * ci: save the Delta contrib Maven caches on main only * fix: adapt the Delta contrib to the prepared plan-data API CometExecRDD now carries the driver-computed plan fingerprint, and PlanDataInjector splits parsing the partition-invariant common from injecting a partition. The Delta scan passes the fingerprint, and DeltaPlanDataInjector parses the common DeltaSparkScan once in prepareCommon and merges only the file list in inject. DeltaPlanDataInjectorSuite pins the contract, including reuse and stale replacement through core's shared memo. * chore: drop redundant string interpolators in CometDeltaNativeScanSuite * fix: keep the rebase pass-through for nullable all-modern batches The null-count guard sent any column holding a single null through try_unary, allocating a fresh buffer per batch even when every valid value is modern. Null-free arrays keep the vectorised minimum; arrays with nulls take one validity-aware pass, and the pass-through test asserts the same Arc comes back for masked dates and timestamps under every policy. * test: let COMET_DELTA_S3_REQUIRED turn a missing MinIO into a hard failure The dedicated runner job exists only to run CometDeltaS3Suite, so a cancelled run and a real one looked the same. The job sets the switch, and the suite then fails instead of cancelling when Docker is unavailable or MinIO does not start. * docs: guard every Delta scan config row and drop the generated-table markers Nothing fills the CONFIG_TABLE markers on the Delta page, so the table is hand-maintained. The guard now renders every DeltaScanConf entry with its default, flags stale rows, and fails rather than cancels when the page cannot be found. * ci: run the Delta contrib suites for one Spark profile in the queue The single Delta job ran three profiles on every queued change that touched native or spark sources. Spark 3.5 now runs in the queue and 4.0 and 4.1 run nightly, with run-delta-tests opting a pull request into all three, the same split the Iceberg tiers use. The reusable workflow takes the profile as an input, and the MinIO, feature-off and dev script jobs run once, on the 3.5 call. * ci: use setup-java v6 in the Delta MinIO job like the other workflows * fix: share one memory consumer across a partition's deletion vectors and drain the listener bus in the test helpers Every DV'd file registered its own memory consumer and the reservation lived in the file's extensions until the task ended, so a partition with many DV'd files lowered the fair pool's limit for every other operator in the task. attach_access_plans now registers one consumer per call and hands each file an empty reservation from it. A test pool that copies the fair limit check pins it. The plan capture helpers unregistered their listener as soon as the body returned, so a late callback was missed. They now drain the listener bus, and collectTaskInputMetrics does the same instead of polling with a floor. DV'd files are split like any other claimed file. The planning helper comment says so and what each split does today, and a test reads one small-row-group file as many splits. contrib/delta-spark reads its own delta.spark.version property so the in-tree contrib's delta.version, which spark/pom.xml overrides per Spark profile, no longer diverges from it. * style: drop a redundant string interpolator in the Delta scan suite * docs: list every check the Delta scan reuses from the core native scan gate * fix: choose the timestamp rebase policy from the requested type The native Delta scan picked a timestamp leaf's rebase policy from its physical Arrow type, while Spark's ParquetVectorUpdaterFactory keys on the Spark type the column is read as. Two cases differed. An INT96 or adjusted INT64 column read as TIMESTAMP_NTZ was checked or rebased, though Spark never rebases NTZ reads, so an ancient value failed under EXCEPTION. A timezone-free INT64 column read as TIMESTAMP was never rebased, though Spark applies the datetime rebase mode whatever isAdjustedToUTC says, so EXCEPTION returned the row and LEGACY returned it unshifted. The walk that already pairs each physical leaf with its requested type now records the leaves read as TIMESTAMP_NTZ and the timezone-free leaves read as TIMESTAMP, and the policy follows that record. Dates keep the date policy, including a DATE column read as TIMESTAMP_NTZ, which Spark 4.x rebases. A metadata-free timezone-free file read as TIMESTAMP under LEGACY is now refused, as adjusted files without a recorded writer time zone already are, because the rebase needs the JVM's time zone tables. * perf: limit Delta deletion-vector selectors to each file split * docs: describe per-split row selectors in the Delta scan planning comment Each split builds selectors only for the row groups that start inside it and shrinks its reservation to match, so the comment no longer says the access plan and reservation cover the whole file. * chore: tighten Delta scan comments Shorten several comment blocks in the Delta scan and rebase code and describe each test case directly. Rename the 1500-01-01 day rebase test after what it checks. * test: check Delta S3 key coverage in the native tests The Delta suite requires its S3 config comparator to cover every fs.s3a.* key the native S3 client reads, but it only runs in the merge queue. A core change that adds a key passed the pull request checks and then failed in the queue. A Rust test in s3.rs now reads S3ConfigKeyConsumers from DeltaScanSupport.scala and fails when a native key is missing, so the pull request run reports it. A Delta suite test checks that the same text parse matches the compiled key list, so a reformat of that block fails in the Delta change's own run instead of breaking the Rust test later. * fix: skip calendar rebase checks when Spark reads a date or timestamp as a number Spark picks a numeric updater before any temporal handling when the requested type is numeric, so a date read as an integer type or a timestamp read as BIGINT comes back as the stored value with no calendar check. The native Delta scan still applied the column's temporal rebase policy and failed ancient values under EXCEPTION mode, for example a metadata-free TIMESTAMP(NANOS) file read as BIGINT with spark.sql.legacy.parquet.nanosAsLong after CONVERT TO DELTA. Leaves requested as one of Spark's numeric pairings now get no rebase policy, while temporal requests keep theirs. * fix: decline split-dependent metadata and resolve SSE-C like Hadoop in the Delta scan Two gaps in the native Delta scan's admission checks: - `_metadata.file_block_start` and `file_block_length` came back with the wrong split's values on a split file, because DataFusion assigns a row group to the split holding its first page while Spark uses its midpoint. Core already falls back for these columns; the Delta scan claimed the read before core's guard ran. It now declines when either column is used above the scan, so deletion-vector reads, which carry the whole `_metadata` struct, stay native. - The SSE-C check resolved the encryption algorithm on the raw Hadoop configuration. S3AFileSystem resolves it on the bucket-propagated view, so a bucket override reached through a `${...}` reference could select SSE-C while the check saw AES256. The algorithm is now resolved on both views and the scan declines unless both are on the allowlist. A credential provider path is read only when every entry is a local keystore or the user provider, since Hadoop drops providers backed by other filesystems. Decline reasons no longer print a resolved configuration value: the algorithm reason names only a known algorithm, and the provider-class reason names the key and bucket. * fix: rebase INT32 values Spark reads as DATE in the Delta scan Spark's Parquet reader rebases an INT32 column requested as DATE whether or not the file annotates it as a date. The native Delta scan treated an unannotated INT32 leaf as already proleptic, so a legacy or metadata-free file converted with CONVERT TO DELTA returned 1500-01-10 where Spark returns 1500-01-01 under LEGACY, and returned the row where Spark refuses it under EXCEPTION. An INT32 leaf requested as DATE now takes the file's date policy, and the stored days are rebased before the schema adapter casts them. * fix: restore the Delta scan with its live pruning filters on transition revert When transition reversion hands a stage back to Spark, it restores each Comet node's original plan. The Delta native scan inherited the default, which returned the original FileSourceScanExec still holding the SubqueryAdaptiveBroadcastExec placeholder that AQE had already rewritten on the live scan, so execution failed with "SubqueryAdaptiveBroadcastExec does not support the execute() code path". The scan now restores the original plan with its current partition filters and logical link, as CometNativeScanExec does. Data filters keep the original scan's list, because the live list also carries subquery predicates taken from the parent filter. * test: fix strict-warnings errors in the Delta contrib suites Literal Hadoop ${...} references now sit in s-strings with $$ so the strings are unchanged, spark.range bounds use Long literals, an unused helper is removed, and an inferred existential type is spelled out. * ci: run the Delta contrib suites nightly and compile the contrib in the PR tier The Spark 3.5 Delta suites leave the merge queue and run nightly with 4.0 and 4.1, still on demand through run-delta-tests. The strict warnings job now compiles with -Pdelta, and changes under contrib/delta-spark route to the Linux PR build, so a change that breaks the contrib build on Spark 3.5 fails on its own pull request instead of in the queue. * docs: say the Delta contrib jar is built from source and mark the guide experimental No release publishes comet-contrib-delta-spark yet, so the setup step points at the contrib README's build steps, and the sidebar entry carries the same (experimental) marker as the page heading. * test: check the timestamp labels of a Delta scan that runs alone #6868 passes each native plan's Spark output types to CometExecRDD, so that under spark.comet.testing.strictTimestampLabels.enabled CometExecIterator checks the labels on every batch the plan returns. CometDeltaNativeScanExec builds its own CometExecRDD when it is not fused into a larger native block, and it passed no output types, so the check skipped it. Pass them, as CometNativeScanExec and CometIcebergNativeScanExec do. --------- Co-authored-by: Scott Schenkein <schenksj@yahoo.com> Co-authored-by: Aditya Vaish <adivaish@microsoft.com> Co-authored-by: Andy Grove <agrove@apache.org> Co-authored-by: Chao Sun <chao@openai.com>
| Commit: | daf33bd | |
|---|---|---|
| Author: | Andy Grove | |
| Committer: | GitHub | |
fix: skip an array or map function's later arguments where the array or map is NULL (#6716) * fix: skip an array or map function's later arguments where the array or map is NULL Spark's array_contains, array_position, array_remove, arrays_overlap, array_union, slice, m[k] and a[i] are null-intolerant BinaryExpressions or TernaryExpressions, which return NULL for a NULL array or map without evaluating the other arguments. The native functions evaluated every argument over the whole batch, so an argument that raises, such as an ANSI cast of a malformed string, failed a query that Spark completes, and a nondeterministic argument saw rows Spark never gives it. Add a native NullShortCircuit wrapper that evaluates the first argument over the whole batch and each later one only for the rows where no earlier argument is NULL, as ArrayInsert already does for its own arguments. A null_short_circuit flag on ScalarFunc and ListExtract enables it, set through QueryPlanSerde.withNullShortCircuit. The planner leaves a function unwrapped when every later argument can be evaluated for any row. Closes #6613. * Simplify NullShortCircuit and skip masking for infallible arguments Track the skipped rows with NullBuffer::union and collect an argument's columns with collect_columns. Keep each argument's batch field on the argument itself instead of in a parallel list. Evaluate a later argument that can be evaluated for any row over the whole batch, as literals already were, and build a projection only for the arguments that are masked. Leave a column whose index does not change unrewritten. Point the guide and the proto comment at withNullShortCircuit instead of repeating its conditions. * Evaluate a shared subexpression for every row, as Spark's generated code does Spark's subexpression elimination evaluates a subexpression that an operator's expressions share for every row, before the expressions that use it, in projections and aggregations. A shared ANSI cast therefore raises in Spark even on rows whose array is NULL. Before serializing an operator, ask Spark's EquivalentExpressions which subexpressions its expressions share, and leave unmasked any call whose later argument holds one, so it raises there as well. * Evaluate a masked argument through one copy, so shuffle keeps its state A masked argument was evaluated through its rebuilt copy, which reads its columns from a batch of just those columns, only for a batch with a NULL in an earlier argument, and through the original for every other batch. Rebuilding an argument whose column indices change gives it new state, so array_union(a, shuffle(b)) restarted shuffle's random sequence each time evaluation moved between the two, and drew different permutations from Spark's. The rebuilt copy now evaluates a masked argument for every batch. A unit test evaluates array_union(a, shuffle(b)) over a batch without a NULL array, one with, and one without again, against the unwrapped function, which draws the same permutations as Spark when b is NULL wherever a is. A Spark 4.0+ SQL fixture reads 8200 rows from one file, so that only the scan's second batch has NULL arrays. Both fail with the previous evaluation. * fix: count LEGACY and TRY string-to-integer casts as infallible, and find shared subexpressions lazily Cast::is_infallible now accepts a cast from Utf8 or LargeUtf8 to an integer in LEGACY or TRY mode, which returns NULL for a string it cannot parse. Comet's parsers for those modes never raise. With ANSI off, the native planner then leaves array_contains(ids, CAST(s AS INT)) unwrapped, and CASE keeps its eager path for such a cast. withSharedSubexpressions now runs EquivalentExpressions on the operator's first canShortCircuitNulls call instead of for every converted operator. canShortCircuitNulls skips it entirely when every later argument is a leaf, which subexpression elimination never shares. array_null_short_circuit_non_ansi.sql runs the array shapes with ANSI off. The casts return NULL there, while a slice start of 0 and a rand argument still depend on the short-circuit. The shuffle fixture pins spark.comet.batchSize, and the map fixture covers map_from_arrays, which its CASE WHEN guards already protect. * perf: keep CASE lazy for string-parsing casts, and give NullShortCircuit its own check Counting a LEGACY or TRY string-to-integer cast as infallible also put CASE and IF on their eager path, which parses every row for each branch that some row chooses. With a selective predicate, that made IF about 8x slower and a three-branch CASE about 3x slower. Cast::is_infallible keeps its original set: casts that cannot fail and cost little more than reading their input. Cast::cannot_fail adds the LEGACY and TRY string-to-integer casts. case_when.rs exposes both checks. CASE uses is_infallible, so it chooses the path it chose before. NullShortCircuit uses cannot_fail, since it masks an argument only to keep it from failing on rows that Spark skips. The conditional bench gains IF and CASE shapes whose branches parse strings. * fix: find shared subexpressions where shuffle keys and TakeOrderedAndProject are serialized The native shuffle writer serializes its partitioning keys, and CometTakeOrderedAndProjectExec its sort keys and projection, when the plan runs, outside the operator conversion that sets the shared subexpressions. canShortCircuitNulls took the missing set as an empty one, so a cast that two partitioning keys share was skipped on the rows whose array is NULL, where Spark's projection evaluates it for every row and raises. canShortCircuitNulls now skips no argument outside withSharedSubexpressions. The shuffle writer calls it with the partitioning keys, which Spark evaluates together in one projection, and CometTakeOrderedAndProjectExec with its own expressions. * fix: scope a TakeOrderedAndProject's sort keys and projection apart, as Spark evaluates them 3090d859a gave CometTakeOrderedAndProjectExec's sort keys and projection one shared set. Spark's ordering evaluates each sort key on its own, without subexpression elimination, and its projection runs separately over the rows the sort keeps. So a cast in two sort keys, or in a sort key and the projection, raised in Comet where Spark returns the rows. The top-K plans now use an empty set, and the projection its projection list.
| Commit: | e7e7198 | |
|---|---|---|
| Author: | Yu Hsun Hung | |
| Committer: | GitHub | |
refactor: remove unused cast compatibility flag (#6135) * refactor: remove unused cast compatibility flag * refactor: drop unused parquet allow_incompat option The flag has no remaining reader after the cast-option cleanup, and the unused CometConf import fails scalafix. * fix: preserve DPP filters when reverting native scans * Fix SparkCastOptions constructor calls * refactor: drop the DPP scan reversion from the cast cleanup * fix: drop the removed cast-option argument from case-when calls * fix: drop the removed parquet-option argument from the page-index test * fix: drop the removed cast-option argument from the cast_nested bench #6648 added this bench after the last merge of main into this branch, and it still passed the third argument that this PR removes from SparkCastOptions::new. --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Andy Grove <agrove@apache.org>
| Commit: | 6960a48 | |
|---|---|---|
| Author: | Andy Grove | |
| Committer: | GitHub | |
fix: truncate per-row date_trunc formats with the literal-format kernel (#6354) * fix: truncate per-row date_trunc formats with the literal-format kernel * fix: pass the SECOND/MILLISECOND overflow policy through the format-column path * fix: handle a NULL format and a literal timestamp with a format column in date_trunc
| Commit: | e8c7ee7 | |
|---|---|---|
| Author: | Peter Lee | |
| Committer: | GitHub | |
fix: honor Spark’s commit protocol in Spark 3.x native writes (#4746) * fix: honor Spark’s commit protocol in Spark 3.x native writes * ci: register legacy native write commit protocol tests * refactor: drop the native Parquet writer's work_dir file naming Both native writers now take the exact file name from Spark's commit protocol, so nothing sets work_dir, job_id or task_attempt_id any more. Remove the Some(work_dir) arm from ParquetWriterExec::execute, reserve the three proto fields, and correct the comments that still described the Spark 3.x writer naming its own files. * fix: handle empty inputs in the Spark 3.x native writer Mirror CometWriteFilesExec and FileFormatWriter: swap a zero-partition child for a single empty partition so the output still carries a schema (SPARK-23271), and only write a file from partition 0 or a partition that has rows. Move createTaskContext to the companion object so the task closure no longer captures the exec node and its child subtree. The SPARK-23271 and empty-partition tests now run on Spark 3.x too. * fix: match Spark 3.x write admission and cover empty scans * docs: say native writes over an empty relation stay native With the empty-relation guard gone from CometDataWritingCommand, the user guide and the CometWriteFilesExec comment still described Parquet writes over an empty relation falling back to Spark's writer. Both native write paths now swap a zero-partition input for one empty partition, so such a write stays native and still leaves a readable schema-only file. * fix: take the Spark 3.x native write codec from the job configuration The Spark 3.x task named its file after the codec in the job configuration, because getFileExtension reads it from there. The native writer used the codec that convert resolved at planning time. The two agree today only because both copy the precedence rule of ParquetOptions. Read the codec with CodecConfig.from(taskContext) next to setOutputPath, as CometWriteFilesExec.executeTask does, so the file name and the footer agree by construction. A new CometNativeWriteSuite test sets a job codec that differs from the planned one and checks the name and the footer. The mixed-case codec test now checks the file name on Spark 3.x too. * test: make the injected cleanup failure throw from SQLMetric.add too Since #6416, CometMetricNode reports a sum metric with add instead of set. The test metric threw only from set, so the two cleanup-failure tests in CometNativeWriteSuite stopped injecting their failure and failed on Spark 3.4 and 3.5. * ci: pre-compile Spark SQL test classes in the manual writer workflow The Parquet writer test step caps the sbt heap at 1 GB for the forked test JVM, but this workflow compiles Spark first, and sbt ran out of heap while it compiled sql/core. Compile the SQL test classes in an earlier step with a 3 GB heap. This branch had the same step in July, before a history rewrite dropped it. * test: build CometNativeWriteExec with the merged constructor The merge with main brings in two callers written against the old constructor. #5957's RevertNativeForTransitionHeavyStagesSuite relied on the removed committer and jobTrackerID defaults, and #6247's write_reserving_from helper still passed the removed work_dir, job_id and task_attempt_id arguments to ParquetWriterExec::try_new. CleanupFailingNativeWriteExec also needs #5957's originalPlan argument. --------- Co-authored-by: Andy Grove <agrove@apache.org>
| Commit: | 7d29453 | |
|---|---|---|
| Author: | Andy Grove | |
| Committer: | GitHub | |
fix: match Spark 4.2 date_trunc overflow for SECOND and MILLISECOND (#6740) * fix: match Spark 4.2 date_trunc overflow for SECOND and MILLISECOND Spark 4.2.0 (SPARK-56663) raises `long overflow` when SECOND or MILLISECOND truncation falls below the smallest timestamp, where earlier versions wrap the Long subtraction. The serde now tells the native kernel which behavior to follow, and the wrapping fixture from #5956 is split into a 4.1-and-earlier file and a 4.2 file. * perf: keep values-only truncation for safe non-UTC SECOND/MILLISECOND dictionaries SECOND and MILLISECOND ignore the timezone, so when they do not wrap they need key masking only for a value near the lower bound. Previously a non-UTC dictionary fell through to the unconditional non-UTC masking branch and scanned every key.
| Commit: | a2b688b | |
|---|---|---|
| Author: | Andy Grove | |
| Committer: | GitHub | |
feat: custom Rust UDFs via arrow-ffi [experimental] (#4459) * feat: custom Rust UDFs via arrow-ffi (alternative to #4283) [skip ci] Adds custom Rust scalar UDF support using arrow-only FFI surfaces, as suggested by paleolimbot and timsaucer in feedback on #4283. This is an alternative implementation for comparison; see #4283 for the bespoke-ABI version. Two ABI flavors are provided side-by-side so reviewers can compare: 1. C ABI (sedona-style): pure C-callable struct of function pointers, parameterized only by Arrow C Data Interface (FFI_ArrowSchema / FFI_ArrowArray). Decoupled from datafusion versions; future-portable to C/C++. Modeled on apache/sedona-db's SedonaCScalarKernel header. 2. datafusion-ffi (FFI_ScalarUDF): wraps user's ScalarUDFImpl as FFI_ScalarUDF. Inherits full ScalarUDFImpl surface (variadic signatures, type coercion, metadata-aware return types) for free, at the cost of a major-version pin against datafusion-ffi. A single library may export either or both; loader walks both discovery functions. `comet-test-udfs` exposes `add_one_c` (C ABI) and `add_one_df` (datafusion-ffi) and the e2e suite drives both through Spark. Scope is intentionally minimal for comparison: scalar-only, happy-path e2e tests only (no panic/error/signature-mismatch coverage). The Scala JVM API mirrors #4283 exactly so the comparison is apples-to-apples. Pieces: - native/comet-udf-sdk: SDK with both ABI flavors + export macros - native/comet-test-udfs: cdylib exposing one UDF per ABI - native/core/src/execution/rust_udf: loader, cache, ImportedCScalarUdf - native/core/src/comet_rust_udf_bridge.rs: JNI for validateLibrary/listUdfs - native/proto: RustUdfCall message - spark/.../udf: CometRustUDF.register, registry, exception classes, JNI bridge stub - spark/.../serde/CometScalaUDF: dispatch ScalaUDF to RustUdfCall when udfName is in the registry * feat: drop datafusion-ffi flavor, contain panics at the FFI boundary, add user guide Keep a single UDF ABI built only on the Arrow C Data Interface. Carrying two public ABIs would mean two permanent compatibility promises; the datafusion-ffi flavor pins every user cdylib to Comet's DataFusion major version, which the 53 -> 54 upgrade demonstrated by removing as_any from ScalarUDFImpl. The rationale is recorded in comet-udf-sdk's crate docs. Contain panics at every extern "C" entry point. A panic escaping one aborts the process, taking down the executor JVM and every task on it. User UDF code is arbitrary and panicking is idiomatic Rust, so a panic is now converted to a query error via the existing get_last_error channel. Previously only execute was guarded; init, new_impl, the discovery entry point and the release callbacks were not. Add a user guide page describing the feature as experimental, including the executor-side library placement requirement and the trust implications of loading native code. * refactor: remove the write-only spark.comet.rustUdfs conf propagateConf wrote the registered UDF set into spark.comet.rustUdfs on every register call, and nothing ever read it. Executors resolve the library from the library_path carried in the RustUdfCall proto, so the conf implied a propagation mechanism that does not exist. Document what actually happens on the register scaladoc instead. CometRustUdfRegistry.snapshot had no other caller and goes with it. * feat: cover all non-nested Spark types, validate the declared return type Add echo_c (identity over any type) and stringify_c ((any) -> Utf8) to the test cdylib, and drive both over every non-nested Spark type with a null row: bool, the four int widths, float, double, decimal, string, binary, date, timestamp and timestamp_ntz. echo_c proves the array survives the round trip with its type and nulls; stringify_c forces the UDF to decode the values rather than hand the array straight back. No ABI change was needed, since the FFI surface is type-agnostic, but nothing had demonstrated that beyond Int64. Also validate at plan time that the return type declared through CometRustUDF.register matches what the kernel's return_field reports, naming both types and pointing at register. Registering decimal(10,2) for a column Spark had widened to decimal(11,2) previously surfaced as a bare DataFusion type assertion partway through execution. Complex types (array, struct, map) remain future work and are documented as unsupported. * feat: support complex types, compare return types ignoring nested nullability Arrays, maps and structs work over the ABI, including nesting (array of struct, struct of array), so extend the type matrix to cover them alongside the non-nested types. No ABI change was needed; the FFI surface carries child arrays already. The new plan-time return type check rejected them, and was right to look: Spark carries containsNull and field nullability inside the declared type, so array<int> with containsNull=false converts to a List with a non-nullable child, while the array actually delivered to the UDF has that child normalized to nullable. Compare with nested nullability erased, and promise DataFusion the kernel's own type rather than the declared one so its exact-match assertion compares like with like. The comparison stays strict about everything that changes how bytes are read: decimal precision and scale, timestamp unit, struct field names and order. Also document the return type model, which came up in review. A kernel has no fixed return type: return_field derives one from the argument types on every call, and echo_c serves all 19 types in the matrix. What is fixed is the type declared to register, because Spark needs a concrete DataType to plan against, and that is per-registration rather than per-kernel. * ci: run CometRustUdfSuite in the PR builds The suite cancelled itself whenever -Dcomet.test.udfs.lib was unset, and nothing set it, so it had never run in CI and the ABI had never been exercised against a Linux .so. Have the suite locate the cdylib under native/target instead of requiring the flag, keeping the property as an override. When the library is missing the suite still skips locally, but fails in CI, where its absence means the native build or the artifact upload changed rather than that someone forgot a flag. An undefined Maven property arrives in the forked JVM as the literal string "null", which is why the override needs filtering rather than a plain null check. Upload libcomet_test_udfs alongside libcomet from both native build jobs: the same cargo invocation already produces it, since the crate is a workspace default member, and the test jobs download the artifact rather than building. Register the suite in the expressions bucket of both workflows, as dev/ci/check-suites.py requires. * style: drop redundant string interpolators in the Rust UDF messages scalafix RedundantSyntax flags s-prefixed literals with nothing to interpolate. These were latent: the previous head commit on this branch carried [skip ci], so the lint jobs had never run against these files. * fix: drop a loaded UDF library after its kernels, not before LoadedLibrary declared `library: Arc<Library>` ahead of `udfs`. Struct fields drop in declaration order, so dropping a LoadedLibrary ran dlclose first and then invoked each kernel's `release` function pointer, which lives in the text of the library just unloaded. Nothing else holds a clone of that Arc. Unreachable in production, since the process-wide cache never drops a LoadedLibrary, but the loader tests build one via `load()` and drop it. Reorder the fields and say why in a comment, so a later edit does not quietly reintroduce it. * fix(ci): build the test UDF cdylib before the Rust test run The rust-test job runs `cargo nextest run` and nothing else. comet-test-udfs is `crate-type = ["cdylib"]` with no test targets, so a test build compiles the crate without ever emitting libcomet_test_udfs, and the three rust_udf tests that dlopen it failed on a clean checkout: execution::rust_udf::cache::tests::same_path_returns_same_arc execution::rust_udf::loader::tests::all_exported_kernels_are_discovered execution::rust_udf::loader::tests::load_test_udfs_succeeds It passed locally only because a previous `cargo build` had left the artifact in target/debug. Build the crate explicitly before the test run, and replace the bare `expect("load")` with a message naming that command, so the next person to hit this reads the fix instead of a dlopen error. * fix: reject deterministic = false when registering a Rust UDF `CometRustUDF.register` accepted a `deterministic` flag, installed a nondeterministic Spark stub for it, and carried it in the RustUdfCall proto, but nothing on the native side ever read it: ImportedCScalarUdf hardcodes Volatility::Immutable. A UDF the caller declared nondeterministic was therefore planned as pure and could be constant-folded, evaluated once and reused, or eliminated as a common subexpression. Honoring the flag needs the cached per-library signature to become per registration, which is #5249. Until then, fail the registration rather than let the parameter lie about what Comet does with it. Raised in review: does the proto field's value mean RustUdfs are always immutable? * docs: clarify the Rust UDF ABI's stability, purity, and path resolution Review follow-ups on the SDK and the user guide, plus two assertion tweaks: - State that the C ABI structs are specific to one Comet version and are internal under the versioning policy, while the host's own arrow and datafusion versions need not match the cdylib's. - Say on CometCScalarUdf that only immutable functions are supported, and add the same to the user guide's limitations. The guide previously told readers to pass `deterministic = false` for impure functions, which is now refused. - Describe what libraryPath actually does: an absolute path is the sensible choice, but a bare name resolves through the platform loader search path, and neither is a security boundary. - Note that CometRustUDF is deliberately not @Public, so it carries no compatibility guarantee. - Report rather than swallow a panic caught in a release callback: there is no error channel there, so it goes to stderr with a context string. - Check `release.is_some()` alongside private_data in the FFI debug asserts, which catches a call made after release rather than only an uninitialized one. * review: address mbutrovich's first pass on the Rust UDF path Drop the dead code and vacuous round-trip the review found, correct the comments that were wrong, and pin the error classification with tests. - `validateLibrary` returns void. It only ever reported back the name it was given, since the native side already filters `lib.udfs` by that name and errors when it is absent, so the JSON construct/parse cycle and the `require(described.name == name)` that followed it could not fail. `listUdfs` had no caller and comes out with it. - `classifyNativeError` matched a bare "ABI", which every loader message can contain because they all interpolate the library path: a library under a directory named `ABI` had its "failed to open" reported as an ABI mismatch. Match the message wording instead, and add a test per failure mode so a rewording on the native side fails a test rather than silently changing the exception a caller sees. - Publish the registry entry before installing the catalog stub. The stub is what makes the name resolvable to the analyzer, so the old order left a window where a query could plan the name with no registry entry and hit the stub's "not evaluated" exception. - Correct the `Mutex` comment in `ImportedCScalarUdf`. It claimed DataFusion serializes invocations of a `ScalarUDFImpl` per batch; it does not, and the planner hands every task the same instance out of the process-wide cache, so concurrent calls are the normal case. The lock is load-bearing rather than defensive, and it serializes all batches of a UDF in a process. Relevant to #5252, which would remove it. - Document `scalar_args` as reserved and always NULL in ABI v1. Nothing on the host can populate it: literals are expanded to full-length arrays before a kernel sees them. - Justify the `CometRustUdfRegistry` singleton's lifetime and bounds where the contributor guide asks for it, and record that session scoping is unresolved. - Add the tests the review asked for: a concurrent-registration test, and a name-collision test which is `ignore`d because it fails today. A Rust UDF does answer calls to an ordinary Scala UDF registered under the same name. The obvious fix does not work: `functions.udf` wraps the closure it is given, so the object in `ScalaUDF.function` is Spark's and not one Comet can recognize, and the `udf(AnyRef, DataType)` overload that would have preserved identity is gone in Spark 4. * docs: point the Rust UDF follow-up comments at their tracking issues Replaces the review-thread links added in the previous commit, and adds pointers where the code has a known gap but carried no reference: - #5294 registry session scoping - #5295 a Rust UDF answers an ordinary Scala UDF of the same name - #5296 the adapter rebuilds the impl and re-resolves the return type per batch - #5297 the library cache holds its write lock across dlopen * refactor: name the UDF mechanism for what it is, not for Rust The ABI this feature loads a UDF through is a set of `#[repr(C)]` structs of function pointers carrying Arrow C Data Interface arrays. No DataFusion type and no Rust type appears in it, the entry points are unmangled C symbols, and every allocation is freed through a callback the library supplies, so nothing about the host requires the library to have been written in Rust. The Rust SDK is one client of that ABI, not the ABI itself. The names said otherwise, and one of them is a wire format that would have been awkward to change later: - proto `RustUdfCall` -> `NativeScalarUdf`, field `rust_udf_call` -> `native_scalar_udf`, which also parallels the existing `JvmScalarUdf` for the JVM codegen dispatch path. Field number 75 is unchanged. - `CometRustUDF` -> `CometNativeUDF`, and the registry, bridge, metadata, exceptions and suite alongside it. - Rust internals move to `execution::c_udf`, matching the C-ABI names already there (`CometCScalarKernel`, `comet_c_udf_list_v1`), and the bridge file to `comet_native_udf_bridge.rs`, matching the JVM class it serves. - User-facing messages and comments say "native UDF" where they mean the mechanism, and keep "Rust" where they mean the SDK. Unchanged on purpose: `comet-udf-sdk` and `comet-test-udfs` were already neutral, and `rust_udfs.md` documents the Rust SDK specifically, so a `cpp_udfs.md` would sit beside it rather than replace it. Also narrows the claim that went with the old naming. The user guide said the ABI "is implementable from C or C++", which reads as an offer; it is true of the design but nothing supports it, since no header is published, the layouts are only defined in Rust source and are not stable across releases, and the SDK's panic guards have no automatic equivalent in another language. The docs now say that plainly. Drive-by: the `NativeScalarUdf` proto comment still described dispatch through "whichever ABI flavor (C ABI / datafusion-ffi)" the library registered under. The datafusion-ffi flavor was removed earlier in this branch. * perf: drop the per-process mutex serializing native UDF batches The adapter held one `Mutex` across the whole `invoke_with_args` body, including the user's compute. Every Spark task in an executor shares one `ImportedCScalarUdf` through the process-wide library cache, so a UDF over a wide scan ran at roughly one core no matter how many tasks were in flight. The lock is not needed. Both kernel-level callbacks the adapter reaches, `function_name` and `new_impl`, take `*const CometCScalarKernel`, and the SDK's exporter only reads through that pointer: everything mutable is the per-call `CometCScalarKernelImpl`, which never leaves the stack frame that built it. State the concurrency requirement plainly in the ABI docs, on the `CometCScalarUdf` trait and in the user guide, and pin it with a test driving eight threads through one shared adapter. * fix: stop rejecting a native UDF over positional list and map field names Two type-mapping traps, both reachable by following the user guide. The guide listed `TimestampType` and `TimestampNTZType` as the same Arrow type. They are not: Comet maps the former to `Timestamp(Microsecond, Some("UTC"))` and the latter to `Timestamp(Microsecond, None)`, and the timezone is compared exactly, so a UDF that built a `TimestampMicrosecondArray` without calling `with_timezone` was rejected at planning against a `TimestampType` registration with no hint as to which axis disagreed. The table now shows the tag, the error message names the axes that must match exactly, and `make_ts_utc_c` / `make_ts_naive_c` pin both directions. The map case was a real rejection over a spelling. Comet emits a map's children as `entries` / `key` / `value` while arrow-rs's `MapBuilder::new(None, ..)` names them `entries` / `keys` / `values`, and the comparison was strict about names at every level. Those names are positional in the Arrow columnar format and Comet's own `CometMapVector` reads them by index, so they are now normalized before comparing, as is a list's element name. Struct field names stay strict: they are part of the Spark type and are how a caller addresses the result. `echo_c` could not have caught either one, since it hands back the array Comet gave it and so inherits Comet's own tag and names. `make_map_c` builds its output through arrow-rs's default builder instead. * test: cover native UDFs in filter, join, grouping key and window Every end-to-end test called the UDF from a projection. Filters, join conditions, grouping keys and window partitioning each route through a different Comet operator, so a regression in the rule wiring for any of them would have shown up only as a silent fallback to Spark. The catalog stub is what makes these assertions load-bearing: it throws if Spark ever evaluates the UDF itself, so each test fails rather than passing on the JVM path. * refactor: expose register and get on the CometNativeUdfRegistry object Callers reached through `CometNativeUdfRegistry.instance` to get at the process-wide registry, which puts the singleton in every call site for no benefit. Forward both methods from the companion so `instance` stays an implementation detail. * test: load the concurrent adapter test's library through the cache concurrent_invocations_share_one_adapter took its adapter from a LoadedLibrary returned by `load` and let that struct drop before the threads ran. The adapter holds no reference to its library, so the drop dlclosed it. On Linux that unmaps the library, and the first `new_impl` call jumps into unmapped text. The Linux rust-test job has failed on both pushes since the mutex was removed, and the 2026-09-07 log shows this test dying with SIGSEGV. macOS keeps an image with thread-locals mapped after dlclose, so it passed locally. Go through `cache::get_or_load` instead, which is how the planner gets an adapter and which never unloads. Also correct the `library` field doc, which said the library is never unloaded when only the cache makes that true. * fix: refuse a UDF result whose type disagrees with return_field An FFI_ArrowArray carries no type, so the host imports a result as whatever return_field declared. A UDF declaring Int64 that returned an Int32Array was read as the wrong values, or past the end of the buffer. Check the result in the SDK, the last point that knows both types, and correct the crate docs that still promised cross-release compatibility and C/C++ support. * fix: keep native UDF libraries loaded and plan canonical child names The adapter now holds an Arc<Library> declared after its kernel, so it cannot outlive the library its callbacks live in. The planner promises the kernel's return type with list and map children renamed to Comet's names, and the adapter relabels each result to match, so a map built with MapBuilder defaults combines with Comet's own maps in if/CASE. Adds sub_c and tests for scalar arguments. * test: cover multi-argument, literal and if-combined native UDF calls Also removes CometNativeUdfSignatureException, which was never thrown. * docs: correct the Rust UDF guide's dependency, panic and reload advice * feat: refuse native UDF calls whose argument types differ from the registered ones The catalog stub is untyped, so Spark inserts no casts and inputTypes was only used for its length. The serde now compares the call's argument types with the registered ones, ignoring nullability, and fails at planning time naming both. Closes the open question from the self-review on what inputTypes means. * fix: release the native UDF cache's read guard before taking the write lock `get_or_load` looked up the canonical path in an `if let` scrutinee and took the write lock inside its body to record the new path. The scrutinee's read guard lives until the end of that block, so a first lookup through a path that resolves to an already-loaded library, such as a symlink, waited on its own read lock forever. Two threads racing on a library's first load through a non-canonical path could reach the same branch. The lookup is now a statement of its own. The new test looks a loaded library up through a symlink on a separate thread, so a regression fails it after 30 seconds instead of hanging the run. * fix: widen a native UDF's nested nullability before it enters the plan The planner promised DataFusion the kernel's own return type, with only list and map child names canonicalized. A kernel may report a nested field as non-nullable where the registration declares it nullable: the declared-vs- reported check disregards nested nullability, so this is accepted. An operator that takes its type from the UDF would then have to narrow a nullable field from another expression to match, and that cast fails with `Cannot cast nullable struct field 'a' to non-nullable field`. `canonicalize_child_names` becomes `promised_return_type`, which also makes every nested field nullable except a map's entries and keys, which Arrow requires to be non-null. The adapter accepts a result whose promised form matches and conforms it with a cast that only renames fields and widens their nullability, reusing the buffers. `make_struct_c` in the test library reports a non-nullable struct field. The planner and adapter tests pin the widened type. The suite runs the reported `if` query in both branch orders. Main's IF branch reconciliation (#6458) now covers that query too, so the planner change keeps other consumers consistent. * fix: ignore struct field metadata when checking native UDF argument types `checkArgumentTypes` compared `deepNullable` forms, which keep each `StructField`'s metadata. A struct argument whose field carries a comment was refused against a registration with the same field name and type, with an error naming `struct<a:int>` on both sides. `DataTypeSupport.equalsIgnoreNullability` re-derives Spark's `DataType.equalsIgnoreNullability`, which compares struct field names but neither nullability nor metadata. Spark's own is `private[sql]` on 3.4. The new test passes a struct whose field has a comment, built with `struct(col.as(name, metadata))`. * fix: discard native UDF registration results explicitly for the strict-warnings build `spark.udf.register` returns the registered `UserDefinedFunction` and `ConcurrentHashMap.put` returns the previous value. Both were discarded implicitly at the end of a `Unit` method, which `-Ywarn-value-discard` under `-Pstrict-warnings` rejects as `discarded non-Unit value`. The fatal error also cut scalac short, which is where the job's four errors in untouched shuffle files came from. * docs: pin the Rust UDF SDK to the release the docs were built for The example dependency pinned `tag = "1.1.0"`, which predates the SDK, so it could not resolve. It now uses `$COMET_VERSION`, which the docs build replaces with the release a page was built for. The development docs, whose version has no tag, say to pin `rev` instead. * fix: recognize native UDFs by their function instead of by name `CometScalaUDF` recognized a native UDF by looking the call's `udfName` up in `CometNativeUdfRegistry`, a process-wide map keyed by bare name. So an ordinary UDF registered under a native UDF's name was answered out of the native library, returning wrong values with no warning (#5295). A registration in one session also claimed the name in every other session sharing the driver JVM (#5294). `register` now installs a builder in the session's own function registry, as `spark.udf.register` does. The builder creates a `ScalaUDF` that holds a `CometNativeUdfFunction`, which carries the registration, and the serde matches on that function. An ordinary UDF registered under the same name holds its own function and stays on the JVM path. A registration is a temporary function of the session it was made in. The process-wide registry is gone. `spark.udf.register` could not do this, because `functions.udf` wraps the function it is handed, and the overload that did not is gone in Spark 4. The builder checks the argument count itself, which the fixed-arity stub used to do implicitly. `ScalaUDF` casts its function to the `FunctionN` of its arity, so there is one subclass per arity, up to the existing limit of four. The name-collision reproduction is enabled. A new test checks that another session neither sees the registration nor has its own UDF of that name answered natively. * refactor: register native UDFs as session-scoped Catalyst expressions Match the registration in #6697. A native UDF's calls now resolve to a `NativeUdfCall` expression rather than a `ScalaUDF` holding a marker function, and `CometNativeUdfCall`, a serde of its own, emits the `NativeScalarUdf`. `CometScalaUDF` and `DataTypeSupport` are back to main's versions. - Spark's analyzer checks the argument types (`ExpectsInputTypes`), disregarding nullability and field metadata, and inserts no casts. A mismatch fails analysis instead of Comet's planning, so Comet's own type check, `equalsIgnoreNullability` and `CometNativeUdfArgumentTypeException` are gone. - A call with the wrong number of arguments fails analysis with Spark's own error. - Nothing caps the arity any more. Four was the most `ScalaUDF`'s `FunctionN` casts allowed for, so `sum_c` and a test cover a five-argument call. - The codegen dispatcher declines an ordinary UDF whose arguments hold a native call, so Spark gets that operator and the failure names the real cause. - If Spark evaluates the call, it throws `CometUdfNotEvaluatedException`. `ShimSessionFunctionRegistry`, `CometUdfErrors` and `CometUdfExceptions` are byte-identical copies of #6697's, so whichever pull request lands second merges them without a conflict.
| Commit: | d16f7bb | |
|---|---|---|
| Author: | Andy Grove | |
| Committer: | GitHub | |
feat: support SortAggregateExec (#4565) * feat: support SortAggregateExec [skip ci] Wire SortAggregateExec through the existing CometBaseAggregate.doConvert path so that queries planned with useObjectHashAggregate=false (or with TypedImperativeAggregate functions whose buffer formats prevent hash aggregation) can run their Partial->Final pair natively instead of falling back to Spark. CollectSet was the motivating function; the same wiring covers any other natively-implemented TypedImperativeAggregate. No proto changes: DataFusion's AggregateExec::try_new auto-detects InputOrderMode::Sorted from the child's output ordering, so a sorted input naturally produces sorted output. Wrapper reuse: createExec produces a CometHashAggregateExec (matching CometObjectHashAggregateExec). CometExec.outputOrdering already defaults to originalPlan.outputOrdering, so SortAggregateExec.outputOrdering flows through unchanged for downstream operators that elided a sort against it. Cleanups landed in passing: - Lifted baseAggregateSupportLevel and adjustOutputForNativeState onto the CometBaseAggregate trait (was duplicated 3x and 2x). - Collapsed isAggregate / findCometPartialAgg's Spark-side branches / canAggregateBeConverted's shuffle guard to BaseAggregateExec. * test: add comprehensive SortAggregate coverage via SQL file tests Add a SQL file test (sort_aggregate.sql) that forces collect_set through SortAggregateExec by disabling ObjectHashAggregate, covering global and grouped aggregation, multiple grouping keys, expression keys, NULL/empty/ single-row groups, mixed aggregates, multiple collect_set, DISTINCT, HAVING, a representative data-type spread, and a clean fallback when the aggregate input is incompatible. Add a global (no grouping keys) plan-shape assertion to CometAggregateSuite alongside the existing grouped one, since the SQL framework cannot assert that a SortAggregateExec was actually planned. * fix: distinguish CometSortAggregateExec and update tests for SortAggregate support Introduce a dedicated CometSortAggregateExec wrapper instead of reusing CometHashAggregateExec for converted SortAggregateExec nodes, so the executed plan reflects whether Spark planned a hash or a sort aggregate. A shared CometBaseAggregateExec base carries the common rendering and serialization, and findCometPartialAgg matches the base type so a partial sort aggregate (e.g. collect_set) is still paired with its final aggregate instead of falling back. Update the Spark test diffs to recognize the new wrapper: ReplaceHashWithSortAgg counts CometSortAggregateExec as a sort aggregate, and the generic aggregate collectors in AdaptiveQueryExecSuite and StreamingAggregationDistributionSuite match CometBaseAggregateExec so they see through both wrappers. Refresh the affected SQL file tests now that SortAggregate runs natively: - min_max.sql: string min/max still falls back, but the reason is now the StringType limitation rather than SortAggregate being unsupported. - first_last.sql: the multi-type first/last IGNORE NULLS queries now run natively; shape test_types so each group has a single non-null value, since first/last are non-deterministic across engines with multiple non-null rows. * fix: widen post-merge aggregate guards to CometBaseAggregateExec Self-review after merging main. Three safety predicates landed on main after this branch was cut and matched only CometHashAggregateExec, so a CometSortAggregateExec Partial slipped past all of them: - CometExecRule.preserveSparkAggregateBuffers: a Celeborn exchange that falls back to Spark must restore the Comet Partial below it. Missing the sort variant let a native collect_list buffer reach a Spark Final. Added a regression test in CometCelebornShufflePlanningSuite that fails without this change. - RevertNativeForTransitionHeavyStages.hasUnsafeMixedAggregateAtStageBoundary: same class of guard at a stage boundary. - CometMetricNode.withoutAggregateMetrics: range-partition sampling would double-count sort-aggregate spill and memory metrics. Also drop the duplicate adjustOutputForNativeState that the merge left behind (main grew its own copy handling CollectList and PartialMerge; the branch's older copy handled neither), move main's aggregate metrics override onto the shared CometBaseAggregateExec base, rename COMET_EXEC_SHUFFLE_ENABLED to COMET_SHUFFLE_ENABLED, and mark SortAggregateExec supported in operators.md. * refactor: share the aggregate serde and plan node equality CometBaseAggregate is now parameterized on the Spark aggregate type and supplies enabledConfig, getSupportLevel and convert for all three aggregate serdes. Each serde object keeps only its operator-specific support check, a requiresCometShuffle flag and createExec. The shared getSupportLevel notes that the partial and final test knobs gate sort aggregates too. equals and hashCode move into CometBaseAggregateExec and require the same class, so the two wrappers keep only withNewChildInternal. * fix: walk through sorts when pairing sort aggregate buffers Spark puts a SortExec between each SortAggregateExec and the exchange below it. The passes that pair a buffer-consuming aggregate with the Partial that produces its buffers stopped at that sort, so a native collect_list Partial could stay native under a Spark Final or merge stage. Spark then read Comet's array buffer where it expected binary, and a distinct collect_list chain slipped past the issue #4724 guard and failed natively. findPartialAggInPlan, revertUnsafePartialAggregates and hasUnrepairedNativeBuffer now walk through SortExec and CometSortExec. * fix: keep sort aggregate output in grouping-key order natively CometSortAggregateExec reports SortAggregateExec's grouping-key output ordering, and Spark may remove sorts above it on that basis. DataFusion emits groups in input order only when it sees the input sorted on them. When the ordering comes from outside the native plan, as with a cached sorted relation read through an unordered scan, its vectorized grouping can emit groups out of order: NULL and an empty array hash alike, so an ORDER BY over array keys returned [1] before []. A new HashAggregate field, ordered_by_grouping_keys, is set for sort aggregates. The native planner then sorts the aggregate output on the grouping columns unless the aggregate's output ordering already satisfies them. It also normalizes float grouping keys the way native sorts do, so that DataFusion sees the sort below a float-keyed final aggregate and streams it instead of falling back to the hash path. * test: run the collect_set fixtures through SortAggregateExec collect_set.sql and collect_set_floating_fallback.sql gain a ConfigMatrix over useObjectHashAggregateExec, so their whole fixtures run through both ObjectHashAggregateExec and SortAggregateExec. sort_aggregate.sql keeps only the shapes those files do not cover (several grouping keys, an expression key, hashable aggregates riding along) and adds the decimal SUM fallback. Its floating-point fallback query, which expected a fallback that Spark 4.2 no longer has, moves to the matrix. A new fixture checks that SortAggregateExec stays in Spark, with the shuffle reason, when Comet shuffle is disabled. min_by.sql and max_by.sql drop the claim that Comet never converts SortAggregate and check the type fallback without the TypedImperativeAggregate too. The two native collect_set Scala tests become one test over both queries. * docs: note how first and last can differ under sort aggregation Spark plans a sort aggregate for first and last over a column whose buffer cannot use hash aggregation, such as a string, and Comet now runs it natively. Within a group the row order is not defined, and Spark and Comet can order rows with equal grouping keys differently, so a group with more than one candidate value can return a different, equally valid, value. * test: run the mixed-engine collect_list tests with sort aggregates Each mixed-engine collect_list test now also runs with useObjectHashAggregateExec=false, so both halves of the split are covered for SortAggregateExec, where a sort sits between each half and its exchange. The tests count every Comet aggregate wrapper rather than only CometHashAggregateExec, and check which aggregate operator Spark planned. * fix: keep decimal AVG with a maximum-precision sum out of sort aggregates Spark's Average keeps its decimal sum at precision p + 10, capped at 38. When a sort aggregate's buffer has a field that is not mutable, such as a string FIRST, Spark holds that sum in a generic row where it is unbounded, so 0.6 + 0.6 - 0.4 over DECIMAL(38,38) recovers to 0.8. The native AVG records the overflow at 1.2 and returns NULL, or raises under ANSI. These queries stayed in Spark before sort aggregates converted. CometSortAggregateExec now declines a decimal AVG or TRY_AVG whose sum is at maximum precision, next to the existing decimal SUM check. The compatibility guide notes both fallbacks. * test: expect the distinct collect_list sort aggregate chain to run natively #4727 added native support for collect_list and collect_set alongside a distinct aggregate, so the PartialMerge stages of the distinct rewrite no longer force the chain back to Spark (#4724). Under sort aggregates the whole chain now runs natively and matches Spark. Check that, and keep the fallback case by checking that the chain stays in Spark, through the sort below each stage, when its final cannot convert.
| Commit: | a0e8b5c | |
|---|---|---|
| Author: | Unik Dahal | |
| Committer: | GitHub | |
feat: support native MergeSummary on Spark 4.1+ (#6640) * feat: support native MergeSummary on Spark 4.1+ * Preserve MERGE summary eligibility through AQE and strengthen parity coverage * Use monotonic cumulative snapshots in the Spark 4.2 retry test CometMetricNode.set adds positive deltas and keeps the highest reported value, so reporting count + 10 and then count made each successful attempt contribute 11. Report the cumulative count twice instead, keeping the failed-attempt injection.
| Commit: | 207fb6f | |
|---|---|---|
| Author: | Unik Dahal | |
| Committer: | GitHub | |
feat: add native support for MergeRowsExec (row-level MERGE INTO) (#5318) * feat: add native MergeRows execution * Address MergeRows review feedback * Review Comments Addressed * Address Review Feedback
| Commit: | 1dcea02 | |
|---|---|---|
| Author: | Andy Grove | |
| Committer: | GitHub | |
feat: add a native Comet version of RangeExec (#6553) * feat: add an experimental Comet version of RangeExec Add CometRangeExec, which writes the values of spark.range and SQL range() straight into Arrow batches on the JVM and feeds them to the native plan, so the operators above a range can run natively without converting Spark rows. Partition bounds and values follow Spark's generated code for RangeExec, including its behavior where the arithmetic overflows. Off by default behind spark.comet.exec.range.enabled. When the Spark-to-Arrow conversion is also enabled, CometRangeExec takes precedence for Range. Adds CometRangeExecSuite and CometRangeBenchmark. * docs: list the experimental Comet RangeExec on the operators page * feat: generate Range values in native code Add a native RangeExec to the operators crate, planned from a new RangeScan protobuf message. Each task computes its partition's bounds from the partition index in i128 arithmetic and walks them with the loop from Spark's generated code for RangeExec, so the values match Spark's default path, overflow cases included. On the JVM, CometNativeRangeExec is a CometLeafExec that reports Spark's partitioning, so the native plan runs one task per slice with no JVM input. The internal spark.comet.exec.range.native.enabled (default true) switches between this and the JVM generator. CometRangeExecSuite runs every test against both, and CometRangeBenchmark has an arm for each. * refactor: keep only the native Range generator Drop the JVM generator (RangeArrowReader) and the internal switch between the two, and name the native operator CometRangeExec. The native generator was faster on every benchmark case. * fix: keep Range on Spark where its interpreted path disagrees with codegen With whole-stage codegen disabled, Spark runs the interpreted RangeExec.doExecute, which returns different rows from its generated code when that code's arithmetic overflows. Comet follows the generated code, so such ranges now fall back when codegen is off. Also list CometRange in the operator reference, mark originalPlan transient as the other native leaves do, and test a global aggregate over an empty range. * fix: fail a Range with no slices as Spark does Spark does not validate a range's slice count. A non-empty range with fewer than one slice fails when Spark runs it, while Comet would plan no tasks and return no rows, so such ranges now stay on Spark. The Spark-to-Arrow conversion now steps aside for Range only where CometRangeExec supports the range, so a range it declines can still use that conversion. * refactor: simplify CometRangeExec and generalize leaf precedence - Build the native range on DataFusion's LazyMemoryExec with a LazyBatchGenerator, as DataFusion's own generate_series does, instead of a hand-written ExecutionPlan and stream. This also gives it cooperative yielding. The loop now reads like Spark's doProduce, and each batch is sized to the values left. - In CometExecRule, a leaf with an enabled Comet operator now tries that operator before the Spark-to-Arrow conversion, which becomes the fallback for a leaf the operator declines. This replaces the Range special case in shouldApplySparkToColumnar and records the decline reason. - Share the codegen-disabled check, including the NO_CODEGEN factory mode on Spark 3.5 and later, between CometRangeExec and the hash aggregate. - Restore Spark's RangeExec where no native operator consumes the range, since generating it natively would only add a conversion back to rows. - Drop overrides CometExec already provides, and duplicate tests and benchmark validation runs. * refactor: keep overflowing ranges on Spark and generate the rest directly Spark's generated code for RangeExec and its interpreted path, which runs when whole-stage codegen is off or the generated code is too large, disagree only for a range whose arithmetic overflows. Such ranges now stay on Spark whatever the codegen setting, so Comet no longer has to predict which path Spark takes. The check is exact: an element count that does not fit in a long, or a batch span that does not. Every range Comet runs now has the values start + i * step in both of Spark's paths, so the native generator drops the emulation of Spark's 1000-value batches and computes them directly. The codegen-disabled helper shared with the hash aggregate is no longer needed, so the aggregate is back to main's version. The config and docs no longer call the feature experimental. It stays off by default because it can be slower than Spark under cheap operators.
| Commit: | 48ec1c0 | |
|---|---|---|
| Author: | Bhargava Vadlamani | |
| Committer: | GitHub | |
feat: wire native existence join support (#4587) * implement_native_existence_joins * implement_native_existence_joins * implement_native_existence_joins_fix_plans * implement_native_existence_joins_fix_plans * implement_native_existence_joins_fix_plans * init_commit * make_doc_changes * make_doc_default_changes * address_review_comments_update_benches * make native join stricter and address review comments * make stricter fallbacks and add further tests * make stricter fallbacks and add further tests lint * make stricter fallbacks and add further tests * make stricter fallbacks and add further tests * fix: use Long literals in CometExistenceJoinBenchmark to satisfy strict Scala warnings * address review: distinct fallback reasons, IN/direct-marker/string-key SQL coverage, plain conf, docs - CometConf: switch existenceJoin flag to a plain conf(...) with a doc listing what still falls back (matches sortMergeJoinWithJoinFilter precedent) - operators.scala: give the flag-off ExistenceJoin case its own fallback reason - operators.md: note native existence support on BHJ/SHJ and what falls back - existence_join.sql: add direct-marker projection, IN-with-OR, and string-key cases; fix the stale 'disabled by default' header - CometJoinSuite: add flag-off fallback test and existence-SMJ-under-forceSHJ fallback test - CometExistenceJoinBenchmark: set spark.comet.exec.onHeap.enabled so both arms measure the native join * remove dead ExistenceJoin arm from CometSortMergeJoinExec.producedAttributes Existence sort-merge joins always fall back to Spark (the SMJ serde has no ExistenceJoin case), so a CometSortMergeJoinExec is never built with that join type. The override only ever returned AttributeSet.empty, which is already the inherited default for a join, so drop it entirely. The reachable hash-join and broadcast-hash-join overrides are unchanged. * make stricter fallbacks and add further tests
| Commit: | 40369a0 | |
|---|---|---|
| Author: | Ping Zhang | |
| Committer: | GitHub | |
feat: push local TopK thresholds into Parquet readers (#6263) * feat: connect local TopK thresholds to native Parquet readers * test: cover TopK reader filtering lifecycle and fallback * perf: benchmark and document TopK reader pruning
| Commit: | 12e8557 | |
|---|---|---|
| Author: | dustin | |
| Committer: | GitHub | |
fix: key each scan's planning data by its plan node in a native block (#6297) Scans in one native block have their planning data collected under a key and merged with toMap, so two scans with the same key both read the last one's files. The Iceberg key (metadata location plus the SparkScan hash) and the Parquet key (source plus a hash of schema, data filters and projection) leave out what can tell two scans of one table apart: storage partitioned join grouping, partition filters and DPP. A storage partitioned self-join with partially clustered distribution returned duplicate rows, and a bucketed Parquet self-join without pushed filters read the wrong partitions. Both keys now include the scan operator's plan_id, which is the id of the Spark plan node the scan was converted from. The merge in findAllPlanData keeps one copy of identical data under a key and fails on different data instead of dropping one side. Closes #6278.
| Commit: | c59021b | |
|---|---|---|
| Author: | Andy Grove | |
| Committer: | GitHub | |
feat: support Spark HyperLogLog sketch functions (hll_sketch_agg, hll_union_agg, hll_sketch_estimate, hll_union) (#4802) * feat: add DataSketches HLL wrapper and Spark compatibility spike Wrap the pure-Rust datasketches crate's HLL_8 sketch/union behind SparkHllSketch/SparkHllUnion, hashing inputs via the crate's hash_value wrappers (raw_bytes, sign_extend) so MurmurHash3-x64-128 input matches DataSketches-Java. Verified cross-engine: Comet-produced sketches are byte-identical to Spark hll_sketch_agg output for HLL-array mode and mutually readable for low-cardinality List/Set mode. * feat: add version-specific aggregate serde registration hook * feat: add HllSketchAgg and HllUnionAgg proto messages Adds the two protobuf messages needed to serialize Spark's hll_sketch_agg and hll_union_agg aggregate functions, wired into the AggExpr oneof as field numbers 19 and 20. * feat: add native hll_sketch_agg accumulator and planner arm Wire up HllSketchAgg as an AggregateUDFImpl backed by SparkHllSketch, accepting Int8/16/32/64, Utf8, and Binary inputs and returning a serialized HLL sketch as Binary. Null groups evaluate to NULL, matching Spark's HllSketchAgg. Wires the new AggExprStruct::HllSketchAgg arm into the native planner's create_agg_expr. Also add a placeholder HllUnionAgg planner arm returning a clear "not yet supported" error, since that oneof variant already exists in the proto but its native accumulator lands in a follow-on task. * feat: add hll_sketch_agg Scala serde for Spark 4.x * feat: add hll_sketch_estimate scalar function * feat: mark HLL expressions incompatible and add opt-in end-to-end test [skip ci] * feat: add native and serde for hll_union_agg [skip ci] * feat: add hll_union scalar function [skip ci] * test: add hll_union end-to-end and error tests, apply formatting [skip ci] * fix: HLL empty-group returns empty sketch and size() accounts for sketch heap [skip ci] * ci: run CI for HLL sketch functions * fix: decode compact HLL sketches and keep empty union partials NULL * fix: honour a NULL allowDifferentLgConfigK and reject undecodable HLL_4 hll_union is a TernaryExpression, so a NULL third argument makes the whole call NULL. Separately, an updatable HLL_4 sketch carrying auxiliary-map exceptions is now rejected: the bundled decoder reads the compact aux layout in both forms, so those sketches would decode to a wrong estimate. * fix: document HLL support and address review notes Remaining review feedback on the HLL sketch functions. `docs/source/user-guide/latest/expressions.md` still listed the `hll_*` family under "Not currently planned", so the page contradicted the feature. Narrow that bullet to the families that stay unplanned, add the four rows to the `agg_funcs` and `misc_funcs` tables, and explain why this is the one sketch family Comet accelerates. The Implementation cells were checked against a `generate-docs` run on spark-4.0 and spark-4.1 rather than filled in by hand. Minor notes from the same review: - Drop the dead `update_i32`/`update_i16`/`update_i8` helpers. The accumulator widens with `as i64`, which sign-extends identically. - Downcast once per batch in `HllSketchAccumulator::update_batch` instead of building a `ScalarValue` per row, which copied every string and binary value onto the heap only to hash it and drop it. `every_input_type_hashes_the_same_as_a_direct_update` pins the new dispatch to the old bytes exactly, per input type. - Invalid sketch bytes are user data, not a Comet invariant, so `from_bytes` reports `Execution` rather than `Internal`. This is also what the "surfaces as a plain Comet execution error" incompatibility note already promised. - Complete `getUnsupportedReasons()`: the lgConfigK range case was missing from `CometHllSketchAgg`, and `CometHllUnion` / `CometHllUnionAgg` documented no unsupported cases at all despite both returning `Unsupported` for a non-foldable flag. - Pin `datasketches` to `=0.3.0`. The compact-decode workaround and the HLL_4 aux guard are both written against that release's array code, so the version should not move without re-checking those two tests. Also record the HLL_4-with-aux-entries rejection as an incompatibility on the three serdes that read sketch bytes: Comet errors there where Spark returns an estimate, which belongs in front of anyone opting in. Add the end-to-end test comphead asked for on the NULL `allowDifferentLgConfigK` fix. Spark marks `HllUnion` null-intolerant, so `NullPropagation` folds a foldable NULL flag away before Comet sees it; the test excludes that rule so the native kernel is what answers, and asserts `hll_union` survives into the plan so it cannot pass on a folded or fallen-back plan. * test: keep the HLL lgConfigK rejection test native on Spark 4.2 Spark 4.2 replaces MergeScalarSubqueries with MergeSubplans, which also merges non-grouping Aggregate nodes. That folded the two hll_sketch_agg branches of the test's UNION ALL into a single CTE projecting a struct with two fields both named 's'. Comet does not support CreateNamedStruct with duplicate field names, so the whole plan - hll_union_agg included - fell back to Spark and the test saw Spark's HLL_UNION_DIFFERENT_LG_K instead of Comet's native error. Materialize the lgConfigK=10 and lgConfigK=12 sketches into Parquet first and run the union over a plain scan, a shape that stays fully native on 3.4 through 4.2, and assert that with checkCometOperators. * fix: charge HLL accumulators for the sketch they hold, not the dense maximum `size()` on both HLL accumulators charged the full 2^lgConfigK register array as soon as a group existed. A sketch stays in an 8-slot coupon list or a small coupon hash set until it has seen enough distinct values, so at lgConfigK 21 each singleton group was charged 2 MiB for 32 bytes. 64 such groups exhausted a 16 MiB pool in the Final aggregation even with spilling, where Spark runs the query fine. The wrapper now reports the representation the sketch actually holds. The datasketches crate keeps the mode private, but its estimate is never below the coupon count in LIST and SET mode and seeds the HIP accumulator on promotion, and a union with a dense input is tracked from that input's preamble. A test walks every SET resize and the promotion and compares against the layout the crate serializes: never below it, and exact at all but a handful of points. --------- Co-authored-by: test <a@b.c>
| Commit: | ee3f239 | |
|---|---|---|
| Author: | Andy Grove | |
| Committer: | GitHub | |
fix: reject a file without field ids at any depth whether or not id matching is on (#6116) (#6266) * fix: reject a file without field ids at any depth whether or not id matching is on Spark's `ParquetReadSupport.getRequestedSchema` raises when the requested schema carries a Parquet field id at any depth and the file carries none at any depth, unless `spark.sql.parquet.fieldId.read.ignoreMissing` is set. The check runs on every read, for both readers, and does not consult `spark.sql.parquet.fieldId.read.enabled`. The native scan ran that check only with the read flag on, only over root fields, and only inside the name remap, which a case-sensitive session with the flag off never reaches. A read schema with ids over a file without ids returned rows where Spark raises, and a file whose ids sit only on nested fields was rejected where Spark reads it and null-fills the unmatched fields. The check now runs at the top of the expression adapter factory with a recursive predicate on both schemas. Id matching itself stays gated on the read flag and root ids, as Spark gates it per struct level. The Iceberg scan opts out, since its reader resolves columns by id and supplies the ids itself. * fix: answer both halves of the missing-field-id check from what Spark reads Spark checks the pruned required schema for ids and the raw Parquet schema for their absence. The requested half now comes from `required_schema` at plan time, since the logical schema DataFusion hands the adapter is the full read schema, so an id on a column the query never projects no longer raises. The file half moves into the eager page index reader, which walks the raw schema from the footer. The Arrow schema the adapter sees has lost container metadata after INT96 coercion and never carries ids on repeated list or key value groups, so a file whose only id sat on a struct holding a timestamp raised where Spark reads. The JNI error conversion unwraps the Parquet external error so the Java side keeps the same exception, and the Iceberg scan no longer needs an opt-out since it does not use that reader. Tests cover the pruned projection, the timestamp struct, an id only on a list or key value group, a directory mixing files with and without ids, nested schema pruning, id zero, count(*), and the exception class on each version. * test: find the missing field ids error anywhere in the cause chain On Spark 4.0 and 4.1 the RuntimeException Spark raises for a read schema with field ids over a file without any arrives wrapped once, in the FAILED_READ_FILE SparkException that collect raises, and the helper assumed one more layer above it. It now walks the cause chain and requires a plain RuntimeException carrying Spark's message, for Spark and for Comet alike, so the layer count no longer matters. * fix: let the serde say whether the file must carry field ids The requested side of the missing field ids check is Spark's own hasFieldIds over the required schema, so the serde now sends one flag, require_field_ids, in place of ignore_missing_field_id, and the native Arrow walk is gone. The reader factory takes that one flag through a builder and the parquet options carry nothing for it. The error names the file, and the module doc leads with the lasting reason the check reads the raw footer: ids on repeated groups and on the message root never reach Arrow. The Rust scan tests that had Scala twins are gone, the repeated group cases moved to Scala where they compare against Spark, and the Scala tests fold into the existing port and one pruned schema test with Spark's own assertion form. (cherry picked from commit 863f11de5eb78bacf11737032cdbcf496ad520b5) Co-authored-by: dustin <dwsmith1983@users.noreply.github.com>
| Commit: | 863f11d | |
|---|---|---|
| Author: | dustin | |
| Committer: | GitHub | |
fix: reject a file without field ids at any depth whether or not id matching is on (#6116) * fix: reject a file without field ids at any depth whether or not id matching is on Spark's `ParquetReadSupport.getRequestedSchema` raises when the requested schema carries a Parquet field id at any depth and the file carries none at any depth, unless `spark.sql.parquet.fieldId.read.ignoreMissing` is set. The check runs on every read, for both readers, and does not consult `spark.sql.parquet.fieldId.read.enabled`. The native scan ran that check only with the read flag on, only over root fields, and only inside the name remap, which a case-sensitive session with the flag off never reaches. A read schema with ids over a file without ids returned rows where Spark raises, and a file whose ids sit only on nested fields was rejected where Spark reads it and null-fills the unmatched fields. The check now runs at the top of the expression adapter factory with a recursive predicate on both schemas. Id matching itself stays gated on the read flag and root ids, as Spark gates it per struct level. The Iceberg scan opts out, since its reader resolves columns by id and supplies the ids itself. * fix: answer both halves of the missing-field-id check from what Spark reads Spark checks the pruned required schema for ids and the raw Parquet schema for their absence. The requested half now comes from `required_schema` at plan time, since the logical schema DataFusion hands the adapter is the full read schema, so an id on a column the query never projects no longer raises. The file half moves into the eager page index reader, which walks the raw schema from the footer. The Arrow schema the adapter sees has lost container metadata after INT96 coercion and never carries ids on repeated list or key value groups, so a file whose only id sat on a struct holding a timestamp raised where Spark reads. The JNI error conversion unwraps the Parquet external error so the Java side keeps the same exception, and the Iceberg scan no longer needs an opt-out since it does not use that reader. Tests cover the pruned projection, the timestamp struct, an id only on a list or key value group, a directory mixing files with and without ids, nested schema pruning, id zero, count(*), and the exception class on each version. * test: find the missing field ids error anywhere in the cause chain On Spark 4.0 and 4.1 the RuntimeException Spark raises for a read schema with field ids over a file without any arrives wrapped once, in the FAILED_READ_FILE SparkException that collect raises, and the helper assumed one more layer above it. It now walks the cause chain and requires a plain RuntimeException carrying Spark's message, for Spark and for Comet alike, so the layer count no longer matters. * fix: let the serde say whether the file must carry field ids The requested side of the missing field ids check is Spark's own hasFieldIds over the required schema, so the serde now sends one flag, require_field_ids, in place of ignore_missing_field_id, and the native Arrow walk is gone. The reader factory takes that one flag through a builder and the parquet options carry nothing for it. The error names the file, and the module doc leads with the lasting reason the check reads the raw footer: ids on repeated groups and on the message root never reach Arrow. The Rust scan tests that had Scala twins are gone, the repeated group cases moved to Scala where they compare against Spark, and the Scala tests fold into the existing port and one pruned schema test with Spark's own assertion form.
| Commit: | 590b669 | |
|---|---|---|
| Author: | Andy Grove | |
| Committer: | GitHub | |
feat: positional round robin shuffle keyed on a row ordinal (#6095) * feat: positional round robin shuffle keyed on a row ordinal Comet implements Spark's round-robin shuffle as hash partitioning over every column of every row. On a wide nested schema that dominates the shuffle write: `create_murmur3_hashes` recurses into every struct child per row, and the resulting row-level scatter forces `interleave_record_batch` to walk every column and child again on flush. It is also not round robin. Placement is a pure function of a row's contents, so a column of one repeated value lands entirely on one reducer where Spark's round robin spreads it. Adds `RoundRobinStrategy::RowGroups`, which places rows the way Spark does: the row at task-global ordinal i goes to `(mapPartitionId + i / groupRows) % numPartitions`. The counter is over rows, not batches, and carries across batch boundaries, so placement does not depend on how the reader frames its input. That matters because no Spark contract covers framing: `DETERMINATE` promises the same rows in the same order and says nothing about chunking, so an operator that spills can reframe under different memory pressure while still honouring it. Keying on a row ordinal reduces the residual assumption to exactly the one Spark's own round robin makes. Positional placement is used only where that assumption is established, in two independent places that both have to hold: * `CometShuffleExchangeExec.replaysRowsInOrder` walks the native subtree fused into the writer, which the RDD graph cannot see because the subtree collapses into one `CometNativeShuffleInputRDD`. Short allowlist rather than a denylist: a native scan under nothing but projections and filters. * `CometNativeShuffleInputRDD.getOutputDeterministicLevel` applies Spark's own `isOrderSensitive` rule to everything below that RDD, reporting INDETERMINATE over a non-determinate parent so the DAGScheduler rolls the stage back instead of re-running one task into consumed output. Anything else keeps `HashAll`, which is safe to re-execute whatever its input does. Both are behind `spark.comet.shuffle.native.partitioning.roundrobin.positional.enabled`, default false. The index representation gains a second shape: positional placement records `(batch, start, len)` runs instead of one `(batch, row)` pair per row, which is smaller against the spill reservation and lets the flush copy whole ranges. `RunIterator` builds a chunk by slicing and concatenating runs, and passes a run covering an entire buffered batch straight through without copying. A schema containing `Utf8View` or `BinaryView` falls back to `HashAll`, because those are the one family whose buffers the IPC writer does not truncate for a slice. * fix: honour a positional group size larger than the batch size Capping `groupRows` at `batch_size` silently ignored what the user asked for. A longer group is meaningful and works as written: it sends several consecutive input batches to the same output partition, which is a legitimate way to trade balance for fewer, larger shuffle blocks. * refactor: simplify positional round robin plumbing Resolve the positional group size into the partitioning itself instead of a parallel field, give each round-robin strategy its own match arm, and buffer runs without moving the scratch vector in and out. Skip the unused partition_starts scratch under positional placement, pre-size the run iterator's chunk scratch, and compute the driver-side positional decision once per exchange so the RDD and the writer read the same value. * fix: rebuild the shuffle writer per iteration in end-to-end benches A ShuffleWriterExec publishes its partition offsets through a OnceLock, so re-executing one exec across criterion iterations fails on the second run with "partition offsets were already published". Build a fresh exec per iteration with iter_batched, keeping construction out of the timing. * bench: add a partitioning-only benchmark for round robin placement The end-to-end shuffle benches are dominated by IPC encoding and the file write, so they cannot separate two placement strategies. Add a group that stops before both: one arm times placement and per-partition index buffering alone, the other adds the flush through a writer that discards its batches, which is where a run-indexed partitioner diverges from a row-indexed one. benches/ compiles as its own crate, so reaching the partitioners needs a pub seam. Rather than export MultiPartitionShuffleRepartitioner and the PartitionWriter trait, add one opaque handle in a doc(hidden) module and leave the rest crate-private. The nested fixture grows a per-row fill. The existing one repeats a single value down every leaf, so every row hashes alike and a hash strategy would place the whole input on one output partition, never performing the scatter that the gather exists to undo. The encoding benches keep the constant fill and their current numbers. * fix: scramble the positional round robin start partition Starting each map task at its own partition id made consecutive tasks start on consecutive partitions. A task walks ceil(rows / groupRows) consecutive partitions from its start, so adjacent starts overlap and the partitions past numMapTasks + groupsPerTask get nothing: ten tasks of 5,000 rows into 200 partitions at the auto group of 64 leave 112 reducers empty. That is the correlation SPARK-21782 fixed. Compute the start the way Spark does, XORShiftRandom(partitionId) .nextInt(numPartitions) + 1, in CometNativeShuffleWriter where the TaskContext is in scope, and pass it in the proto. Still a pure function of the map partition, so a retried task reproduces its own placement, and the + 1 matches Spark's pre-increment so groupRows = 1 places rows exactly where Spark's round robin would. The planner no longer reads self.partition, which removes the jni_api partition-0 caveat. * docs: correct the positional round robin balance and alignment claims Three wording fixes from review, none of them behavioural. The groupRows bound is per map task. A reducer sees the sum over all of them, which only evens out when each task emits many more groups than there are output partitions, so say that in the config doc rather than promising a stage-wide bound. MIN_AUTO_GROUP_ROWS is a cap on how finely a batch is cut, not an alignment guarantee. A run only starts on a byte boundary when the batch starts on a group boundary, and row_seq counts across batches, so after a filter every run in a batch is offset. The start has to be decorrelated across mappers, not merely distinct, which is what XORShiftRandom is for. Also update the round robin item in the shuffle review skill, which still said a positional round robin was a bug by construction. * refactor: place positional runs straight into the run index positional_runs now returns an iterator of (partition, rows) that the repartitioner consumes directly into BufferedRuns, which drops the PositionalRun struct, the scratch vector holding a batch's runs, and the second pass over it. PartitionIndices::empty_like takes no argument, since its one caller passed the index's own partition count, and the gather timer is named interleave_time again, after the metric it feeds and that the Spark UI and the docs show. RunIterator's zero-copy branch compared a whole buffered batch against batch_size with >=, but insert_batch slices every batch to at most batch_size. It is == now, and the two paths' comments agree that every chunk but a partition's last is batch_size rows. The view-type fallback built RoundRobinStrategy::default(), silently dropping a configured maxHashColumns. RowGroups carries max_hash_columns now, and the fallback, factored into partitioning_for_schema so it can be tested, hashes with it. Nothing spilled under the run-indexed shape in any test. The spill metrics and heterogeneous-batching tests run it alongside the row-indexed shape, and positional_placement_survives_spilling requires placement under both spill triggers, the buffer limit and a pool that refuses to grow, to equal the unspilled placement exactly. Three unit tests the end-to-end placement tests subsumed are gone, and the start-partition test is folded into the walk test, which now asserts every group's partition, a wrapping start included. The rustdoc and proto comments shrink to what each item does and point at native_shuffle.md for the argument. * fix: keep positional round robin off under Celeborn Positional placement would be the first path to hand the Celeborn push writer sliced batches, and an indeterminate stage's rollback has not been worked through for push shuffle, so positionalRoundRobinSpec now declines under the Celeborn shuffle manager. The planning suite checks it over a bare native scan, so the manager is the only thing ruling it out. A missing TaskContext now throws instead of starting the task at XORShiftRandom(0), which would put every task at the same partition: the correlation the scrambled start exists to prevent. usesPositionalRoundRobin moves from a public companion predicate to a package-private method on the exec, with shuffleType folded into the decision, since nothing consulted it for a columnar exchange. The two tests that exercised only the placement arithmetic are replaced by one that runs ten real map tasks of 5,000 rows into 200 reducers through the shuffle and requires none of them empty. With the start reverted to the bare map partition id it reports exactly the 112 the simulation predicted. The exchange, RDD and config docs now say what Spark's round robin requires. By default it sorts each map partition first, so a retry only has to produce the same rows; positional placement never sorts and needs them in the same order, which makes replaysRowsInOrder the whole safety argument. The RDD-level isOrderSensitive check is described as the defence in depth it is: a native scan leaf contributes no input RDD, so under today's allowlist it cannot fire. * docs: correct the Spark round robin comparison and keep the argument in one place Spark's round robin sorts each map partition before assigning positions unless spark.sql.execution.sortBeforeRepartition is off, so by default a retry only has to produce the same rows. Positional placement never sorts and needs them in the same order. The guide said the two made the same assumption, which undersold how much rests on replaysRowsInOrder, and it said groupRows = 1 matches Spark, which holds only with the sort off. The rationale was written out in about seven places and had already needed a three-way fix. The round robin section of native_shuffle.md now holds it, opening with what Spark does in both modes, and the review skill's item is a pointer plus the checks a reviewer should make. The interleave_time metric description covers the run copy as well. * bench: build each partitioning fixture batch separately Both partitioning fixtures were one batch cloned eight times. A clone shares its buffers, so the reservation charged seven of the eight nothing, and the gather kept rereading one cache-resident batch. Each batch is now built from where the previous one left off. On the same machine, the cloned fixture reproduces the numbers first posted for this branch, while distinct batches roughly triple the nested gather wherever rows are copied: HashAll's place+gather goes from 37.7 ms to 106 ms and RowGroups(auto)'s from 8.5 ms to 20.4 ms. The zero-copy RowGroups(8192) arm, which copies nothing, is unchanged. * fix: gate positional round robin on deterministic expressions and freeze its group size Admit a native project or filter under positional placement only when its expressions are deterministic, since a nondeterministic UDF can reorder or re-filter rows on a retry while the RDD still reports DETERMINATE. Resolve the default group size on the driver and carry it in the shuffle dependency, so a map stage re-run after spark.comet.batchSize changes places rows exactly as the attempt it replaces. The native planner now rejects a non-positive group size instead of deriving one from the executor's batch size. Qualify the "wraps once per batch" claim for partition counts past the 64-row floor. --------- Co-authored-by: Oleks V <comphead@users.noreply.github.com>
| Commit: | ef7bda5 | |
|---|---|---|
| Author: | Andy Grove | |
| Committer: | GitHub | |
feat: support listagg / string_agg aggregate (Spark 4.0+) (#4816) * feat: support listagg / string_agg aggregate (Spark 4.0+) Adds a `SparkListAgg` UDAF and a `CometListAgg` serde so that Comet can natively execute the simple form of Spark 4.0's `LISTAGG(child, delimiter)` / `string_agg` on `StringType` inputs with a literal delimiter. `WITHIN GROUP (ORDER BY ...)`, `BinaryType` inputs, non-literal delimiters, and non-default collations fall back to Spark. `DISTINCT` falls back because Comet already rejects multi-column distinct aggregates. The native accumulator returns `Utf8` but keeps its intermediate state as `Binary` to match Spark's `TypedImperativeAggregate` buffer schema so the Comet shuffle layer does not insert a `Utf8` → `Binary` cast the merge side cannot read back. Scaffolding produced by the `implement-comet-expression` skill. * refactor: hoist CometListAgg reason strings and use `.foldable` Consolidate the reason strings into `private val`s so `getSupportLevel` and `getUnsupportedReasons` reference the same source of truth, and replace the manual `case _: Literal` delimiter check with the standard `.foldable` gate used elsewhere (e.g. `CometPercentile`). * refactor: address listagg review feedback Add a GroupsAccumulator fast path to SparkListAgg for grouped aggregation, remove the dead `datatype` proto field and unreachable serialization branch, drop the never-emitted `distinctReason`, and add a multi-partition merge test. Keep the `Binary` intermediate state: Spark's `ListAgg` is a `TypedImperativeAggregate` whose `aggBufferAttributes` is `BinaryType`, and the shuffle Exchange between the partial and final aggregate uses that Spark-declared schema even though partial and final run in the same engine. The module doc and `state_fields` comment are rewritten to explain this. * test: cover listagg collation fallback and align docs with the Binary state - Add a UTF8_LCASE collation fallback case to listagg.sql; the aggregate serde's collation guard is reachable even though the scan falls back. - Drop the comments claiming the collation guard is unreachable. - Describe DISTINCT as falling back via the multi-column distinct check in aggExprToProto, not Spark's multi-stage rewrite (serde, native, proto). - Update the audit doc: intermediate state is Binary to match Spark's TypedImperativeAggregate buffer at the Exchange.
| Commit: | 5442c93 | |
|---|---|---|
| Author: | Andy Grove | |
| Committer: | GitHub | |
feat: hook native Parquet writes into Spark's WriteFilesExec seam on Spark 4.0+ (#5763) * feat: hook native Parquet writes into Spark's WriteFilesExec seam on Spark 4.0+ Native writes replace the whole DataWritingCommandExec, which means InsertIntoHadoopFsRelationCommand.run never runs. Everything that method does has to be re-implemented inside CometNativeWriteExec: a hardcoded SQLHadoopMapReduceCommitProtocol (so spark.sql.sources.commitProtocolClass is ignored), dynamicPartitionOverwrite pinned to false, a hand-ported copy of the SaveMode logic, a bespoke commit-message accumulator, and its own commitJob call. Most of the open native-writer issues are symptoms of that one decision rather than independent defects. On Spark 4.0+, V1WritesUtils.getWriteFilesOpt matches the WriteFilesExecBase trait (introduced in 4.0 precisely for this), so a Comet node that extends it gets driven through FileFormatWriter.executeWrite -> SparkPlan.executeWrite -> doExecuteWrite, and Spark keeps ownership of everything above the per-task write. Spark 3.x has no such trait: getWriteFilesOpt matches the concrete WriteFilesExec case class, a Comet node there would not be found, and Spark would silently take FileFormatWriter's non-planned, row-based branch. So the new seam is additive. CometDataWritingCommand and CometNativeWriteExec are kept unchanged and remain the 3.4/3.5 path; CometExecRule picks the path by version and the two never both fire. The legacy path goes away with Spark 3.x support. Add: - CometWriteFilesExec, overriding doExecuteWrite and mirroring FileFormatWriter.executeTask for the parts Comet must do itself: build the TaskAttemptContext, ask the commit protocol for a path, run the native writer, drive the stats trackers, commit or abort. Plus the CometWriteFiles serde and a two-line ShimCometWriteFilesExec in spark-4.x / spark-3.x. - File paths come from FileCommitProtocol.newTaskTempFile and are used verbatim, so names match Spark's part-<id>-<uuid>-c000.<codec>.parquet and committers that track individual files (S3A magic, streaming manifest) work. - Column names, nullability and field IDs come from WriteJobDescription.dataColumns rather than the query output, so INSERT INTO t SELECT a+1 writes the target column's name. - Byte and row counts come from BasicWriteTaskStatsTracker, which stats files through the FileSystem API and is therefore correct on HDFS. - ParquetWriter proto: work_dir is now optional. When set (3.x) the native writer derives the file name as before; when unset (4.0+) output_path is the exact file to write and is used verbatim. - On 4.0+ the opt-in moves to spark.comet.operator.WriteFilesExec .allowIncompatible, with the old DataWritingCommandExec key kept as a deprecated alternative. isOperatorAllowIncompat now resolves alternatives, which the planner's by-name lookup previously bypassed. AQE re-plans the write command's child and re-inserts a WriteFilesExec above the node Comet already converted; leaving DataWritingCommandExec in place means Comet no longer has to guard against the resulting nested native writes. * refactor: take a single deprecated alternative for operator incompat configs Matches the reviewed form on #5293: ConfigBuilder mutates in place, so the Seq destructuring was rebinding the same object. Only one operator has an alternative and there is no reason to expect more. * fix: escape the output path before parsing it as a URI The output path reaches both write serdes as `Path.toString`, which decodes percent escapes: a directory containing a space or a literal `%` yields a string that is not a valid URI, and `URI.create` throws on it. On Spark 4.0+ that exception escaped `CometWriteFiles.convert` and failed the query. On 3.x `CometDataWritingCommand.convert` caught it and silently handed the write back to Spark, so the native writer was never used for those paths. Round-trip through `Path` instead, which re-escapes. Only the scheme and authority reach `extractObjectStoreOptions`, but parsing has to succeed to get at them. Also assert that the INSERT INTO visibility test's write actually went native, rather than inferring it from the read-back. * fix: decline HDFS writes whose output path needs URI escaping * fix: also decline Unicode HDFS write destinations The raw/decoded URI comparison only catches what java.net.URI had to escape, and java.net.URI leaves non-ASCII path characters alone, so an hdfs://ns/cafe<U+0301>/output destination was admitted. percent_encoding's should_percent_encode is !byte.is_ascii() || set.contains(byte), so the native parser escapes every non-ASCII byte regardless of the encode set and the writer creates caf%C3%A9 outside Spark's staging directory. The guard now also declines any character the native parser rewrites. The ASCII half of that set was determined against the locked url 2.5 crate by parsing hdfs://ns/pre<c>post/output for every printable ASCII c: space, ", #, <, >, ?, backtick, { and } are rewritten and the rest survive, so partition directories and Spark's _temporary attempt paths still qualify. The comment no longer claims the Java comparison detects the divergence on its own; both conditions are kept because the Java one still catches a literal % that the native parser leaves alone. Tests add accented (precomposed and combining), CJK, emoji and nested non-ASCII cases plus the remaining escaped ASCII characters, built from code points since scalastyle forbids non-ASCII source. Disabling the new condition makes the accented case fail, so the Java comparison alone demonstrably does not cover it. * fix: address review feedback on the native WriteFilesExec seam Correctness: - Decline HDFS writes whose *file names* would diverge, not just their directory. `mapreduce.output.basename` is caller-controlled and reaches every committed name through `HadoopMapReduceCommitProtocol.getFilename`; a basename holding `?` or `#` makes the native URL parser truncate, so every task writes the same name and they overwrite each other at commit. Adds an execution-time backstop over the complete `newTaskTempFile` path, which a custom commit protocol owns and planning cannot predict. - Read the compression option case-insensitively, as Spark's `ParquetOptions` does. `option("Compression", "lz4_raw")` used to fall through to the SQLConf default, so the unsupported-codec guard was bypassed and Comet wrote SNAPPY into a file Spark had named `.lz4raw.parquet`. The codec is now also re-derived per task from `CodecConfig.from(taskAttemptContext)`, the same place the file extension comes from, so the name and the contents agree by construction. The shared helpers move to `NativeWriteUtils`, which fixes the identical bug on the Spark 3.x path. - Use `Utils.tryWithSafeFinallyAndFailureCallbacks` / `tryWithSafeFinally` in `executeTask` and `writeNatively`, matching `FileFormatWriter`: a failure while aborting or closing the iterator is attached as a suppressed exception instead of replacing the failure that caused it. `statsTrackers` moves inside the guard so a throwing `newTaskInstance` still reaches `abortTask`. Planning and reporting: - Convert `WriteFilesExec` from its enclosing `DataWritingCommandExec` rather than from a separate tag pre-pass, so the output path comes straight from the command that owns it and neither `withNewChildren` copying tags nor "nothing hands us a bare WriteFilesExec" has to hold. - Only skip the fallback reason on `DataWritingCommandExec` when its child really was converted; a write with no native child now says why it fell back. - Drop the node's duplicate `files_written`/`bytes_written`/`rows_written`. `BasicWriteJobStatsTracker` is authoritative here, and the native `bytes_written` reads 0 on HDFS. Tests and docs: - Rust `url_path_rewritten_characters` pins the `url` crate's path encode set, which the JVM guard mirrors; a crate upgrade can no longer reopen the hole with a green build. - New JVM coverage: mixed-case `compression` (honored, and declined when unsupported), the #3426 nested-name INSERT, an empty non-zero partition writing no file, the third-party `WriteTaskStatsTracker` warning, and the basename/committer-path guards. - The abort test no longer claims to demonstrate task retry or speculation. - `installation.md` says which Spark versions its EXPLAIN output applies to. * fix: drop a redundant string interpolator flagged by scalafix RedundantSyntax * fix: decline percent-bearing HDFS basenames at planning mapreduce.output.basename=part%foo passed the planning guard and then failed checkNativeWriteDestination at execution, aborting the job instead of falling back to Spark's writer. needsNativeUrlEscaping deliberately excludes % because the url crate leaves it alone, so only Java's raw-vs-decoded comparison sees it. The basename now goes through the same hdfsPathDivergence predicate the task guard uses, applied to the path the basename produces. That closes the gap for % and for the other characters java.net.URI escapes but the url crate keeps ([, ^, |), and makes the two guards agree by construction rather than by two character sets staying in sync. Also updates CometEmptyRelationParquetWriterSuite, which main added while this branch was open: a native empty relation is the zero-partition input CometWriteFilesExec swaps a single-partition RDD in for, so on 4.0+ that write is accelerated rather than declined. * fix: drop imports left unused by moving the write assertions into the base
| Commit: | 7472561 | |
|---|---|---|
| Author: | Peter Lee | |
| Committer: | GitHub | |
perf: spill every shuffle partition of a task into one file (#5916) * perf: return native shuffle partition offsets over JNI instead of an index file The native shuffle writer knew every partition offset by the time it finished a map task, but handed them to the JVM through a temporary file: LocalPartitionWriter::finish_all created an index file and wrote num_output_partitions + 1 little-endian i64 offsets into it, and CometNativeShuffleWriter read the whole file back with Files.readAllBytes, converted the offsets to lengths, deleted it, and passed the lengths to IndexShuffleBlockResolver.writeMetadataFileAndCommit, which writes Spark's real index file. The temp file existed only to move an array of longs across the JNI boundary, and cost every map task a create, write, read and unlink on top of the index file Spark commits anyway. The parse also allocated an intermediate array and a ByteBuffer per partition (item 5 of #5198), which goes away with the file. The offsets are now published in memory through a PartitionOffsets slot shared by the writer and its ShuffleWriterDestination, and read back over JNI by Native.getShufflePartitionOffsets. The index path no longer travels in the plan, so LocalPartitionWriter.output_index_file and the legacy ShuffleWriter.output_index_file are removed and their field numbers reserved. The offsets have to be read while the native plan is still alive. CometExecIterator closes itself when its stream reaches the end, and close releases the execution context that owns the writer, so reading after drainAndClose returned freed memory and produced garbage lengths. The iterator instead captures the offsets at end of stream, before close, when built with capturePartitionOffsets, which only the local destination sets: RSS reports its partition lengths through its pusher. Partition lengths are derived from effectivePartitionCount, the output partition count, not the numParts constructor argument, which is the input partition count. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * review: address feedback on naming and layering - The shuffle crate does not know about JNI, so PartitionOffsets and the local writer describe the handover in terms of the caller driving the plan rather than the JVM reading over JNI. - ShuffleWriterExec::try_new says what it writes where: partition data to a local file, offsets in memory. - CometExecIterator had three lookalike names for one thing. The read is now a named method, readPartitionOffsetsBeforeClose, whose name and doc carry the constraint that made the placement surprising: the offsets live in the native execution context, close releases it, and hasNext closes as soon as the plan runs out of output, so the final hasNext is the last point they can be read. The field is partitionOffsets and the constructor flag is documented. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix: size shuffle partition lengths from the offsets the writer returned Two CI failures, both from assuming more than the writer guarantees. The proto crate's own tests still referenced ShuffleWriter.output_index_file and LocalPartitionWriter.output_index_file, so datafusion-comet-proto failed to compile its test target. I had only checked the shuffle and core crates locally rather than the whole workspace. The round-trip tests now assert that a new plan carries no index path, and that a plan still carrying the retired tag 4 decodes cleanly because the tag is reserved rather than reused. partitionLengths was sized by effectivePartitionCount, which is not what the writer produces. isSinglePartitioning serializes a range partitioning whose sampled bounds came out empty as SinglePartition, so native writes one partition while the declared output partitioning still reports several, and the require failed with "returned 2 partition offsets for 10 output partitions". The index file was always sized by what the writer produced, so deriving the length count from the returned offsets restores the previous behaviour exactly. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * review: drop the reserved declarations for the retired index path Requested in review. The plan proto is built by the JVM and consumed by native in the same process from the same artifact, so no old plan ever meets a new reader and there is nothing for a reserved tag to protect against. Decoding is unaffected either way: an undeclared tag is skipped as an unknown field, and reserved only stops protoc from later reusing the number. The proto round-trip test covering a plan that still carries the retired tag 4 keeps passing, and its comment no longer credits reserved for that. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * review: trim comments and close the gap left by the retired field number Comments cut back to what they need to say, per review: the JNI entry point, the PartitionOffsets type and its set/get, the destination field, try_new, the partition_offsets accessor, the zero-offset test, the finish_all note, and the two comments in CometNativeShuffleWriter. ShuffleWriter field numbers 5 through 11 shift down to 4 through 10, closing the gap the retired output_index_file left. Both sides of the plan are generated from this file and ship together, so no encoded plan outlives the change. That does mean tag 4 now belongs to codec, and the LegacyShuffleWriter test struct claimed it for a string. Decoding a plan carrying it would be a wire type mismatch rather than a skipped unknown field, so the struct drops that field. The test still covers a legacy plan decoding without a partition writer. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * perf: spill every shuffle partition of a task into one file The native shuffle writer spilled each output partition to its own temporary file, created on that partition's first spill and held open until the task finished. A task with P partitions that spilled held up to P spill files open at once, each with a create and an unlink, and every spill scattered its bytes across P files. A task now spills to a single file. Each write appends the partition's blocks and records the range they occupy, and finish_partition copies a partition's ranges into the output in write order. Correctness does not depend on the order partitions are written in, which PartitionWriter leaves unspecified. A single spill round gives contiguous ascending ranges, so the merge reads sequentially. A write that fails partway can leave uncounted bytes in the shared file, which would shift every later range, so the spill refuses further writes and range reads after a failure. The merge also checks each copy's length, so a spill file shorter than its ranges fails the partition instead of writing it short. Closes #3859. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * perf: merge short spill ranges with one positional read Copying a spill range with io::copy costs an lseek, two statx calls and a copy_file_range. Ranges that fit in the write buffer are now read with one read_exact_at into a scratch buffer; longer ranges still use io::copy. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * perf: buffer spill writes across partitions Each partition's spill write ended with a flush, so a spill round issued one write syscall per partition. The spill file's writer is now buffered across partitions and flushed before the merge reads it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * review: reserve the retired index file tags and drop stale index file docs Requested in review: ShuffleWriter keeps its fields on tags 5 through 11 with reserved 4, and LocalPartitionWriter gets reserved 2, as operator.proto does for other retired fields. The proto test again decodes a plan carrying tag 4. native_shuffle.md steps 6 and 7, and the LocalPartitionWriter, shuffle_write and shuffle_bench output_dir comments, no longer describe an index file. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
| Commit: | 8c45f3e | |
|---|---|---|
| Author: | dustin | |
| Committer: | GitHub | |
perf: cache parsed plan data across a stage tasks (#5615) * perf: cache parsed plan data across a stage tasks Every task of a stage deserializes byte-identical plan bytes, yet each one parsed the full operator tree, re-derived the scan source key by stringifying its schema and filter lists, and re-parsed the scan common message before injecting its partition data. Three bounded per-executor caches now share that work: the parsed base plan keyed on content, the parsed NativeScanCommon, and a source-key memo that hits protobuf reference-identity fast path once the base plan is shared. Injection itself stays per task, since partition data genuinely differs, and the injected tree is never cached. Per-task overhead drops from roughly 300us to 60us on a 100-column scan plan and about 3x on a 1000-column plan. Cache misses compute outside any lock so unrelated stages never serialize behind one parse, and racing threads on a cold key adopt a single instance so reference sharing holds. * perf: carry the scan key in the plan and scope prepared data to the base plan entry The executor derived each native scan's lookup key by stringifying its schema and filter lists, memoized in an LRU probed under the map monitor, and parsed scan commons through a second per-scan LRU that a single plan with 17 scans churned to zero reuse. The driver already derives the key once in CometNativeScanExec, so carry it inside the NativeScan proto and read it back on every injection path, including the native shuffle writer's, which previously missed the fast paths entirely. Prepared commons now live inside the base plan's own cache entry (scoped to the shuffleId on the shuffle-write path), so a plan and its scans form one eviction unit. Entries pin the finalized common bytes because scalar-subquery data filters resolve per execution: equal base plan bytes do not guarantee equal finalized commons, and a stale entry is replaced rather than served. The base plan cache keys on a stored hash computed once per task outside the monitor instead of a raw ByteBuffer that rescanned the bytes on every probe. * fix: release prepared shuffle data when a shuffle is unregistered or the manager stops The shuffle-scoped prepared-commons store is a JVM singleton keyed by shuffleId, and shuffle ids restart at zero for every SparkContext, so a local or embedded caller that stops and recreates its context kept stacking new scan keys under ids the previous context had used. Both CometShuffleManager and CometCelebornShuffleManager now drop a shuffle's entry in unregisterShuffle (reached from Spark's ContextCleaner via BlockManagerStorageEndpoint) and clear the store in stop(), so a recreated context starts empty and long-lived contexts release each shuffle's data with the shuffle. * perf: fingerprint the plan bytes on the driver and key the base plan cache by it Every task of a stage built its PlanKey by hashing the full plan bytes with Arrays.hashCode, a scalar loop that took about a quarter of the cached path on JDK 17 for a wide plan. The driver now computes one XXH64 fingerprint where it serializes the plan and carries it on CometExecRDD, so the executor probe hashes nothing. Equality still compares the bytes, so a fingerprint collision cannot serve another stage's plan. The reference check in PlanKey.equals goes too, since HashMap already performs it before calling equals. * refactor: tighten the plan data injector SPI and release both caches on stop Type the injector's two halves through an abstract Prepared member so an implementation's prepareCommon and inject must agree, and move the single remaining cast to the memo boundary. Prefix memo keys with the injector class: every contrib scan arrives as the same CONTRIB_SCAN envelope, so two injectors agreeing on a key and byte-identical commons would otherwise share one prepared object. Drop the Iceberg injector's inner cache and the injectPlanData overload with no production caller, since both real entry points now memoize. The shuffle store gets its own maxCachedShuffles bound, the shuffle managers clear the base plan cache together with the shuffle store on stop, and the lifecycle suite asserts which stage shapes land in which store: a map-only stage in the base plan cache, a scan fused into a native shuffle under the shuffle id. * perf: ship only the source key hash in the NativeScan proto The transported key was the scan's source followed by a hash, and the source already sits in the common of the same message. The proto now carries the int32 hash alone as an optional field, so presence is explicit and a zero hash is still a transported value, and the executor rebuilds the key from the source next to it. The field is unreleased, so it changes in place. * docs: plan bytes are parsed once per stage, not per task QueryContextInterner described the plan as re-parsed per task by CometExecRDD.compute; parsing now happens once per stage on that path, while the per-task re-serialization it also mentions still holds. * docs: state the JVM-wide scope of the plan data stores and the memo key's classloader assumption * fix: adopt the winning prepared common when two tasks race a cold scan key prepareShared put its freshly prepared common unconditionally, so tasks racing the first access to a scan could each keep their own instance. It now merges into the memo atomically and returns whichever equal-bytes entry landed first, on both the base-plan and shuffle-scoped paths. Barrier tests cover both. --------- Co-authored-by: Andy Grove <agrove@apache.org>
| Commit: | e893b43 | |
|---|---|---|
| Author: | Xuanyi Li | |
| Committer: | GitHub | |
feat: add Lance contrib build gate (#5728)
| Commit: | db79067 | |
|---|---|---|
| Author: | Matt Butrovich | |
| Committer: | GitHub | |
feat(iceberg): Iceberg table format V3: apply deletion vector on reads (#5853) * upmerge main, resolve delta * serialize file type, clean up reflection * dedupe delete file path * add missing scala file changes * add new test for deduping puffin files, update docs * address PR feedback --------- Co-authored-by: Matt Butrovich <mbutrovich@apache.org> Co-authored-by: Andy Grove <agrove@apache.org>
| Commit: | 149a1db | |
|---|---|---|
| Author: | Andy Grove | |
| Committer: | GitHub | |
feat: support max_by and min_by aggregate expressions (#4817)
| Commit: | 99237e1 | |
|---|---|---|
| Author: | Andy Grove | |
| Committer: | GitHub | |
feat: support native aggregate function `mode` (#4782) * feat: support native mode aggregate function Add native support for the Spark `mode` aggregate, the most frequent value within a group. Spark breaks ties on the default `mode(col)` form non-deterministically (the chosen value depends on JVM hash-map iteration order), so the function is registered as Incompatible and opt-in via allowIncompatible; Comet resolves ties deterministically by returning the smallest tied value. NULLs are ignored, empty input returns NULL, and float keys are normalized to match Spark. The deterministic-flag and WITHIN GROUP forms fall back to Spark. Closes #3970 * fix: make mode's -0.0 key handling match the Spark version, plus review cleanups Addresses review feedback on #4782. The main item was the `-0.0` normalization in `normalize_key`. The review asked to stop collapsing `-0.0` into `0.0`, because Spark keys the frequency map on `java.lang.Double.equals` (via `OpenHashSet`), which distinguishes the two, and `NormalizeFloatingNumbers` never touches aggregate arguments. That is correct for Spark 3.4 through 4.1, but Spark 4.2.0 changed it: SPARK-57329 treats the split `-0.0`/`0.0` counts as a correctness bug and normalizes the key at update time. The fix landed in branch-4.2 after v4.2.0-rc1, so released 4.2.0 has it. Correct behaviour is therefore version-dependent, and neither always-collapsing nor never-collapsing is right across the profiles this repo builds. Added a `normalize_neg_zero` flag to the `Mode` proto message, set from `isSpark42Plus` in the serde (same pattern as `BloomFilterVersion` and `setIsSpark4Plus`), and gated the fold on it natively. `NaN` canonicalization stays unconditional, since `doubleToLongBits` collapses `NaN` on every supported version. `ScalarValue`'s `PartialEq`/`Hash` for floats are both bit-based, so once `NaN` is canonical the zeros stay distinct without further work. Test fixtures used `CAST(-0.0 AS DOUBLE)`, which does not produce a negative zero: an unsuffixed `-0.0` is a DecimalType literal and Decimal has no signed zero, so the column contained only `+0.0` and the existing signed-zero coverage was vacuous. Switched to `-0.0D`. Verified the new fixture is non-vacuous by forcing the flag to the wrong value and confirming it fails. Also in this commit: - correct the doc comment, which claimed the old behaviour matched `NormalizeFloatingNumbers`, and record which Spark comparison path governs `mode` versus `max_by`/`min_by` so the two are not "fixed" to match each other - drop the redundant `default_value` override - count key heap bytes in `size()` so string/binary/decimal modes do not under-report to the memory pool - assert the non-empty-groups invariant at both grouped emit sites - note that the `ScalarValue` frequency map is intentionally type-generic - add a `timestamp_ntz` compared query - add mode_within_group.sql pinning the Spark 4.x ordered forms as fallbacks, including that `ModeBuilder` rewrites `mode(col, false)` to the plain form so it still runs natively - TODO recording that the ASC WITHIN GROUP form could be Compatible * fix: account for group slot storage in mode's grouped accumulator size() --------- Co-authored-by: test <a@b.c>
| Commit: | 8c7b706 | |
|---|---|---|
| Author: | Ping Zhang | |
| Committer: | GitHub | |
feat: native dynamic filter pushdown for hash join into Parquet scans (#5699) * feat: push native join runtime filters into Parquet scans * test: fix right outer join hint for Spark 3.4 * fix: evaluate join runtime filters on the probe key * test: extract dynamic filter tests into a child module * refactor: move join dynamic filter planning into planner * fix: adapt join dynamic filters to DataFusion 55
| Commit: | 2da3291 | |
|---|---|---|
| Author: | Peter Lee | |
| Committer: | GitHub | |
fix: prevent silent overflow when reading Parquet TIMESTAMP_MILLIS values (#5177) * refactor: use Arrow casts for temporal conversions * fix: preserve Spark temporal cast semantics * test: match Spark micros-to-millis cases * test: link Spark micros-to-millis cases * test: use Spark tag in source link * Use imported arity kernel for Spark timestamp downscaling * fix: enforce Spark temporal conversion semantics * andy's 3rd review * address review: drop temporal.rs refactor, improve adapter error message - Revert cast_date_to_timestamp to main's version to avoid conflicting with #5457, which rewrites the same function as a safety fix - Document the Int32 input guarantee and zero-copy reinterpret path in date_from_unix_date - Make the microsecond-physical/millisecond-target rejection message actionable (report link + scan fallback workaround) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore: remove unused imports in parquet_support Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test: exercise dictionary-encoded pages in TIMESTAMP_MILLIS overflow regression A single row falls back to PLAIN even with dictionary encoding enabled, so the dictionary leg never covered a dictionary read. Write 16 repeated rows and assert hasDictionaryEncodedPages matches the writer setting. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: preserve timestamp pruning by rewriting predicates to the millisecond domain The predicate over a TIMESTAMP_MILLIS file column was wrapped in CometCastColumnExpr, which DataFusion's pruning analyzer cannot see through, so row groups Spark prunes from millisecond statistics were read and hit the checked millis->micros conversion. Rewrite predicate comparisons into the millisecond domain (exact integer rescaling of the literal, mirroring Spark's ParquetFilters) and unwrap IS NULL checks, so pruning works and predicate evaluation never converts file values. The scan output conversion stays checked. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: cover IN, null-safe equality, and nested predicates in the millis-domain rewrite Extend the millisecond-domain predicate rewrite to InListExpr and IsDistinctFrom/IsNotDistinctFrom, both of which DataFusion's pruning predicate can analyze, so row-group pruning protects flat TIMESTAMP_MILLIS columns for those forms too. Nested-field predicates can be neither pruned (PruningPredicate has no nested-field support) nor evaluated as Parquet row filters (struct columns are classified non-pushable), so no rewrite can keep the checked conversion from failing queries Spark answers via nested statistics pruning. Scans whose data filters reference nested fields fall back to the safe cast (overflow -> NULL, the pre-existing behavior), and the checked conversion is scoped to top-level columns for the same reason. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test: cover empty NOT IN with row filters * fix: retain dropped parquet filters * fix: retain scalar subquery filters --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
| Commit: | 165e2c6 | |
|---|---|---|
| Author: | Peter Lee | |
| Committer: | GitHub | |
feat: carry VariantType identity through schema serialization (#5631) * feat: carry VariantType identity through schema serialization * ci: register CometVariantTypeSuite * fix: address Variant type identity review feedback
| Commit: | 81d637b | |
|---|---|---|
| Author: | Andy Grove | |
| Committer: | GitHub | |
feat: implement regr_slope, regr_intercept, regr_r2, regr_sxx, regr_syy, regr_sxy aggregates (#4775) * feat: implement regr_slope, regr_intercept, regr_r2, regr_sxx, regr_syy, regr_sxy aggregates Add native support for the six simple linear regression aggregates that previously fell back to Spark. regr_avgx, regr_avgy and regr_count were already accelerated because Spark rewrites them to Average/Count. The native accumulators are composed from Comet's existing Spark-compatible covariance and variance accumulators so the partial aggregation state matches the buffer layout Spark's planner expects between partial and final stages: RegrReplacement (regr_sxx/regr_syy) -> 3 fields, Covariance (regr_sxy) -> 4, PearsonCorrelation (regr_r2) -> 6, and the slope/intercept composite -> 7. regr_r2 matches Spark's behavior of returning 1.0 when the dependent variable is constant but the independent variable varies (a perfect horizontal fit), which differs from DataFusion's regr_r2. * fix: match Spark's per-version regr_slope/intercept/r2 semantics The regr aggregates diverged from Spark in three ways that surfaced as CI failures across Spark versions: - regr_r2 degenerate cases were inverted. Spark 3.4/3.5/4.0 return null when the dependent variable is constant and 1.0 when the independent variable is constant; Comet had these swapped. - Spark 4.1 swapped that degenerate handling again (constant dependent -> 1.0, constant independent -> null). Route the behaviour through a new r2_constant_dependent_is_perfect_fit proto flag set from isSpark41Plus. - regr_slope/regr_intercept compute VariancePop(x) over both-non-null pairs on Spark 3.5+, but over every x-non-null row on Spark 3.4. Route this through a new filter_var_by_pair_nulls proto flag set from isSpark35Plus. Also evaluate regr_r2 as corr = ck / sqrt(m2_y * m2_x); corr * corr to mirror Spark's exact float rounding, so the golden-file postgres aggregates tests match bit-for-bit. * fix: regr_r2 degenerate-case swap (SPARK-55969) applies from Spark 3.5, not 4.1 SPARK-55969 swapped which regr_r2 degenerate case returns NULL and which returns 1.0. It shipped in Spark 3.5.9, 4.0.3 and 4.1.0, so every Spark version Comet builds against except 3.4 has the new behavior. Comet was gating it on isSpark41Plus, which made regr_r2 disagree with Spark on the 3.5 and 4.0 profiles.
| Commit: | 5b332b9 | |
|---|---|---|
| Author: | Jordan Epstein | |
| Committer: | GitHub | |
feat: implement native Iceberg V2 writer via iceberg-rust (#5361) Co-authored-by: Jordan Epstein <jordan.epstein@imc.com>
| Commit: | a2c6bd4 | |
|---|---|---|
| Author: | Ping Zhang | |
| Committer: | GitHub | |
feat: add partition writer destinations to native shuffle plans (#5476)
| Commit: | 5baa6b0 | |
|---|---|---|
| Author: | Oleks V | |
| Committer: | GitHub | |
feat: support `WindowGroupLimitExec` (#4870)
| Commit: | 6716318 | |
|---|---|---|
| Author: | Scott Schenkein | |
| Committer: | GitHub | |
feat: build gate + inert wiring for contrib Delta scans [Delta contrib split, part 2] (#4952) Part 2 of the Delta contrib split: the build gate and the inert core wiring an out-of-tree scan contrib plugs into. Nothing here is reachable on a default build -- no contrib is registered, no contrib class is compiled, and the native library carries zero contrib symbols. Core gains two format-agnostic extension points, both discovered at runtime so core holds no compile-time reference to any contrib: - `CometScanContrib`, a ServiceLoader-discovered hook (mirroring `PlanDataInjector`) that lets a contrib claim a V1 or V2 scan before Comet's built-in handling runs, plus `CometContribScanMarker` so `CometExecRule` can route a contrib's scan node to the contrib's own serde handler by a plain type test. - `ContribScan contrib_scan = 200`, a single permanent `Any`-shaped proto envelope (`type_url` + packed `value`) dispatched by `type_url` on the native side. Core's oneof never grows per-format, so independent contrib PRs cannot collide on a field number -- as `main` taking field 118 for `Sample` has since demonstrated. Plus the build machinery: the `contrib-delta` Maven profile and Cargo feature, and `dev/verify-contrib-delta-gate.sh`, which asserts a default build compiles no contrib classes, packages no contrib `META-INF/services` files, and links no contrib symbols. Where the hooks sit, and why. Both run *before* Comet's built-in guards for their scan kind, because a contrib may support things the built-in scan does not -- the Delta contrib synthesises `_metadata.*` in its own reader, and a contrib's table name may end in `files`/`snapshots` like an Iceberg metadata table. Applying those guards first would decline such a scan before the contrib was ever offered it. So `transformV1Scan` consults the contrib ahead of the metadata-column guard, and the Iceberg metadata-table check moves out of the outer `transformScan` match into `transformV2Scan`, after its hook. Core's per-path metadata handling is otherwise untouched: `main` serves `fileConstantMetadataColumns` natively in V1 and the Iceberg metadata columns in V2, and both keep doing so. Ownership contract. An implementation MUST return `None` for a scan it does not own: contribs are offered a scan one at a time and the first claim wins, so a contrib claiming another format's scan hides it from the contrib that could have read it, with the outcome depending on unspecified ServiceLoader ordering. "Own but cannot handle" is a distinct, expressible case -- claim the scan and terminate it with `withFallbackReason` rather than declining. Core cannot arbitrate competing claims (a claim is opaque; the only way to know a second contrib would also have claimed is to ask it, which is what claiming prevents), so the contract carries it. Tests. `CometScanContribSuite` covers the registry contract on a default build: no contribs registered (asserted against raw ServiceLoader discovery, not just the registry -- `contribs` swallows a ServiceConfigurationError, so "empty" alone is ambiguous), a stub discovered through a URLClassLoader whose claim is returned, decline-passes-through, first-claim-wins with later contribs not consulted, throw-is-a-decline, and LinkageError still propagating. `CometScanRuleSuite` gains a V1 case asserting the fallback *reason* for `_metadata.row_index`; verified red with the guard removed. Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
| Commit: | e4ec630 | |
|---|---|---|
| Author: | Chao Sun | |
| Committer: | GitHub | |
fix: preserve Catalyst nullability and field IDs in native Parquet writes (#5369)
| Commit: | 3df58d5 | |
|---|---|---|
| Author: | Andy Grove | |
| Committer: | GitHub | |
perf: intern QueryContext SQL text into a per-plan pool (up to 20x smaller serialized plans for TPC-DS) (#5204)
| Commit: | 60413eb | |
|---|---|---|
| Author: | Parth Chandra | |
| Committer: | GitHub | |
feat: support Iceberg metadata columns _pos, _spec, _file, and _partition (#4752) * feat: [iceberg] support iceberg metadata columns _pos, _spec, _file, _partition
| Commit: | 1ce9df1 | |
|---|---|---|
| Author: | Andy Grove | |
| Committer: | GitHub | |
feat: native uuid() implementation compatible with Spark (#5034) * feat: native uuid() implementation compatible with Spark * refactor: share MersenneTwister, use uuid crate for formatting Extract SparkMersenneTwister into internal/mersenne.rs so shuffle and uuid both depend on it rather than uuid reaching into the shuffle module. Format UUIDs via the uuid crate encoded into a pre-sized StringBuilder, removing per-row String allocation and the hand-rolled hex formatting. * test: address review feedback on uuid tests Fills the five coverage gaps flagged on #5034: - Multi-partition bit-for-bit: uuid(42) FROM ... DISTRIBUTE BY id exercises partitionIndex != 0, which a single-partition test cannot catch (both engines would silently agree even if partitionIndex were ignored). - Aliased projections: uuid(0) = uuid(0) (seeded) and uuid() = uuid() (unseeded, spark_answer_only) pin the stateful-alias semantics that freshCopyIfContainsStatefulExpression relies on. - Empty batch: a Rust test that evaluate() on a zero-row batch does not advance RNG state, guarding against a stray next_uuid() outside the row loop. - Golden fixture: five seeds x five UUIDs captured from Commons Math3's MersenneTwister (the exact RNG behind Spark's RandomUUIDGenerator) lock all 128 bits and cover negative / MIN / MAX seeds; catches next_long regressions that shuffle's next_int tests would miss. - No-scan path: length(uuid(0)) pins the OneRowRelation shape with a deterministic assertion alongside the existing SELECT uuid(0). * test: make uuid partition and seed coverage non-vacuous The DISTRIBUTE BY query never exercised partitionIndex != 0. `... FROM t DISTRIBUTE BY id` parses as RepartitionByExpression on top of the Project, so uuid ran on the scan's partitions, and five tiny files pack into a single FilePartition. Moved the projection above the exchange: SELECT uuid(42) FROM (SELECT id FROM test_uuid_parts DISTRIBUTE BY id) Add CometUuidExpressionSuite, which builds `Uuid(Some(seed))` directly through the existing version-shimmed `getColumnFromExpression`. This gives 3.4 and 3.5 a real cross-engine assertion for the first place: the SQL `uuid(seed)` form is 4.0+, so uuid_with_seed.sql is skipped there and the Rust golden constants were the only guard, with no way to confirm they came from Commons Math3 rather than from the Rust code they are meant to test. One test covers five seeds including negative/MIN/MAX; the other repartitions and asserts getNumPartitions > 1 before comparing, so the partition-offset coverage is checked rather than assumed. Verified by mutation: dropping `expr.seed.wrapping_add(planner.partition())` in UuidBuilder::build fails both new Scala tests. Restored, all pass on spark-3.5 and spark-4.1. Also: - Drop `spark_answer_only` from `uuid() = uuid()` in uuid.sql. The values are random but the comparison is not -- distinct fresh seeds mean false on every row -- so default query mode keeps the value check and adds the nativeness assertion that the guard depends on. - Record the real reason typeof(uuid(0)) cannot be used: Spark's TypeOf.doGenCode interpolates `child.dataType.catalogString` unquoted into generated Java, emitting `UTF8String.fromString(string)`. That code is byte-identical on 3.5, 4.0 and master, so it is not a 4.1 quirk as previously recorded. It is normally hidden because TypeOf.foldable is true and ConstantFolding removes the node; this suite excludes ConstantFolding, which exposes it. Captured as a harness constraint: no SQL fixture here can use typeof on a non-foldable input. - Note on the Rust golden test that CometUuidExpressionSuite is now its backstop, and that regenerating the constants from the Rust side would make it circular. No partition-count assertion is added to the SQL fixture: that suite compares Comet against Spark, so if the plan collapsed to one partition both engines would collapse identically and any such check would still pass. It is only assertable from Scala, which is where it now lives. * ci: register CometUuidExpressionSuite in PR build workflows The check-suites.py preflight requires every *Suite.scala to be listed in both pr_build_linux.yml and pr_build_macos.yml. Add the new uuid suite to the expressions bucket.
| Commit: | d2a61bd | |
|---|---|---|
| Author: | Matt Butrovich | |
| Committer: | GitHub | |
fix: disambiguate Iceberg scans that share a metadata_location (#5180)
| Commit: | c21fe12 | |
|---|---|---|
| Author: | Andy Grove | |
| Committer: | GitHub | |
fix: throw ARITHMETIC_OVERFLOW for Long.MinValue div -1 under ANSI mode (#5084) (#5146) * fix: throw ARITHMETIC_OVERFLOW for Long.MinValue div -1 under ANSI mode * refactor: reuse Spark's checkDivideOverflow and dedupe overflow check * test: use ARITHMETIC_OVERFLOW condition name in integral divide expect_error Matches the convention used by the other expect_error queries in arithmetic_ansi.sql. Verified against the spark-3.4, spark-3.5, spark-4.0, spark-4.1, and spark-4.2 profiles. (cherry picked from commit 38c3f8bce35642479117ba49f44a02fee31e6158)
| Commit: | e6577ed | |
|---|---|---|
| Author: | Andy Grove | |
| Committer: | GitHub | |
feat: support SampleExec natively for sampling without replacement (#5110) (#5144) * feat: support SampleExec natively for sampling without replacement Adds a native Comet operator for Spark's SampleExec when sampling is performed without replacement, covering DataFrame.sample, SQL TABLESAMPLE, and DataFrame.randomSplit. The native operator ports Spark's BernoulliCellSampler on top of the existing XorShiftRandom, drawing one value per row and seeding per partition with seed + partitionIndex, so it selects the same rows as Spark for a given seed. Sampling with replacement reports Unsupported and falls back to Spark. * refactor: address review feedback on native sample operator - Build the selection mask with BooleanBuffer::collect_bool, which packs bits without allocating an all-ones null buffer - Precompute the empty-range check in BernoulliCellSampler instead of repeating it per row, and mark sample() inline - Drop the redundant schema() override and the single-use sample_batch helper - Use checkSparkAnswerAndFallbackReason in the fallback tests so they assert the reason, not just the absence of the Comet operator - Trim documentation that restated the same paragraph in four places - Point the contributor guide at the directory operator serdes actually live in * fix: shim the SampleExec seed for Spark 4.2 Spark 4.2 changed SampleExec.seed from Long to Option[Long] and resolves an absent seed into a resolvedSeed field, which is what the operator samples with. Read the seed through CometSampleShim, following the CometCollectShim precedent for the same 4.2 divergence. * test: expand sample operator test coverage and add benchmark Add tests for ordering preservation above a sort, sampling above an aggregate, SQL TABLESAMPLE, and empty batch interleaving in the native operator. Add a Sample case to CometExecBenchmark comparing against Spark. (cherry picked from commit 4b091acc9acc442c301754876348480f49b7321d)
| Commit: | 4b091ac | |
|---|---|---|
| Author: | Andy Grove | |
| Committer: | GitHub | |
feat: support SampleExec natively for sampling without replacement (#5110) * feat: support SampleExec natively for sampling without replacement Adds a native Comet operator for Spark's SampleExec when sampling is performed without replacement, covering DataFrame.sample, SQL TABLESAMPLE, and DataFrame.randomSplit. The native operator ports Spark's BernoulliCellSampler on top of the existing XorShiftRandom, drawing one value per row and seeding per partition with seed + partitionIndex, so it selects the same rows as Spark for a given seed. Sampling with replacement reports Unsupported and falls back to Spark. * refactor: address review feedback on native sample operator - Build the selection mask with BooleanBuffer::collect_bool, which packs bits without allocating an all-ones null buffer - Precompute the empty-range check in BernoulliCellSampler instead of repeating it per row, and mark sample() inline - Drop the redundant schema() override and the single-use sample_batch helper - Use checkSparkAnswerAndFallbackReason in the fallback tests so they assert the reason, not just the absence of the Comet operator - Trim documentation that restated the same paragraph in four places - Point the contributor guide at the directory operator serdes actually live in * fix: shim the SampleExec seed for Spark 4.2 Spark 4.2 changed SampleExec.seed from Long to Option[Long] and resolves an absent seed into a resolvedSeed field, which is what the operator samples with. Read the seed through CometSampleShim, following the CometCollectShim precedent for the same 4.2 divergence. * test: expand sample operator test coverage and add benchmark Add tests for ordering preservation above a sort, sampling above an aggregate, SQL TABLESAMPLE, and empty batch interleaving in the native operator. Add a Sample case to CometExecBenchmark comparing against Spark.
| Commit: | 38c3f8b | |
|---|---|---|
| Author: | Andy Grove | |
| Committer: | GitHub | |
fix: throw ARITHMETIC_OVERFLOW for Long.MinValue div -1 under ANSI mode (#5084) * fix: throw ARITHMETIC_OVERFLOW for Long.MinValue div -1 under ANSI mode * refactor: reuse Spark's checkDivideOverflow and dedupe overflow check * test: use ARITHMETIC_OVERFLOW condition name in integral divide expect_error Matches the convention used by the other expect_error queries in arithmetic_ansi.sql. Verified against the spark-3.4, spark-3.5, spark-4.0, spark-4.1, and spark-4.2 profiles.
| Commit: | 8a9473c | |
|---|---|---|
| Author: | Andy Grove | |
| Committer: | GitHub | |
feat: native randstr implementation compatible with Spark (#5035) * feat: native randstr implementation compatible with Spark * refactor: gate randstr shape restrictions in getSupportLevel, avoid per-row UTF-8 scan Move the literal-length/literal-seed/non-negative-length restrictions from convert into getSupportLevel + getUnsupportedReasons so they surface in EXPLAIN and the compatibility docs, matching the rand/randn serde pattern. Build each string in a reused String buffer instead of validating a byte buffer per row. * test: add golden-value, partition-index, and filter coverage for randstr Address review feedback on the randstr PR: - Rust golden-value test asserting bit-for-bit equality with Spark 4.1.1 across positive, zero, and negative seeds, independent of the SQL tests. - Rust partition-index test exercising the seed + partition_index arithmetic. - Broaden the zero-length test across a wider seed set (seed is irrelevant when length is zero). - Rust large-length smoke test guarding the builder capacity math, and saturating_mul on the capacity hint to avoid overflow on huge lengths. - SQL test asserting an Int seed and its Long literal produce identical output (serde toLong sign extension). - SQL test running randstr through a native projection feeding a filter.
| Commit: | 69df0f1 | |
|---|---|---|
| Author: | Peter Lee | |
| Committer: | GitHub | |
feat: add CalendarIntervalType support (#4898) * Add CalendarIntervalType Arrow support * Support interval vectors in UDF and shuffle codegen * remove stale test * spotless apply
| Commit: | 2e907e5 | |
|---|---|---|
| Author: | Andy Grove | |
| Committer: | GitHub | |
feat: native collect_list / array_agg aggregate (#4720) * feat: native collect_list / array_agg aggregate Wires Spark's CollectList aggregate to datafusion-spark's SparkCollectList. array_agg, registered as a SQL alias of CollectList in FunctionRegistry, is also covered. Closes #2524. * fix: fall back collect_list/collect_set in multi-stage distinct aggregates A distinct aggregate combined with collect_list/collect_set produces a multi-stage plan (Partial -> PartialMerge -> Final). CollectList/CollectSet declare a BinaryType buffer in Spark but produce a native ArrayType state, so Comet cannot read a Spark-produced Binary buffer, nor round-trip its own ArrayType buffer across the intermediate PartialMerge stages. Both led to native crashes ("could not cast Binary to List" / "cast List to Binary"). Force these multi-stage aggregates to fall back to Spark consistently: - tag the feeding Partial when a PartialMerge stage of CollectList/CollectSet is present (CometExecRule.tagUnsafePartialAggregates), and - fall back a PartialMerge stage whose buffer was produced by a Spark partial (CometBaseAggregate.doConvert). Two-stage collect_list/collect_set continue to run natively. Patch the upstream SPARK-22223 plan-shape test to disable Comet, since native collect_list removes the ObjectHashAggregateExec it asserts on. Enabling fully-native multi-stage execution is tracked in #4724. * feat: fall back collect_list/collect_set on Spark 4.2 RESPECT NULLS Spark 4.2 adds an ignoreNulls field to CollectList and CollectSet, and collect_list(x) RESPECT NULLS sets it to false, keeping null elements. The native path delegates to SparkCollectList/SparkCollectSet, which always drop nulls, so it would silently return a different result from Spark. Add a per-version CometCollectShim that reads ignoreNulls (always true on Spark 3.4 through 4.1, where the field is absent) and fall back to Spark in getSupportLevel when it is false. Also rename QueryPlanSerde.hasIncompatibleBufferAgg to hasNativeArrayBufferAgg to describe what it detects: an aggregate whose native ArrayType state cannot round-trip Spark's declared BinaryType buffer. * test: accept Comet operators in SPARK-22223 instead of disabling Comet The SPARK-22223 ObjectHashAggregate test asserts on the executed plan. With Comet enabled, collect_list runs natively as CometHashAggregateExec, so ObjectHashAggregateExec is no longer present. Rather than disabling Comet, update the operator assertion to also accept CometHashAggregateExec, matching the pattern already used elsewhere in the diffs. The exchange assertion already matches ShuffleExchangeLike, which CometShuffleExchangeExec implements, so the single-shuffle check still holds. * style: reflow adjustOutputForNativeState doc comment Adding CollectList to the comment pushed a line past the column limit during the apache/main merge; reflow to satisfy spotless. * fix: coerce collect_list/collect_set nested field nullability collect_list and collect_set build their result list with all element fields marked nullable, but SparkCollectList/SparkCollectSet derive the return type from the child, preserving non-nullable nested fields. When the child is a nested type with a non-nullable inner field (e.g. a struct field built from non-nullable columns), the declared aggregate output disagrees with the array the accumulator produces, and the grouped native AggregateExec fails validating its output batch with "column types must match schema types". Cast the collect child to the all-nullable variant of its type so the declared and produced types stay consistent. * review: drop unreachable buffer-source block; consolidate 3.4/3.5 shim; add FILTER + map tests operators.scala: drop the CollectList/CollectSet PartialMerge fallback block. It is unreachable now that the general missingCometProducer + aggsNotSupportingMixedExecution guard just above returns None for the same case. CometExecRule.scala: add a comment on the collect-specific tagging block explaining it is separate from the tagging block just above because canAggregateBeConverted skips the child-native check, so an all-native distinct collect chain would otherwise slip past. QueryPlanSerde.scala: narrow the hasNativeArrayBufferAgg doc comment to describe what the code actually matches and note that Percentile has the same shape but is not matched here. Consolidate CometCollectShim: move the identical spark-3.4 and spark-3.5 copies to spark-3.x. Add collect_list FILTER (WHERE ...) test and a map-input test hitting make_all_fields_nullable's Map arm (size-only via spark_answer_only, since sort_array cannot order a MapType). * test: cover collect_list/collect_set RESPECT NULLS fallback on Spark 4.2 Add a Spark 4.2-gated SQL file test asserting that collect_list and collect_set with RESPECT NULLS fall back to Spark with Spark-identical results, and expand the getSupportLevel comments to explain that the fallback branch is only reachable on 4.2+ (a no-op on 3.4 through 4.1).
| Commit: | bc53ad5 | |
|---|---|---|
| Author: | Matt Butrovich | |
| Committer: | GitHub | |
feat: Iceberg table format V3: native table decryption, fall back for other V3 features (#4991)
| Commit: | 0761e54 | |
|---|---|---|
| Author: | Matt Butrovich | |
| Committer: | GitHub | |
perf: dedupe Iceberg residuals and delete files in native scan serde (#4982)
| Commit: | f06aa31 | |
|---|---|---|
| Author: | Andy Grove | |
| Committer: | GitHub | |
feat: support approx_count_distinct aggregate expression (#4819) * feat: support approx_count_distinct aggregate expression Add native support for Spark's approx_count_distinct, a faithful port of Spark's HyperLogLogPlusPlus / HyperLogLogPlusPlusHelper. Each non-null input is hashed with Comet's Spark-compatible XxHash64 (seed 42, floats normalized first), and the HyperLogLog++ registers are stored in Spark's exact packed-Long buffer layout (10 six-bit registers per word). The cardinality is estimated with the same linear-counting and bias-correction tables Spark uses, so results are bit-identical to Spark and the partial-aggregation state matches Spark's aggBufferSchema. Includes a vectorized GroupsAccumulator, SQL file tests comparing against Spark across a range of cardinalities, native unit tests, benchmark coverage, and documentation updates. * feat: address review feedback on approx_count_distinct - restrict decimal inputs to precision <= 18 (Spark hashes wider decimals through BigDecimal, which the native i128 path does not match) - set supportsMixedPartialFinal=true; the register buffer matches Spark's aggBufferSchema, enabling mixed Comet/Spark partial and final aggregation - add getUnsupportedReasons so the Compatibility Guide reflects the type limits - derive Hash, add assert/debug_assert invariants and a float-order comment, reuse the shared normalize_float, drop a redundant field, make bias correction lazy, and reuse a per-accumulator hash scratch buffer - compact BIAS_DATA layout with rustfmt::skip (values unchanged) - add tests: decimal boundary and wide-decimal fallback, non-UTC timestamp, collated-string fallback, and mixed partial/final plan shape
| Commit: | 1981091 | |
|---|---|---|
| Author: | Andy Grove | |
| Committer: | GitHub | |
feat: add spark.comet.shuffle.maxBufferBytes to cap native shuffle writer memory (#4989)
| Commit: | c65a5ee | |
|---|---|---|
| Author: | Andy Grove | |
| Committer: | GitHub | |
feat: support gzip compression in native Parquet writes (#4930) * feat: support gzip compression in the native Parquet writer Add Gzip to the CompressionCodec proto enum and introduce a Parquet-specific ParquetCompression enum in parquet_writer.rs so the native writer can honor gzip without affecting the shuffle codec. The planner maps the new proto variant to ParquetCompression::Gzip, which writes with parquet's default GzipLevel (6), matching parquet-mr's zlib default. The shuffle writer path is unchanged and still rejects Gzip via its catch-all error arm. * feat: write Parquet natively when the compression codec is gzip * fix: request parquet-rs codec features explicitly for native Parquet writer The native writer's gzip, snappy, lz4, and zstd support only worked because Cargo feature unification happened to pull in parquet-rs's flate2/snap/lz4/zstd features via other workspace dependencies. Declare these features directly on the parquet dependency so the native writer does not silently lose codec support if that unification changes. * fix: honor parquet.compression precedence and uncompressed codec alias parseCompressionCodec only checked the compression option and the SQLConf default, skipping the parquet.compression option that Spark's own ParquetOptions treats as the middle rung of precedence. Fix the lookup to match Spark's compression, parquet.compression, SQLConf order, and accept "uncompressed" as an alias for "none" since Spark does the same. Add coverage for the parquet.compression option taking precedence over the SQLConf default, for uncompressed as a none alias, and for an unsupported codec (brotli) causing the write to fall back to Spark's own writer instead of CometNativeWriteExec. * refactor: inline single-call-site compression_to_parquet delegate compression_to_parquet was a one-line wrapper around ParquetCompression::to_parquet with a single caller; call it directly instead. * test: replace brittle brotli fallback test with lz4_raw round trip The unsupported-codec fallback test relied on Spark's write failing with ClassNotFoundException for BrotliCodec, so it only passed because the environment lacks a Brotli codec class rather than because Comet routed the write through the fallback path. Use lz4_raw instead, which Spark can write directly via parquet-mr and Comet does not support, so the test can assert a real successful round trip and the correct footer codec instead of intercepting an unrelated failure. Revert the allowFailure plumbing added to captureWritePlan for that test since it is no longer needed. * test: skip lz4_raw fallback test on Spark 3.4 Spark 3.4's PARQUET_COMPRESSION config validates against a fixed set of codecs (brotli, uncompressed, lz4, gzip, lzo, snappy, none, zstd) that does not include lz4_raw, so withSQLConf threw before the fallback path ran. Guard the test with assume(isSpark35Plus, ...) so it runs on versions where lz4_raw is a valid codec value. * test: cover full three-tier codec precedence for Parquet writes Adds a test that pins the `compression` > `parquet.compression` > `spark.sql.parquet.compression.codec` precedence: SQLConf is `zstd`, `parquet.compression` is `snappy`, and `compression` is `gzip`; the written file must report GZIP. A leak from either lower layer would surface as a codec mismatch. The existing test still covers `parquet.compression` beating the SQLConf default when `compression` is absent.
| Commit: | 88039f1 | |
|---|---|---|
| Author: | Oleks V | |
| Committer: | GitHub | |
chore: use Datafusion `substring` (#4161)
| Commit: | eb5b761 | |
|---|---|---|
| Author: | Andy Grove | |
| Committer: | GitHub | |
feat: support approx_percentile / percentile_approx aggregate (#4801)
| Commit: | a282d29 | |
|---|---|---|
| Author: | Matt Butrovich | |
| Committer: | GitHub | |
chore: [branch-17] backport #4760 (#4829) * fix: size Iceberg delete files in the native scan to avoid dropping deletes (#4760) (cherry picked from commit b70e529ae945393ac24cb2647d560de1cd747f2a) * run prettier
| Commit: | a4b6bc9 | |
|---|---|---|
| Author: | Andy Grove | |
| Committer: | GitHub | |
feat: support shuffle array expression (#4797) * feat: support shuffle array expression Add Comet support for Spark's `shuffle` array expression with full Spark compatibility. Comet reproduces Spark's exact random permutation by porting Apache Commons Math3's MersenneTwister and the inside-out Fisher-Yates algorithm from RandomIndicesGenerator, combining the resolved random seed with the partition index like Spark does. The native ShuffleExpr is a stateful PhysicalExpr that carries the RNG state across batches within a partition, mirroring the existing Rand/Randn implementation. * review: address feedback on shuffle expression - rename RNG to PRNG in doc comments - drop unreachable DataType::Null arm from ShuffleExpr::evaluate
| Commit: | f994b23 | |
|---|---|---|
| Author: | Andy Grove | |
| Committer: | GitHub | |
feat: support PreciseTimestampConversion for native batch time-window grouping (#4784) * feat: support PreciseTimestampConversion for native batch time-window grouping Wire Spark's internal PreciseTimestampConversion expression, which the analyzer emits when resolving window()/session_window() grouping. It is a pure reinterpret between the timestamp types and Long, so it maps to an Arrow cast between microsecond Timestamp and Int64. Also support the KnownNullable tagging expression that the window resolution wraps around window bounds. Together these let batch tumbling and sliding time-window aggregations run natively. * test: convert time-window tests to a Comet SQL file test; simplify serde Convert the window() coverage from Scala tests to a SQL file test at expressions/datetime/window.sql. The default query mode still asserts native execution (checkSparkAnswerAndOperator), and a ConfigMatrix over session timezones confirms timestamp-window results match Spark in every zone. Simplify CometPreciseTimestampConversion.convert to use the existing optExprWithFallbackReason helper instead of a hand-rolled if/else. * docs: point session_window support entry at dedicated issue #4785
| Commit: | 464afe1 | |
|---|---|---|
| Author: | Andy Grove | |
| Committer: | GitHub | |
feat: support interval types and make_ym_interval / make_dt_interval (#4541) * feat: support interval types and make_ym_interval / make_dt_interval [skip ci] Implements the type-support prerequisite from issue #4540: add Spark YearMonthIntervalType and DayTimeIntervalType as physical types that round-trip through Comet's Arrow FFI, and route the make_ym_interval / make_dt_interval constructors through the JVM codegen dispatcher so they execute natively and match Spark exactly. Type plumbing: - proto: add YEAR_MONTH_INTERVAL (18) and DAY_TIME_INTERVAL (19) to DataTypeId. - native serde.rs: map them to Arrow Interval(YearMonth) and Duration(Microsecond) respectively. DayTime stores microseconds in an int64, which matches Duration(Microsecond) rather than the lossy Interval(DayTime) {days, millis}. - Utils.toArrowType / fromArrowType: same mapping on the JVM side. - QueryPlanSerde.serializeDataType: emit the new type ids. - CometBatchKernelCodegen: accept the two interval types in isSupportedDataType and resolve IntervalYearVector / DurationVector; emit primitive set() writes. Expressions: - make_ym_interval -> CometMakeYMInterval, make_dt_interval -> CometMakeDTInterval, both via CometCodegenDispatch, registered in temporalExpressions. CalendarIntervalType and interval arithmetic remain follow-ups under #4540. Tests: SQL file tests for both constructors assert answer parity and native execution (checkSparkAnswerAndOperator), covering column and literal inputs, defaults, negatives, and nulls. * test: add overflow cases for make_ym_interval and make_dt_interval Confirm the codegen-dispatch path propagates Spark's arithmetic-overflow exception identically. The expect_error pattern uses the lowercase word overflow so it matches every Spark version: 4.x raises INTERVAL_ARITHMETIC_OVERFLOW while 3.x raises a raw ArithmeticException.
| Commit: | d2d976e | |
|---|---|---|
| Author: | Andy Grove | |
| Committer: | GitHub | |
feat: support exact percentile and median aggregates natively (#4542) * feat: support exact percentile aggregate natively [skip ci] Wire Spark's exact `Percentile` aggregate (and the `percentile_cont` ANSI form, which Spark rewrites to `Percentile`) to DataFusion's `percentile_cont` aggregate. DataFusion uses the same `index = p * (n - 1)` linear interpolation as Spark, so results match for the common single-percentage form. - proto: add `Percentile` AggExpr message (child, percentage, datatype). - native planner: map it to `percentile_cont_udaf()` with [child, percentile]. - CometPercentile serde: Compatible for a single literal double percentage, default frequency, and numeric input; the child is cast to double so the native result is DoubleType. Array-of-percentages, a non-default frequency argument, and interval inputs fall back to Spark. - operators.adjustOutputForNativeState: map Percentile's TypedImperativeAggregate Binary partial buffer to the native List<Float64> state (ArrayType(DoubleType)), mirroring CollectSet, so the partial/shuffle/final exchange schema is correct. Codegen dispatch is not applicable: aggregates (TypedImperativeAggregate) cannot run in the per-row scalar kernel, so native is the only path. Tests: SQL file test covering global, grouped, integer-input, all-null, exact and interpolated percentiles, plus fallback assertions for the array and frequency forms. No new regressions in the SQL suite. * bench: add percentile cases to CometAggregateExpressionBenchmark [skip ci] * feat: guard percentile DESC fallback, broaden tests and docs Address audit findings on the native percentile aggregate: - getSupportLevel now falls back for the descending WITHIN GROUP form (percentile_cont/disc WITHIN GROUP ... ORDER BY ... DESC on Spark 4.0+), where Percentile.reverse=true. The native percentile_cont always interpolates ascending, so the descending form would return a wrong answer. - Extract the fallback reason strings into shared private vals and add a getUnsupportedReasons override so they reach the compatibility guide. - Expand percentile.sql with long/float/decimal/smallint/tinyint inputs, negative values, and median() coverage (median rewrites to percentile). - Add percentile_within_group.sql (Spark 4.0+) covering the ascending native path and the descending fallback. - Mark percentile, percentile_cont, and median as supported in the expression reference and record the cross-version audit in agg_funcs.md. - Document the DataFusion 6-decimal interpolation quantization (#4719). * fix: mark percentile aggregates Incompatible by default (#4719) DataFusion's percentile_cont quantizes the linear interpolation weight to 6 decimal places, so a deeply-interpolated percentile can differ from Spark by up to roughly (upper - lower) * 1e-6. Gate the otherwise-supported percentile / median / percentile_cont form as Incompatible so it falls back to Spark by default and is opt-in via spark.comet.expression.Percentile.allowIncompatible=true. Add the allowIncompatible config to the percentile SQL file tests so the native path stays covered, and update the expression reference and audit doc to reflect the opt-in status. * style: wrap percentile precision comment under 100 chars
| Commit: | b70e529 | |
|---|---|---|
| Author: | Matt Butrovich | |
| Committer: | GitHub | |
fix: size Iceberg delete files in the native scan to avoid dropping deletes (#4760)
| Commit: | ba6429f | |
|---|---|---|
| Author: | Oleks V | |
| Committer: | GitHub | |
feat: extend native windows support (#4209)
| Commit: | 1ff9555 | |
|---|---|---|
| Author: | Andy Grove | |
| Committer: | GitHub | |
fix: reject Parquet INT96 as TimestampNTZ on Spark 3.x (#4357) * fix: reject Parquet TimestampLTZ as TimestampNTZ on Spark 3.x for native_datafusion Pre-Spark-4 (SPARK-36182) rejects reading a Parquet TimestampLTZ column as TimestampNTZ; native_datafusion previously did not, and silently returned the UTC instant. Plumb a per-Spark-version flag from ShimCometConf through the NativeScan proto into SparkParquetOptions, and gate a new rejection arm in the schema adapter on it. INT96 remains a gap because DataFusion's coerce_int96 strips the source timezone before the schema adapter runs, so it is indistinguishable from a true TIMESTAMP_NTZ source. Compatibility guide updated to describe the correctness implications. * style: cargo fmt * fix: reject Parquet INT96 as TimestampNTZ on Spark 3.x for native_datafusion Closes the INT96 gap left by the parent commit. INT96 columns previously surfaced as Timestamp(us, None) on the Rust side because DataFusion's coerce_int96 stripped the timezone, making them indistinguishable from a true TimestampNTZ source. With the new coerce_int96_tz option (DataFusion PR apache/datafusion#22318) we ask DataFusion to coerce INT96 to Timestamp(us, Some("UTC")), restoring the LTZ signal the schema adapter already pattern-matches against. Comet-side change is small: set coerce_int96_tz = "UTC" alongside the existing coerce_int96 = "us"; unskip the INT96 + native_datafusion variant of ParquetTimestampLtzAsNtzSuite. The schema adapter's existing Timestamp(_, Some(_)) -> Timestamp(_, None) rejection now fires for INT96 reads as well. EXPERIMENTAL: pinned via [patch.crates-io] to an andygrove/datafusion fork branch. Cannot be merged until apache/datafusion#22318 ships in a release.
| Commit: | 175a77d | |
|---|---|---|
| Author: | Bhargava Vadlamani | |
| Committer: | GitHub | |
Revert "feat: Native Broadcast nested loop join support (#4429)" This reverts commit 4c88f5d4863f55c370b05cc104dcd0950b3ad2bd.
| Commit: | 4c88f5d | |
|---|---|---|
| Author: | Bhargava Vadlamani | |
| Committer: | GitHub | |
feat: Native Broadcast nested loop join support (#4429) * native_support_broadcast_nested_loop_join
| Commit: | 9e86dd9 | |
|---|---|---|
| Author: | Matt Butrovich | |
| Committer: | GitHub | |
perf: replace CometBatchIterator FFI input path with the Arrow C Stream Interface (#4572)
| Commit: | aa6be27 | |
|---|---|---|
| Author: | Matt Butrovich | |
| Committer: | GitHub | |
perf: avoid FFI import/export between native subtree and ShuffleWriter (#4507)
| Commit: | a08cb4e | |
|---|---|---|
| Author: | Matt Butrovich | |
| Committer: | GitHub | |
feat: vendor-pluggable S3 credentials for native scans (#4309)
| Commit: | b23b760 | |
|---|---|---|
| Author: | Parth Chandra | |
| Committer: | GitHub | |
feat: implement make_time and to_time (#4256)
| Commit: | 0ca37e1 | |
|---|---|---|
| Author: | Andy Grove | |
| Committer: | GitHub | |
feat: add support for `posexplode` and `posexplode_outer` (#4270)