Proto commits in apache/datafusion-comet

These commits are when the Protocol Buffers files have changed: (only the last 100 relevant commits are shown)

2026-07-06

chore: [branch-17] backport #4760 (#4829) * fix: size Iceberg delete files in the native scan to avoid dropping deletes (#4760) (cherry picked from commit b70e529ae945393ac24cb2647d560de1cd747f2a) * run prettier

2026-07-03

feat: support shuffle array expression (#4797) * feat: support shuffle array expression Add Comet support for Spark's `shuffle` array expression with full Spark compatibility. Comet reproduces Spark's exact random permutation by porting Apache Commons Math3's MersenneTwister and the inside-out Fisher-Yates algorithm from RandomIndicesGenerator, combining the resolved random seed with the partition index like Spark does. The native ShuffleExpr is a stateful PhysicalExpr that carries the RNG state across batches within a partition, mirroring the existing Rand/Randn implementation. * review: address feedback on shuffle expression - rename RNG to PRNG in doc comments - drop unreachable DataType::Null arm from ShuffleExpr::evaluate

The documentation is generated from this commit.

2026-07-01

feat: support PreciseTimestampConversion for native batch time-window grouping (#4784) * feat: support PreciseTimestampConversion for native batch time-window grouping Wire Spark's internal PreciseTimestampConversion expression, which the analyzer emits when resolving window()/session_window() grouping. It is a pure reinterpret between the timestamp types and Long, so it maps to an Arrow cast between microsecond Timestamp and Int64. Also support the KnownNullable tagging expression that the window resolution wraps around window bounds. Together these let batch tumbling and sliding time-window aggregations run natively. * test: convert time-window tests to a Comet SQL file test; simplify serde Convert the window() coverage from Scala tests to a SQL file test at expressions/datetime/window.sql. The default query mode still asserts native execution (checkSparkAnswerAndOperator), and a ConfigMatrix over session timezones confirms timestamp-window results match Spark in every zone. Simplify CometPreciseTimestampConversion.convert to use the existing optExprWithFallbackReason helper instead of a hand-rolled if/else. * docs: point session_window support entry at dedicated issue #4785

feat: support interval types and make_ym_interval / make_dt_interval (#4541) * feat: support interval types and make_ym_interval / make_dt_interval [skip ci] Implements the type-support prerequisite from issue #4540: add Spark YearMonthIntervalType and DayTimeIntervalType as physical types that round-trip through Comet's Arrow FFI, and route the make_ym_interval / make_dt_interval constructors through the JVM codegen dispatcher so they execute natively and match Spark exactly. Type plumbing: - proto: add YEAR_MONTH_INTERVAL (18) and DAY_TIME_INTERVAL (19) to DataTypeId. - native serde.rs: map them to Arrow Interval(YearMonth) and Duration(Microsecond) respectively. DayTime stores microseconds in an int64, which matches Duration(Microsecond) rather than the lossy Interval(DayTime) {days, millis}. - Utils.toArrowType / fromArrowType: same mapping on the JVM side. - QueryPlanSerde.serializeDataType: emit the new type ids. - CometBatchKernelCodegen: accept the two interval types in isSupportedDataType and resolve IntervalYearVector / DurationVector; emit primitive set() writes. Expressions: - make_ym_interval -> CometMakeYMInterval, make_dt_interval -> CometMakeDTInterval, both via CometCodegenDispatch, registered in temporalExpressions. CalendarIntervalType and interval arithmetic remain follow-ups under #4540. Tests: SQL file tests for both constructors assert answer parity and native execution (checkSparkAnswerAndOperator), covering column and literal inputs, defaults, negatives, and nulls. * test: add overflow cases for make_ym_interval and make_dt_interval Confirm the codegen-dispatch path propagates Spark's arithmetic-overflow exception identically. The expect_error pattern uses the lowercase word overflow so it matches every Spark version: 4.x raises INTERVAL_ARITHMETIC_OVERFLOW while 3.x raises a raw ArithmeticException.

2026-06-30

feat: support exact percentile and median aggregates natively (#4542) * feat: support exact percentile aggregate natively [skip ci] Wire Spark's exact `Percentile` aggregate (and the `percentile_cont` ANSI form, which Spark rewrites to `Percentile`) to DataFusion's `percentile_cont` aggregate. DataFusion uses the same `index = p * (n - 1)` linear interpolation as Spark, so results match for the common single-percentage form. - proto: add `Percentile` AggExpr message (child, percentage, datatype). - native planner: map it to `percentile_cont_udaf()` with [child, percentile]. - CometPercentile serde: Compatible for a single literal double percentage, default frequency, and numeric input; the child is cast to double so the native result is DoubleType. Array-of-percentages, a non-default frequency argument, and interval inputs fall back to Spark. - operators.adjustOutputForNativeState: map Percentile's TypedImperativeAggregate Binary partial buffer to the native List<Float64> state (ArrayType(DoubleType)), mirroring CollectSet, so the partial/shuffle/final exchange schema is correct. Codegen dispatch is not applicable: aggregates (TypedImperativeAggregate) cannot run in the per-row scalar kernel, so native is the only path. Tests: SQL file test covering global, grouped, integer-input, all-null, exact and interpolated percentiles, plus fallback assertions for the array and frequency forms. No new regressions in the SQL suite. * bench: add percentile cases to CometAggregateExpressionBenchmark [skip ci] * feat: guard percentile DESC fallback, broaden tests and docs Address audit findings on the native percentile aggregate: - getSupportLevel now falls back for the descending WITHIN GROUP form (percentile_cont/disc WITHIN GROUP ... ORDER BY ... DESC on Spark 4.0+), where Percentile.reverse=true. The native percentile_cont always interpolates ascending, so the descending form would return a wrong answer. - Extract the fallback reason strings into shared private vals and add a getUnsupportedReasons override so they reach the compatibility guide. - Expand percentile.sql with long/float/decimal/smallint/tinyint inputs, negative values, and median() coverage (median rewrites to percentile). - Add percentile_within_group.sql (Spark 4.0+) covering the ascending native path and the descending fallback. - Mark percentile, percentile_cont, and median as supported in the expression reference and record the cross-version audit in agg_funcs.md. - Document the DataFusion 6-decimal interpolation quantization (#4719). * fix: mark percentile aggregates Incompatible by default (#4719) DataFusion's percentile_cont quantizes the linear interpolation weight to 6 decimal places, so a deeply-interpolated percentile can differ from Spark by up to roughly (upper - lower) * 1e-6. Gate the otherwise-supported percentile / median / percentile_cont form as Incompatible so it falls back to Spark by default and is opt-in via spark.comet.expression.Percentile.allowIncompatible=true. Add the allowIncompatible config to the percentile SQL file tests so the native path stays covered, and update the expression reference and audit doc to reflect the opt-in status. * style: wrap percentile precision comment under 100 chars

fix: size Iceberg delete files in the native scan to avoid dropping deletes (#4760)

2026-06-24

feat: extend native windows support (#4209)

2026-06-22

fix: reject Parquet INT96 as TimestampNTZ on Spark 3.x (#4357) * fix: reject Parquet TimestampLTZ as TimestampNTZ on Spark 3.x for native_datafusion Pre-Spark-4 (SPARK-36182) rejects reading a Parquet TimestampLTZ column as TimestampNTZ; native_datafusion previously did not, and silently returned the UTC instant. Plumb a per-Spark-version flag from ShimCometConf through the NativeScan proto into SparkParquetOptions, and gate a new rejection arm in the schema adapter on it. INT96 remains a gap because DataFusion's coerce_int96 strips the source timezone before the schema adapter runs, so it is indistinguishable from a true TIMESTAMP_NTZ source. Compatibility guide updated to describe the correctness implications. * style: cargo fmt * fix: reject Parquet INT96 as TimestampNTZ on Spark 3.x for native_datafusion Closes the INT96 gap left by the parent commit. INT96 columns previously surfaced as Timestamp(us, None) on the Rust side because DataFusion's coerce_int96 stripped the timezone, making them indistinguishable from a true TimestampNTZ source. With the new coerce_int96_tz option (DataFusion PR apache/datafusion#22318) we ask DataFusion to coerce INT96 to Timestamp(us, Some("UTC")), restoring the LTZ signal the schema adapter already pattern-matches against. Comet-side change is small: set coerce_int96_tz = "UTC" alongside the existing coerce_int96 = "us"; unskip the INT96 + native_datafusion variant of ParquetTimestampLtzAsNtzSuite. The schema adapter's existing Timestamp(_, Some(_)) -> Timestamp(_, None) rejection now fires for INT96 reads as well. EXPERIMENTAL: pinned via [patch.crates-io] to an andygrove/datafusion fork branch. Cannot be merged until apache/datafusion#22318 ships in a release.

2026-06-12

Revert "feat: Native Broadcast nested loop join support (#4429)" This reverts commit 4c88f5d4863f55c370b05cc104dcd0950b3ad2bd.

2026-06-11

feat: Native Broadcast nested loop join support (#4429) * native_support_broadcast_nested_loop_join

perf: replace CometBatchIterator FFI input path with the Arrow C Stream Interface (#4572)

2026-06-10

perf: avoid FFI import/export between native subtree and ShuffleWriter (#4507)

2026-05-28

feat: vendor-pluggable S3 credentials for native scans (#4309)

2026-05-20

feat: implement make_time and to_time (#4256)

2026-05-18

feat: add support for `posexplode` and `posexplode_outer` (#4270)

2026-05-17

fix: complete native_datafusion Parquet schema-mismatch rejections (#4229)

2026-05-08

feat: add JVM UDF framework for native execution (#4232)

2026-05-06

feat: support Parquet field ID matching in native_datafusion scan (#4216)

2026-05-05

fix: support Spark 4.1 BloomFilter V2 format and bit-scattering (#4196)

fix: [Spark 4.1.1] preserve parent struct nullness when all requested fields missing in Parquet (#4190)

2026-05-04

feat: support `PartialMerge` aggregation mode (#4003)

2026-05-03

feat: support regular BuildRight+LeftAnti hash join (#4073)

2026-04-22

feat: Support Spark expression: arrays_zip (#3643) * Define ArraysZip expr proto * Create ArraysZip SerDe * Register SerDe to arrayExpressions * Add SQL test * Register expression to planner * Rust wrapper around DF's arrays_zip * Null checks * Update supported Spark expressions doc

2026-04-20

feat: support `collect_set` (#3954)

2026-04-13

feat: Support Spark expression hours (#3804) * feat: Add Spark V2 partition transform `Hours` to calculate hours since epoch from timestamps.

2026-04-08

fix: Make cast string to timestamp compatible with Spark (#3884) * fix: Make cast string to timestamp compatible with Spark Add addtional formats and handle edge cases. Update compatibility guide Spark version specific behaviour for cast string to timestamp

2026-04-06

feat: support LEAD and LAG window functions with IGNORE NULLS (#3876) * feat: support LAG window function with IGNORE NULLS - Add ignore_nulls field to WindowExpr proto message - Serialize Lag window function with its ignoreNulls flag in CometWindowExec - Extend find_df_window_function to also look up WindowUDFs (not just AggregateUDFs) - Pass ignore_nulls to DataFusion's create_window_expr - Enable previously-ignored LAG tests Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: make LAG window tests deterministic by adding secondary sort key ORDER BY b alone has ties, causing Spark and DataFusion to produce different but both-valid row orderings. Add c as a secondary sort key so tie-breaking is deterministic and results are comparable. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: enable WindowExec allowIncompatible for LAG tests CometWindowExec is marked Incompatible by default. Add allowIncompatible=true config so LAG tests actually run via Comet and checkSparkAnswerAndOperator can verify native execution. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: skip partition/order validation for offset window functions LAG/LEAD (FrameLessOffsetWindowFunction) support arbitrary partition and order specs. The existing validatePartitionAndSortSpecsForWindowFunc check (which requires partition columns == order columns) is only needed for aggregate window functions, not offset functions. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * feat: add LEAD support and IGNORE NULLS test for LAG window function - Handle Lead case in windowExprToProto alongside Lag - Add comment explaining hasOnlyOffsetFunctions guard - Add test for LAG IGNORE NULLS - Enable LEAD tests with allowIncompatible config and deterministic ORDER BY Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * test: add LEAD with IGNORE NULLS test case Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * test: add SQL tests for LAG/LEAD window functions with IGNORE NULLS Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>

2026-03-30

feat: support SQL aggregate FILTER (WHERE ...) clause in native execution (#3835)

2026-03-21

perf: stop using FFI in native shuffle read path (#3731)

2026-03-07

feat: [ANSI] Ansi sql error messages (#3580) * feat: [ANSI] Ansi sql error messages

2026-02-27

chore: bump iceberg-rust dependency to latest [iceberg] (#3606)

2026-02-25

perf: Add Comet config for native Iceberg reader's data file concurrency (#3584)

2026-02-13

feat: CometNativeScan per-partition plan serde (#3511)

2026-02-08

feat: CometExecRDD supports per-partition plan data, reduce Iceberg native scan serialization, add DPP for Iceberg scans (#3349)

2026-02-07

Feat: to_csv (#3004)

2026-01-25

feat: Add support for round-robin partitioning in native shuffle (#3076)

2026-01-23

perf: [iceberg] Use protobuf instead of JSON to serialize Iceberg partition values (#3247) * perf: Use protobuf instead of JSON to serialize Iceberg partition values

Experimental: Native CSV files read (#3044)

2026-01-21

feat: Add support for `unix_timestamp` function (#2936)

2026-01-15

feat: Comet Writer should respect object store settings (#3042)

2025-12-29

feat: Support ANSI mode avg expr (int inputs) (#2817)

2025-12-20

feat: Add partial support for `from_json` (#2934)

2025-12-18

perf: [iceberg] Deduplicate serialized metadata for Iceberg native scan (#2933)

2025-12-16

feat: Make shuffle writer buffer size configurable (#2899)

2025-12-05

feat: Support ANSI mode SUM (Decimal types) (#2826)

feat: Add support for `explode` and `explode_outer` for array inputs (#2836)

2025-12-02

feat: Partially implement file commit protocol for native Parquet writes (#2828)

2025-11-26

feat: Add experimental support for native Parquet writes (#2812)

2025-11-20

feat: [iceberg] Native scan by serializing FileScanTasks to iceberg-rust (#2528)

2025-11-06

feat: Add support for `abs` (#2689)

2025-10-29

chore: Remove code for unpacking dictionaries prior to FilterExec (#2659)

2025-10-13

feat:support ansi mode rounding function (#2542)

2025-10-07

feat: Parquet Modular Encryption with Spark KMS for native readers (#2447)

2025-09-24

fix: distributed RangePartitioning bounds calculation with native shuffle (#2258)

2025-09-20

fix: regressions in `CometToPrettyStringSuite` (#2384) * Introduce BinaryOutputStyle from Spark 4.0 * Allow casting from binary to string * Pass binaryOutputStyle to query plan serde * Take binaryOutputStyle in planner * Implement Spark-style ToPrettyString * Match file name w/ test name * Test all 5 BinaryOutputStyle in Spark 4.0 * Fix package: 'org.apache.sql' -> 'org.apache.spark.sql' * Add CometToPrettyStringSuite back to CI * Specify binaryOutputStyle for Spark 3.4 * Let Comet deal with non pretty string casting * Enable binary to string casting test * Attempt to fix the build; ToPrettyString is Spark 3.5+ * Removed resolved issues * Type casting only function * Extract test setup logic to CometFuzzTestBase * Move binary_output_style proto <-> enum mapping to core * Move BinaryOutputStyle from cast.rs to lib.rs * Remove incorrect comments

2025-09-02

perf: Avoid FFI copy in `ScanExec` when reading data from exchanges (#2268)

2025-08-30

fix: split expr.proto file (new) (#2267) * split expr.proto file

2025-08-21

chore: Pass Spark configs to native `createPlan` (#2180)

2025-08-19

fix: [branch-0.9] Backport FFI fix (#2164)

2025-08-15

chore: Simplify approach to avoiding memory corruption due to buffer reuse (#2156)

2025-08-13

feat: limit with offset support (#2070) * feat: positive offset support for queries with limit * fix stability suite and spark tests * fix stability suite and spark tests * rolled back accidentally changed logic in a test * replace assert with explicit Err return * fix clippy warnings * Update native/core/src/execution/planner.rs Co-authored-by: Oleks V <comphead@users.noreply.github.com> * Update spark/src/main/scala/org/apache/spark/sql/comet/CometCollectLimitExec.scala Co-authored-by: Oleks V <comphead@users.noreply.github.com> * Update native/core/src/execution/planner.rs Co-authored-by: Oleks V <comphead@users.noreply.github.com> * fix compile errors and format warnings * zero offset and limit test --------- Co-authored-by: Oleks V <comphead@users.noreply.github.com>

2025-08-11

feat: Support Array Literal (#2057) * feat: support literal for ARRAY top level

2025-08-04

Chore: implement string_space as ScalarUDFImpl (#2041)

2025-07-30

feat: monotonically_increasing_id and spark_partition_id implementation (#2037)

2025-07-21

fix: Refactor arithmetic serde and fix correctness issues with EvalMode::TRY (#2018)

2025-07-18

feat: randn expression support (#2010)

2025-07-10

refactor: remove RightSemi and RightAnti join types from planner and proto (#1935) Co-authored-by: Andy Grove <agrove@apache.org>

2025-07-09

chore: use DF scalar functions for StartsWith, EndsWith, Contains, DF LikeExpr (#1887)

2025-06-25

feat: rand expression support (#1199)

2025-06-24

feat: Implement ToPrettyString (#1921)

2025-06-18

feat: support RangePartitioning with native shuffle (#1862)

2025-06-16

chore: Implement date_trunc as ScalarUDFImpl (#1880)

2025-06-03

feat: Translate Hadoop S3A configurations to object_store configurations (#1817)

2025-06-02

Chore: implement bit_not as ScalarUDFImpl (#1825) * implement bit_not as ScalarUDFImpl * Revert expr.proto --------- Co-authored-by: Kazantsev Maksim <mn.kazantsev@gmail.com>

2025-05-27

fix: [native_scans] Support `CASE_SENSITIVE` when reading Parquet (#1782)

2025-05-23

fix: default values for native_datafusion scan (#1756)

2025-05-09

perf: Add performance tracing capability (#1706)

2025-05-02

chore: Remove fast encoding option (#1703)

2025-03-25

Chore: simplify array related functions impl (#1490) ## Which issue does this PR close? Related to issue: https://github.com/apache/datafusion-comet/issues/1459 ## Rationale for this change Defined under Issue: https://github.com/apache/datafusion-comet/issues/1459 ## What changes are included in this PR? In functions related to arrays, scalarExprToProtoWithReturnType or scalarExprToProto is used instead of creating a separate proto for each function. ## How are these changes tested? Regression with available unit tests

2025-02-28

feat: Support IntegralDivide function (#1428) ## Which issue does this PR close? Closes #1422. ## Rationale for this change Support IntegralDivide function ## What changes are included in this PR? Since datafusion div operator conforms to the logic of intergal div, we only need to convert `IntegralDivide(...)` to `Cast(Divide(...), LongType)` and then convert it to native. ## How are these changes tested? added unit test

2025-02-13

perf: Use DataFusion FilterExec for experimental native scans (#1395) * Add boolean field to Filter's proto, set based on Comet native scan implementation. Planner uses that field to construct the correct FilterExec implementation. CometFilterExec does a deep copy of the batch due to logic in Comet Scan, while DF FilterExec can do a shallow copy because native Scans do not reuse batch buffers. * Refactor to reduce duplicate code. * Fix native test. * Address nit.

2025-02-08

perf: improve performance of update metrics (#1329)

2025-01-28

Support arrays_overlap function (#1312)

2025-01-23

Feat: Support array_join function (#1290)

2025-01-21

Feat: Support array_intersect function (#1271) * Feat: Support array_intersect * Address review comment

chore: merge comet-parquet-exec branch into main (#1318)

2025-01-16

chore: Comet parquet exec merge from main(20250114) (#1293) * feat: support array_append (#1072) * feat: support array_append * formatted code * rewrite array_append plan to match spark behaviour and fixed bug in QueryPlan serde * remove unwrap * Fix for Spark 3.3 * refactor array_append binary expression serde code * Disabled array_append test for spark 4.0+ * chore: Simplify CometShuffleMemoryAllocator to use Spark unified memory allocator (#1063) * docs: Update benchmarking.md (#1085) * feat: Require offHeap memory to be enabled (always use unified memory) (#1062) * Require offHeap memory * remove unused import * use off heap memory in stability tests * reorder imports * test: Restore one test in CometExecSuite by adding COMET_SHUFFLE_MODE config (#1087) * Add changelog for 0.4.0 (#1089) * chore: Prepare for 0.5.0 development (#1090) * Update version number for build * update docs * build: Skip installation of spark-integration and fuzz testing modules (#1091) * Add hint for finding the GPG key to use when publishing to maven (#1093) * docs: Update documentation for 0.4.0 release (#1096) * update TPC-H results * update Maven links * update benchmarking guide and add TPC-DS results * include q72 * fix: Unsigned type related bugs (#1095) ## Which issue does this PR close? Closes https://github.com/apache/datafusion-comet/issues/1067 ## Rationale for this change Bug fix. A few expressions were failing some unsigned type related tests ## What changes are included in this PR? - For `u8`/`u16`, switched to use `generate_cast_to_signed!` in order to copy full i16/i32 width instead of padding zeros in the higher bits - `u64` becomes `Decimal(20, 0)` but there was a bug in `round()` (`>` vs `>=`) ## How are these changes tested? Put back tests for unsigned types * chore: Include first ScanExec batch in metrics (#1105) * include first batch in ScanExec metrics * record row count metric * fix regression * chore: Improve CometScan metrics (#1100) * Add native metrics for plan creation * make messages consistent * Include get_next_batch cost in metrics * formatting * fix double count of rows * chore: Add custom metric for native shuffle fetching batches from JVM (#1108) * feat: support array_insert (#1073) * Part of the implementation of array_insert * Missing methods * Working version * Reformat code * Fix code-style * Add comments about spark's implementation. * Implement negative indices + fix tests for spark < 3.4 * Fix code-style * Fix scalastyle * Fix tests for spark < 3.4 * Fixes & tests - added test for the negative index - added test for the legacy spark mode * Use assume(isSpark34Plus) in tests * Test else-branch & improve coverage * Update native/spark-expr/src/list.rs Co-authored-by: Andy Grove <agrove@apache.org> * Fix fallback test In one case there is a zero in index and test fails due to spark error * Adjust the behaviour for the NULL case to Spark * Move the logic of type checking to the method * Fix code-style --------- Co-authored-by: Andy Grove <agrove@apache.org> * feat: enable decimal to decimal cast of different precision and scale (#1086) * enable decimal to decimal cast of different precision and scale * add more test cases for negative scale and higher precision * add check for compatibility for decimal to decimal * fix code style * Update spark/src/main/scala/org/apache/comet/expressions/CometCast.scala Co-authored-by: Andy Grove <agrove@apache.org> * fix the nit in comment --------- Co-authored-by: himadripal <hpal@apple.com> Co-authored-by: Andy Grove <agrove@apache.org> * docs: fix readme FGPA/FPGA typo (#1117) * fix: Use RDD partition index (#1112) * fix: Use RDD partition index * fix * fix * fix * fix: Various metrics bug fixes and improvements (#1111) * fix: Don't create CometScanExec for subclasses of ParquetFileFormat (#1129) * Use exact class comparison for parquet scan * Add test * Add comment * fix: Fix metrics regressions (#1132) * fix metrics issues * clippy * update tests * docs: Add more technical detail and new diagram to Comet plugin overview (#1119) * Add more technical detail and new diagram to Comet plugin overview * update diagram * add info on Arrow IPC * update diagram * update diagram * update docs * address feedback * Stop passing Java config map into native createPlan (#1101) * feat: Improve ScanExec native metrics (#1133) * save * remove shuffle jvm metric and update tuning guide * docs * add source for all ScanExecs * address feedback * address feedback * chore: Remove unused StringView struct (#1143) * Remove unused StringView struct * remove more dead code * docs: Add some documentation explaining how shuffle works (#1148) * add some notes on shuffle * reads * improve docs * test: enable more Spark 4.0 tests (#1145) ## Which issue does this PR close? Part of https://github.com/apache/datafusion-comet/issues/372 and https://github.com/apache/datafusion-comet/issues/551 ## Rationale for this change To be ready for Spark 4.0 ## What changes are included in this PR? This PR enables more Spark 4.0 tests that were fixed by recent changes ## How are these changes tested? tests enabled * chore: Refactor cast to use SparkCastOptions param (#1146) * Refactor cast to use SparkCastOptions param * update tests * update benches * update benches * update benches * Enable more scenarios in CometExecBenchmark. (#1151) * chore: Move more expressions from core crate to spark-expr crate (#1152) * move aggregate expressions to spark-expr crate * move more expressions * move benchmark * normalize_nan * bitwise not * comet scalar funcs * update bench imports * remove dead code (#1155) * fix: Spark 4.0-preview1 SPARK-47120 (#1156) ## Which issue does this PR close? Part of https://github.com/apache/datafusion-comet/issues/372 and https://github.com/apache/datafusion-comet/issues/551 ## Rationale for this change To be ready for Spark 4.0 ## What changes are included in this PR? This PR fixes the new test SPARK-47120 added in Spark 4.0 ## How are these changes tested? tests enabled * chore: Move string kernels and expressions to spark-expr crate (#1164) * Move string kernels and expressions to spark-expr crate * remove unused hash kernel * remove unused dependencies * chore: Move remaining expressions to spark-expr crate + some minor refactoring (#1165) * move CheckOverflow to spark-expr crate * move NegativeExpr to spark-expr crate * move UnboundColumn to spark-expr crate * move ExpandExec from execution::datafusion::operators to execution::operators * refactoring to remove datafusion subpackage * update imports in benches * fix * fix * chore: Add ignored tests for reading complex types from Parquet (#1167) * Add ignored tests for reading structs from Parquet * add basic map test * add tests for Map and Array * feat: Add Spark-compatible implementation of SchemaAdapterFactory (#1169) * Add Spark-compatible SchemaAdapterFactory implementation * remove prototype code * fix * refactor * implement more cast logic * implement more cast logic * add basic test * improve test * cleanup * fmt * add support for casting unsigned int to signed int * clippy * address feedback * fix test * fix: Document enabling comet explain plan usage in Spark (4.0) (#1176) * test: enabling Spark tests with offHeap requirement (#1177) ## Which issue does this PR close? ## Rationale for this change After https://github.com/apache/datafusion-comet/pull/1062 We have not running Spark tests for native execution ## What changes are included in this PR? Removed the off heap requirement for testing ## How are these changes tested? Bringing back Spark tests for native execution * feat: Improve shuffle metrics (second attempt) (#1175) * improve shuffle metrics * docs * more metrics * refactor * address feedback * fix: stddev_pop should not directly return 0.0 when count is 1.0 (#1184) * add test * fix * fix * fix * feat: Make native shuffle compression configurable and respect `spark.shuffle.compress` (#1185) * Make shuffle compression codec and level configurable * remove lz4 references * docs * update comment * clippy * fix benches * clippy * clippy * disable test for miri * remove lz4 reference from proto * minor: move shuffle classes from common to spark (#1193) * minor: refactor decodeBatches to make private in broadcast exchange (#1195) * minor: refactor prepare_output so that it does not require an ExecutionContext (#1194) * fix: fix missing explanation for then branch in case when (#1200) * minor: remove unused source files (#1202) * chore: Upgrade to DataFusion 44.0.0-rc2 (#1154) * move aggregate expressions to spark-expr crate * move more expressions * move benchmark * normalize_nan * bitwise not * comet scalar funcs * update bench imports * save * save * save * remove unused imports * clippy * implement more hashers * implement Hash and PartialEq * implement Hash and PartialEq * implement Hash and PartialEq * benches * fix ScalarUDFImpl.return_type failure * exclude test from miri * ignore correct test * ignore another test * remove miri checks * use return_type_from_exprs * Revert "use return_type_from_exprs" This reverts commit febc1f1ec1301f9b359fc23ad6a117224fce35b7. * use DF main branch * hacky workaround for regression in ScalarUDFImpl.return_type * fix repo url * pin to revision * bump to latest rev * bump to latest DF rev * bump DF to rev 9f530dd * add Cargo.lock * bump DF version * no default features * Revert "remove miri checks" This reverts commit 4638fe3aa5501966cd5d8b53acf26c698b10b3c9. * Update pin to DataFusion e99e02b9b9093ceb0c13a2dd32a2a89beba47930 * update pin * Update Cargo.toml Bump to 44.0.0-rc2 * update cargo lock * revert miri change --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> * feat: add support for array_contains expression (#1163) * feat: add support for array_contains expression * test: add unit test for array_contains function * Removes unnecessary case expression for handling null values * chore: Move more expressions from core crate to spark-expr crate (#1152) * move aggregate expressions to spark-expr crate * move more expressions * move benchmark * normalize_nan * bitwise not * comet scalar funcs * update bench imports * remove dead code (#1155) * fix: Spark 4.0-preview1 SPARK-47120 (#1156) ## Which issue does this PR close? Part of https://github.com/apache/datafusion-comet/issues/372 and https://github.com/apache/datafusion-comet/issues/551 ## Rationale for this change To be ready for Spark 4.0 ## What changes are included in this PR? This PR fixes the new test SPARK-47120 added in Spark 4.0 ## How are these changes tested? tests enabled * chore: Move string kernels and expressions to spark-expr crate (#1164) * Move string kernels and expressions to spark-expr crate * remove unused hash kernel * remove unused dependencies * chore: Move remaining expressions to spark-expr crate + some minor refactoring (#1165) * move CheckOverflow to spark-expr crate * move NegativeExpr to spark-expr crate * move UnboundColumn to spark-expr crate * move ExpandExec from execution::datafusion::operators to execution::operators * refactoring to remove datafusion subpackage * update imports in benches * fix * fix * chore: Add ignored tests for reading complex types from Parquet (#1167) * Add ignored tests for reading structs from Parquet * add basic map test * add tests for Map and Array * feat: Add Spark-compatible implementation of SchemaAdapterFactory (#1169) * Add Spark-compatible SchemaAdapterFactory implementation * remove prototype code * fix * refactor * implement more cast logic * implement more cast logic * add basic test * improve test * cleanup * fmt * add support for casting unsigned int to signed int * clippy * address feedback * fix test * fix: Document enabling comet explain plan usage in Spark (4.0) (#1176) * test: enabling Spark tests with offHeap requirement (#1177) ## Which issue does this PR close? ## Rationale for this change After https://github.com/apache/datafusion-comet/pull/1062 We have not running Spark tests for native execution ## What changes are included in this PR? Removed the off heap requirement for testing ## How are these changes tested? Bringing back Spark tests for native execution * feat: Improve shuffle metrics (second attempt) (#1175) * improve shuffle metrics * docs * more metrics * refactor * address feedback * fix: stddev_pop should not directly return 0.0 when count is 1.0 (#1184) * add test * fix * fix * fix * feat: Make native shuffle compression configurable and respect `spark.shuffle.compress` (#1185) * Make shuffle compression codec and level configurable * remove lz4 references * docs * update comment * clippy * fix benches * clippy * clippy * disable test for miri * remove lz4 reference from proto * minor: move shuffle classes from common to spark (#1193) * minor: refactor decodeBatches to make private in broadcast exchange (#1195) * minor: refactor prepare_output so that it does not require an ExecutionContext (#1194) * fix: fix missing explanation for then branch in case when (#1200) * minor: remove unused source files (#1202) * chore: Upgrade to DataFusion 44.0.0-rc2 (#1154) * move aggregate expressions to spark-expr crate * move more expressions * move benchmark * normalize_nan * bitwise not * comet scalar funcs * update bench imports * save * save * save * remove unused imports * clippy * implement more hashers * implement Hash and PartialEq * implement Hash and PartialEq * implement Hash and PartialEq * benches * fix ScalarUDFImpl.return_type failure * exclude test from miri * ignore correct test * ignore another test * remove miri checks * use return_type_from_exprs * Revert "use return_type_from_exprs" This reverts commit febc1f1ec1301f9b359fc23ad6a117224fce35b7. * use DF main branch * hacky workaround for regression in ScalarUDFImpl.return_type * fix repo url * pin to revision * bump to latest rev * bump to latest DF rev * bump DF to rev 9f530dd * add Cargo.lock * bump DF version * no default features * Revert "remove miri checks" This reverts commit 4638fe3aa5501966cd5d8b53acf26c698b10b3c9. * Update pin to DataFusion e99e02b9b9093ceb0c13a2dd32a2a89beba47930 * update pin * Update Cargo.toml Bump to 44.0.0-rc2 * update cargo lock * revert miri change --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> * update UT Signed-off-by: Dharan Aditya <dharan.aditya@gmail.com> * fix typo in UT Signed-off-by: Dharan Aditya <dharan.aditya@gmail.com> --------- Signed-off-by: Dharan Aditya <dharan.aditya@gmail.com> Co-authored-by: Andy Grove <agrove@apache.org> Co-authored-by: KAZUYUKI TANIMURA <ktanimura@apple.com> Co-authored-by: Parth Chandra <parthc@apache.org> Co-authored-by: Liang-Chi Hsieh <viirya@gmail.com> Co-authored-by: Raz Luvaton <16746759+rluvaton@users.noreply.github.com> Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> * feat: Add a `spark.comet.exec.memoryPool` configuration for experimenting with various datafusion memory pool setups. (#1021) * feat: Reenable tests for filtered SMJ anti join (#1211) * feat: reenable filtered SMJ Anti join tests * feat: reenable filtered SMJ Anti join tests * feat: reenable filtered SMJ Anti join tests * feat: reenable filtered SMJ Anti join tests * Add CoalesceBatchesExec around SMJ with join filter * adding `CoalesceBatches` * adding `CoalesceBatches` * adding `CoalesceBatches` * feat: reenable filtered SMJ Anti join tests * feat: reenable filtered SMJ Anti join tests --------- Co-authored-by: Andy Grove <agrove@apache.org> * chore: Add safety check to CometBuffer (#1050) * chore: Add safety check to CometBuffer * Add CometColumnarToRowExec * fix * fix * more * Update plan stability results * fix * fix * fix * Revert "fix" This reverts commit 9bad173c7751f105bf3ded2ebc2fed0737d1b909. * Revert "Revert "fix"" This reverts commit d527ad1a365d3aff64200ceba6d11cf376f3919f. * fix BucketedReadWithoutHiveSupportSuite * fix SparkPlanSuite * remove unreachable code (#1213) * test: Enable Comet by default except some tests in SparkSessionExtensionSuite (#1201) ## Which issue does this PR close? Part of https://github.com/apache/datafusion-comet/issues/1197 ## Rationale for this change Since `loadCometExtension` in the diffs were not using `isCometEnabled`, `SparkSessionExtensionSuite` was not using Comet. Once enabled, some test failures discovered ## What changes are included in this PR? `loadCometExtension` now uses `isCometEnabled` that enables Comet by default Temporary ignore the failing tests in SparkSessionExtensionSuite ## How are these changes tested? existing tests * extract struct expressions to folders based on spark grouping (#1216) * chore: extract static invoke expressions to folders based on spark grouping (#1217) * extract static invoke expressions to folders based on spark grouping * Update native/spark-expr/src/static_invoke/mod.rs Co-authored-by: Andy Grove <agrove@apache.org> --------- Co-authored-by: Andy Grove <agrove@apache.org> * chore: Follow-on PR to fully enable onheap memory usage (#1210) * Make datafusion's native memory pool configurable * save * fix * Update memory calculation and add draft documentation * ready for review * ready for review * address feedback * Update docs/source/user-guide/tuning.md Co-authored-by: Liang-Chi Hsieh <viirya@gmail.com> * Update docs/source/user-guide/tuning.md Co-authored-by: Kristin Cowalcijk <bo@wherobots.com> * Update docs/source/user-guide/tuning.md Co-authored-by: Liang-Chi Hsieh <viirya@gmail.com> * Update docs/source/user-guide/tuning.md Co-authored-by: Liang-Chi Hsieh <viirya@gmail.com> * remove unused config --------- Co-authored-by: Kristin Cowalcijk <bo@wherobots.com> Co-authored-by: Liang-Chi Hsieh <viirya@gmail.com> * feat: Move shuffle block decompression and decoding to native code and add LZ4 & Snappy support (#1192) * Implement native decoding and decompression * revert some variable renaming for smaller diff * fix oom issues? * make NativeBatchDecoderIterator more consistent with ArrowReaderIterator * fix oom and prep for review * format * Add LZ4 support * clippy, new benchmark * rename metrics, clean up lz4 code * update test * Add support for snappy * format * change default back to lz4 * make metrics more accurate * format * clippy * use faster unsafe version of lz4_flex * Make compression codec configurable for columnar shuffle * clippy * fix bench * fmt * address feedback * address feedback * address feedback * minor code simplification * cargo fmt * overflow check * rename compression level config * address feedback * address feedback * rename constant * chore: extract agg_funcs expressions to folders based on spark grouping (#1224) * extract agg_funcs expressions to folders based on spark grouping * fix rebase * extract datetime_funcs expressions to folders based on spark grouping (#1222) Co-authored-by: Andy Grove <agrove@apache.org> * chore: use datafusion from crates.io (#1232) * chore: extract strings file to `strings_func` like in spark grouping (#1215) * chore: extract predicate_functions expressions to folders based on spark grouping (#1218) * extract predicate_functions expressions to folders based on spark grouping * code review changes --------- Co-authored-by: Andy Grove <agrove@apache.org> * build(deps): bump protobuf version to 3.21.12 (#1234) * extract json_funcs expressions to folders based on spark grouping (#1220) Co-authored-by: Andy Grove <agrove@apache.org> * test: Enable shuffle by default in Spark tests (#1240) ## Which issue does this PR close? ## Rationale for this change Because `isCometShuffleEnabled` is false by default, some tests were not reached ## What changes are included in this PR? Removed `isCometShuffleEnabled` and updated spark test diff ## How are these changes tested? existing test * chore: extract hash_funcs expressions to folders based on spark grouping (#1221) * extract hash_funcs expressions to folders based on spark grouping * extract hash_funcs expressions to folders based on spark grouping --------- Co-authored-by: Andy Grove <agrove@apache.org> * fix: Fall back to Spark for unsupported partition or sort expressions in window aggregates (#1253) * perf: Improve query planning to more reliably fall back to columnar shuffle when native shuffle is not supported (#1209) * fix regression (#1259) * feat: add support for array_remove expression (#1179) * wip: array remove * added comet expression test * updated test cases * fixed array_remove function for null values * removed commented code * remove unnecessary code * updated the test for 'array_remove' * added test for array_remove in case the input array is null * wip: case array is empty * removed test case for empty array * fix: Fall back to Spark for distinct aggregates (#1262) * fall back to Spark for distinct aggregates * update expected plans for 3.4 * update expected plans for 3.5 * force build * add comment * feat: Implement custom RecordBatch serde for shuffle for improved performance (#1190) * Implement faster encoder for shuffle blocks * make code more concise * enable fast encoding for columnar shuffle * update benches * test all int types * test float * remaining types * add Snappy and Zstd(6) back to benchmark * fix regression * Update native/core/src/execution/shuffle/codec.rs Co-authored-by: Liang-Chi Hsieh <viirya@gmail.com> * address feedback * support nullable flag --------- Co-authored-by: Liang-Chi Hsieh <viirya@gmail.com> * docs: Update TPC-H benchmark results (#1257) * fix: disable initCap by default (#1276) * fix: disable initCap by default * Update spark/src/main/scala/org/apache/comet/serde/QueryPlanSerde.scala Co-authored-by: Andy Grove <agrove@apache.org> * address review comments --------- Co-authored-by: Andy Grove <agrove@apache.org> * chore: Add changelog for 0.5.0 (#1278) * Add changelog * revert accidental change * move 2 items to performance section * update TPC-DS results for 0.5.0 (#1277) * fix: cast timestamp to decimal is unsupported (#1281) * fix: cast timestamp to decimal is unsupported * fix style * revert test name and mark as ignore * add comment * Fix build after merge * Fix tests after merge * Fix plans after merge * fix partition id in execute plan after merge (from Andy Grove) --------- Signed-off-by: Dharan Aditya <dharan.aditya@gmail.com> Co-authored-by: NoeB <noe.brehm@bluewin.ch> Co-authored-by: Liang-Chi Hsieh <viirya@gmail.com> Co-authored-by: Raz Luvaton <raz.luvaton@flarion.io> Co-authored-by: Andy Grove <agrove@apache.org> Co-authored-by: KAZUYUKI TANIMURA <ktanimura@apple.com> Co-authored-by: Sem <ssinchenko@apache.org> Co-authored-by: Himadri Pal <mehimu@gmail.com> Co-authored-by: himadripal <hpal@apple.com> Co-authored-by: gstvg <28798827+gstvg@users.noreply.github.com> Co-authored-by: Adam Binford <adamq43@gmail.com> Co-authored-by: Matt Butrovich <mbutrovich@users.noreply.github.com> Co-authored-by: Raz Luvaton <16746759+rluvaton@users.noreply.github.com> Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> Co-authored-by: Dharan Aditya <dharan.aditya@gmail.com> Co-authored-by: Kristin Cowalcijk <bo@wherobots.com> Co-authored-by: Oleks V <comphead@users.noreply.github.com> Co-authored-by: Zhen Wang <643348094@qq.com> Co-authored-by: Jagdish Parihar <jatin6972@gmail.com>

2025-01-13

feat: Implement custom RecordBatch serde for shuffle for improved performance (#1190) * Implement faster encoder for shuffle blocks * make code more concise * enable fast encoding for columnar shuffle * update benches * test all int types * test float * remaining types * add Snappy and Zstd(6) back to benchmark * fix regression * Update native/core/src/execution/shuffle/codec.rs Co-authored-by: Liang-Chi Hsieh <viirya@gmail.com> * address feedback * support nullable flag --------- Co-authored-by: Liang-Chi Hsieh <viirya@gmail.com>

chore: [comet-parquet-exec] Unit test fixes, default scan impl to native_comet (#1265) * fix: fix tests failing in native_recordbatch but not in native_full * fix: use session timestamp in native scans * Revert "fix: use session timestamp in native scans" This reverts commit e601deb472037338a36300992434a987bdb026e8. * Revert Change to native record batch timezone * Change stability plans to match original scan. * fix after rebase * Update plans; generate distinct plans for full native scan * generate plans for native_recordbatch * In struct tests, check Comet operator only for scan types that support complex types * Revert "Revert Change to native record batch timezone" This reverts commit 4a147f3766b25dc9245448e529e16a781086f3c6. * Reapply "fix: use session timestamp in native scans" This reverts commit 370f9015d3b0d134737a1864fab8638d6740bb2f. * Fix previous commit * Rename configs and default scan impl to 'native_comet' * add missing change * fix build * update plans for spark 3.5 * Add new plans for spark 3.5 * Update plans for Spark 4.0 * Plans updated from Spark 4

2025-01-12

feat: add support for array_remove expression (#1179) * wip: array remove * added comet expression test * updated test cases * fixed array_remove function for null values * removed commented code * remove unnecessary code * updated the test for 'array_remove' * added test for array_remove in case the input array is null * wip: case array is empty * removed test case for empty array

2025-01-07

feat: Move shuffle block decompression and decoding to native code and add LZ4 & Snappy support (#1192) * Implement native decoding and decompression * revert some variable renaming for smaller diff * fix oom issues? * make NativeBatchDecoderIterator more consistent with ArrowReaderIterator * fix oom and prep for review * format * Add LZ4 support * clippy, new benchmark * rename metrics, clean up lz4 code * update test * Add support for snappy * format * change default back to lz4 * make metrics more accurate * format * clippy * use faster unsafe version of lz4_flex * Make compression codec configurable for columnar shuffle * clippy * fix bench * fmt * address feedback * address feedback * address feedback * minor code simplification * cargo fmt * overflow check * rename compression level config * address feedback * address feedback * rename constant

2025-01-02

feat: add support for array_contains expression (#1163) * feat: add support for array_contains expression * test: add unit test for array_contains function * Removes unnecessary case expression for handling null values * chore: Move more expressions from core crate to spark-expr crate (#1152) * move aggregate expressions to spark-expr crate * move more expressions * move benchmark * normalize_nan * bitwise not * comet scalar funcs * update bench imports * remove dead code (#1155) * fix: Spark 4.0-preview1 SPARK-47120 (#1156) ## Which issue does this PR close? Part of https://github.com/apache/datafusion-comet/issues/372 and https://github.com/apache/datafusion-comet/issues/551 ## Rationale for this change To be ready for Spark 4.0 ## What changes are included in this PR? This PR fixes the new test SPARK-47120 added in Spark 4.0 ## How are these changes tested? tests enabled * chore: Move string kernels and expressions to spark-expr crate (#1164) * Move string kernels and expressions to spark-expr crate * remove unused hash kernel * remove unused dependencies * chore: Move remaining expressions to spark-expr crate + some minor refactoring (#1165) * move CheckOverflow to spark-expr crate * move NegativeExpr to spark-expr crate * move UnboundColumn to spark-expr crate * move ExpandExec from execution::datafusion::operators to execution::operators * refactoring to remove datafusion subpackage * update imports in benches * fix * fix * chore: Add ignored tests for reading complex types from Parquet (#1167) * Add ignored tests for reading structs from Parquet * add basic map test * add tests for Map and Array * feat: Add Spark-compatible implementation of SchemaAdapterFactory (#1169) * Add Spark-compatible SchemaAdapterFactory implementation * remove prototype code * fix * refactor * implement more cast logic * implement more cast logic * add basic test * improve test * cleanup * fmt * add support for casting unsigned int to signed int * clippy * address feedback * fix test * fix: Document enabling comet explain plan usage in Spark (4.0) (#1176) * test: enabling Spark tests with offHeap requirement (#1177) ## Which issue does this PR close? ## Rationale for this change After https://github.com/apache/datafusion-comet/pull/1062 We have not running Spark tests for native execution ## What changes are included in this PR? Removed the off heap requirement for testing ## How are these changes tested? Bringing back Spark tests for native execution * feat: Improve shuffle metrics (second attempt) (#1175) * improve shuffle metrics * docs * more metrics * refactor * address feedback * fix: stddev_pop should not directly return 0.0 when count is 1.0 (#1184) * add test * fix * fix * fix * feat: Make native shuffle compression configurable and respect `spark.shuffle.compress` (#1185) * Make shuffle compression codec and level configurable * remove lz4 references * docs * update comment * clippy * fix benches * clippy * clippy * disable test for miri * remove lz4 reference from proto * minor: move shuffle classes from common to spark (#1193) * minor: refactor decodeBatches to make private in broadcast exchange (#1195) * minor: refactor prepare_output so that it does not require an ExecutionContext (#1194) * fix: fix missing explanation for then branch in case when (#1200) * minor: remove unused source files (#1202) * chore: Upgrade to DataFusion 44.0.0-rc2 (#1154) * move aggregate expressions to spark-expr crate * move more expressions * move benchmark * normalize_nan * bitwise not * comet scalar funcs * update bench imports * save * save * save * remove unused imports * clippy * implement more hashers * implement Hash and PartialEq * implement Hash and PartialEq * implement Hash and PartialEq * benches * fix ScalarUDFImpl.return_type failure * exclude test from miri * ignore correct test * ignore another test * remove miri checks * use return_type_from_exprs * Revert "use return_type_from_exprs" This reverts commit febc1f1ec1301f9b359fc23ad6a117224fce35b7. * use DF main branch * hacky workaround for regression in ScalarUDFImpl.return_type * fix repo url * pin to revision * bump to latest rev * bump to latest DF rev * bump DF to rev 9f530dd * add Cargo.lock * bump DF version * no default features * Revert "remove miri checks" This reverts commit 4638fe3aa5501966cd5d8b53acf26c698b10b3c9. * Update pin to DataFusion e99e02b9b9093ceb0c13a2dd32a2a89beba47930 * update pin * Update Cargo.toml Bump to 44.0.0-rc2 * update cargo lock * revert miri change --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> * update UT Signed-off-by: Dharan Aditya <dharan.aditya@gmail.com> * fix typo in UT Signed-off-by: Dharan Aditya <dharan.aditya@gmail.com> --------- Signed-off-by: Dharan Aditya <dharan.aditya@gmail.com> Co-authored-by: Andy Grove <agrove@apache.org> Co-authored-by: KAZUYUKI TANIMURA <ktanimura@apple.com> Co-authored-by: Parth Chandra <parthc@apache.org> Co-authored-by: Liang-Chi Hsieh <viirya@gmail.com> Co-authored-by: Raz Luvaton <16746759+rluvaton@users.noreply.github.com> Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>

2024-12-20

feat: Make native shuffle compression configurable and respect `spark.shuffle.compress` (#1185) * Make shuffle compression codec and level configurable * remove lz4 references * docs * update comment * clippy * fix benches * clippy * clippy * disable test for miri * remove lz4 reference from proto

[comet-parquet-exec] Merge upstream/main and resolve conflicts (#1183) * feat: support array_append (#1072) * feat: support array_append * formatted code * rewrite array_append plan to match spark behaviour and fixed bug in QueryPlan serde * remove unwrap * Fix for Spark 3.3 * refactor array_append binary expression serde code * Disabled array_append test for spark 4.0+ * chore: Simplify CometShuffleMemoryAllocator to use Spark unified memory allocator (#1063) * docs: Update benchmarking.md (#1085) * feat: Require offHeap memory to be enabled (always use unified memory) (#1062) * Require offHeap memory * remove unused import * use off heap memory in stability tests * reorder imports * test: Restore one test in CometExecSuite by adding COMET_SHUFFLE_MODE config (#1087) * Add changelog for 0.4.0 (#1089) * chore: Prepare for 0.5.0 development (#1090) * Update version number for build * update docs * build: Skip installation of spark-integration and fuzz testing modules (#1091) * Add hint for finding the GPG key to use when publishing to maven (#1093) * docs: Update documentation for 0.4.0 release (#1096) * update TPC-H results * update Maven links * update benchmarking guide and add TPC-DS results * include q72 * fix: Unsigned type related bugs (#1095) ## Which issue does this PR close? Closes https://github.com/apache/datafusion-comet/issues/1067 ## Rationale for this change Bug fix. A few expressions were failing some unsigned type related tests ## What changes are included in this PR? - For `u8`/`u16`, switched to use `generate_cast_to_signed!` in order to copy full i16/i32 width instead of padding zeros in the higher bits - `u64` becomes `Decimal(20, 0)` but there was a bug in `round()` (`>` vs `>=`) ## How are these changes tested? Put back tests for unsigned types * chore: Include first ScanExec batch in metrics (#1105) * include first batch in ScanExec metrics * record row count metric * fix regression * chore: Improve CometScan metrics (#1100) * Add native metrics for plan creation * make messages consistent * Include get_next_batch cost in metrics * formatting * fix double count of rows * chore: Add custom metric for native shuffle fetching batches from JVM (#1108) * feat: support array_insert (#1073) * Part of the implementation of array_insert * Missing methods * Working version * Reformat code * Fix code-style * Add comments about spark's implementation. * Implement negative indices + fix tests for spark < 3.4 * Fix code-style * Fix scalastyle * Fix tests for spark < 3.4 * Fixes & tests - added test for the negative index - added test for the legacy spark mode * Use assume(isSpark34Plus) in tests * Test else-branch & improve coverage * Update native/spark-expr/src/list.rs Co-authored-by: Andy Grove <agrove@apache.org> * Fix fallback test In one case there is a zero in index and test fails due to spark error * Adjust the behaviour for the NULL case to Spark * Move the logic of type checking to the method * Fix code-style --------- Co-authored-by: Andy Grove <agrove@apache.org> * feat: enable decimal to decimal cast of different precision and scale (#1086) * enable decimal to decimal cast of different precision and scale * add more test cases for negative scale and higher precision * add check for compatibility for decimal to decimal * fix code style * Update spark/src/main/scala/org/apache/comet/expressions/CometCast.scala Co-authored-by: Andy Grove <agrove@apache.org> * fix the nit in comment --------- Co-authored-by: himadripal <hpal@apple.com> Co-authored-by: Andy Grove <agrove@apache.org> * docs: fix readme FGPA/FPGA typo (#1117) * fix: Use RDD partition index (#1112) * fix: Use RDD partition index * fix * fix * fix * fix: Various metrics bug fixes and improvements (#1111) * fix: Don't create CometScanExec for subclasses of ParquetFileFormat (#1129) * Use exact class comparison for parquet scan * Add test * Add comment * fix: Fix metrics regressions (#1132) * fix metrics issues * clippy * update tests * docs: Add more technical detail and new diagram to Comet plugin overview (#1119) * Add more technical detail and new diagram to Comet plugin overview * update diagram * add info on Arrow IPC * update diagram * update diagram * update docs * address feedback * Stop passing Java config map into native createPlan (#1101) * feat: Improve ScanExec native metrics (#1133) * save * remove shuffle jvm metric and update tuning guide * docs * add source for all ScanExecs * address feedback * address feedback * chore: Remove unused StringView struct (#1143) * Remove unused StringView struct * remove more dead code * docs: Add some documentation explaining how shuffle works (#1148) * add some notes on shuffle * reads * improve docs * test: enable more Spark 4.0 tests (#1145) ## Which issue does this PR close? Part of https://github.com/apache/datafusion-comet/issues/372 and https://github.com/apache/datafusion-comet/issues/551 ## Rationale for this change To be ready for Spark 4.0 ## What changes are included in this PR? This PR enables more Spark 4.0 tests that were fixed by recent changes ## How are these changes tested? tests enabled * chore: Refactor cast to use SparkCastOptions param (#1146) * Refactor cast to use SparkCastOptions param * update tests * update benches * update benches * update benches * Enable more scenarios in CometExecBenchmark. (#1151) * chore: Move more expressions from core crate to spark-expr crate (#1152) * move aggregate expressions to spark-expr crate * move more expressions * move benchmark * normalize_nan * bitwise not * comet scalar funcs * update bench imports * remove dead code (#1155) * fix: Spark 4.0-preview1 SPARK-47120 (#1156) ## Which issue does this PR close? Part of https://github.com/apache/datafusion-comet/issues/372 and https://github.com/apache/datafusion-comet/issues/551 ## Rationale for this change To be ready for Spark 4.0 ## What changes are included in this PR? This PR fixes the new test SPARK-47120 added in Spark 4.0 ## How are these changes tested? tests enabled * chore: Move string kernels and expressions to spark-expr crate (#1164) * Move string kernels and expressions to spark-expr crate * remove unused hash kernel * remove unused dependencies * chore: Move remaining expressions to spark-expr crate + some minor refactoring (#1165) * move CheckOverflow to spark-expr crate * move NegativeExpr to spark-expr crate * move UnboundColumn to spark-expr crate * move ExpandExec from execution::datafusion::operators to execution::operators * refactoring to remove datafusion subpackage * update imports in benches * fix * fix * chore: Add ignored tests for reading complex types from Parquet (#1167) * Add ignored tests for reading structs from Parquet * add basic map test * add tests for Map and Array * feat: Add Spark-compatible implementation of SchemaAdapterFactory (#1169) * Add Spark-compatible SchemaAdapterFactory implementation * remove prototype code * fix * refactor * implement more cast logic * implement more cast logic * add basic test * improve test * cleanup * fmt * add support for casting unsigned int to signed int * clippy * address feedback * fix test * fix: Document enabling comet explain plan usage in Spark (4.0) (#1176) * test: enabling Spark tests with offHeap requirement (#1177) ## Which issue does this PR close? ## Rationale for this change After https://github.com/apache/datafusion-comet/pull/1062 We have not running Spark tests for native execution ## What changes are included in this PR? Removed the off heap requirement for testing ## How are these changes tested? Bringing back Spark tests for native execution * feat: Improve shuffle metrics (second attempt) (#1175) * improve shuffle metrics * docs * more metrics * refactor * address feedback * Fix redundancy in Cargo.lock. * Format, more post-merge cleanup. * Compiles * Compiles * Remove empty file. * Attempt to fix JNI issue and native test build issues. * Test Fix * Update planner.rs Remove println from test. --------- Co-authored-by: NoeB <noe.brehm@bluewin.ch> Co-authored-by: Liang-Chi Hsieh <viirya@gmail.com> Co-authored-by: Raz Luvaton <raz.luvaton@flarion.io> Co-authored-by: Andy Grove <agrove@apache.org> Co-authored-by: Parth Chandra <parthc@apache.org> Co-authored-by: KAZUYUKI TANIMURA <ktanimura@apple.com> Co-authored-by: Sem <ssinchenko@apache.org> Co-authored-by: Himadri Pal <mehimu@gmail.com> Co-authored-by: himadripal <hpal@apple.com> Co-authored-by: gstvg <28798827+gstvg@users.noreply.github.com> Co-authored-by: Adam Binford <adamq43@gmail.com>

2024-12-05

[comet-parquet-exec] Simplify schema logic for CometNativeScan (#1142) * Serialize original data schema and required schema, generate projection vector on the Java side. * Sending over more schema info like column names and nullability. * Using the new stuff in the proto. About to take the old out. * Remove old logic. * remove errant print. * Serialize original data schema and required schema, generate projection vector on the Java side. * Sending over more schema info like column names and nullability. * Using the new stuff in the proto. About to take the old out. * Remove old logic. * remove errant print. * Remove commented print. format. * Remove commented print. format. * Fix projection_vector to include partition_schema cols correctly. * Rename variable.

2024-12-02

fix: Various metrics bug fixes and improvements (#1111)

2024-11-22

fix: Support partition values in feature branch comet-parquet-exec (#1106) * init * more * more * fix clippy * Use Spark and Arrow types for partition schema

feat: support array_insert (#1073) * Part of the implementation of array_insert * Missing methods * Working version * Reformat code * Fix code-style * Add comments about spark's implementation. * Implement negative indices + fix tests for spark < 3.4 * Fix code-style * Fix scalastyle * Fix tests for spark < 3.4 * Fixes & tests - added test for the negative index - added test for the legacy spark mode * Use assume(isSpark34Plus) in tests * Test else-branch & improve coverage * Update native/spark-expr/src/list.rs Co-authored-by: Andy Grove <agrove@apache.org> * Fix fallback test In one case there is a zero in index and test fails due to spark error * Adjust the behaviour for the NULL case to Spark * Move the logic of type checking to the method * Fix code-style --------- Co-authored-by: Andy Grove <agrove@apache.org>

2024-11-13

feat: support array_append (#1072) * feat: support array_append * formatted code * rewrite array_append plan to match spark behaviour and fixed bug in QueryPlan serde * remove unwrap * Fix for Spark 3.3 * refactor array_append binary expression serde code * Disabled array_append test for spark 4.0+

[comet-parquet-exec] Pass Spark's partitions to DF's ParquetExec (#1081) * I think serde works. Gonna try removing the old stuff. * Fixes after merging in upstream. * Remove previous file_config logic. Clippy. * Temporary assertion for testing. * Remove old path proto value. * Selectively generate projection vector.