Proto commits in kwai/blaze

These commits are when the Protocol Buffers files have changed: (only the last 100 relevant commits are shown)

2026-07-01

[AURON #2321] Support Iceberg column rename and drop-then-add in the native scan (#2322) # Which issue does this PR close? Closes #2321 # Rationale for this change The native Iceberg scan matches data-file columns by name, but Iceberg tracks them by field-id. After a column rename, old files read as all-NULL; after a drop-then-add of the same name, the new column reads the old column's data. # What changes are included in this PR? Resolve columns by Iceberg field-id instead of by name: - proto: add `field_id` to `Field`. - JVM (`AuronIcebergSourceUtil`, `IcebergScanSupport`, `NativeConverters`): extract top-level `name → field-id` from the scan's `expectedSchema()` and serialize it into the plan. - native (`auron-planner`, `scan/mod.rs`): stamp the id into Arrow field metadata (`PARQUET:field_id`); `fields_match` matches by id when present, else falls back to case-insensitive name matching (non-Iceberg scans unchanged). Nested-struct evolution and ORC rename/drop fall back to Spark, additive evolution stays native. # Are there any user-facing changes? Yes. Iceberg queries on renamed or drop-then-added columns now return correct results under the native scan. Unsupported cases fall back to Spark. No API change. # How was this patch tested? Added cases to `AuronIcebergIntegrationSuite`

[AURON #2193] Implement native support for inner residual join conditions on SMJ/SHJ (#2197) # Which issue does this PR close? Closes #2193 # Rationale for this change Auron currently only converts SMJ/SHJ joins when the join condition is empty. As a result, inner joins that contain both equi-join keys and a residual predicate fall back to Spark even though the equi-join part is already native-compatible. The native join plan only models equi-join keys today, so this change keeps the native join focused on the equi-join portion and evaluates the residual predicate with a native filter above the join output. # What changes are included in this PR? - Allow native conversion of SortMergeJoinExec and ShuffledHashJoinExec when an InnerLike join has a residual condition. - Keep native join conversion based on equi-join keys only. - Apply the residual predicate as a native filter on top of the native join output. - Continue to reject residual join conditions for non-inner join types. - Add query tests for: - native SMJ with an inner residual condition - native SHJ with an inner residual condition in force-SHJ mode - Add a small test helper for Auron configs that are read from SparkEnv/SparkContext. # Are there any user-facing changes? Yes. Inner joins with equi-join keys plus a residual predicate can now remain on the native SMJ/SHJ path instead of falling back entirely to Spark. # How was this patch tested? CI. --------- Signed-off-by: weimingdiit <weimingdiit@gmail.com> Co-authored-by: Shilun Fan <slfan1989@apache.org>

2026-06-10

[AURON #2289] support SQL aggregate FILTER (WHERE ...) clause in native execution (#2290) ## Which issue does this PR close? Closes #2289 ## Rationale for this change Currently, queries using the SQL standard FILTER (WHERE ...) clause on aggregate functions fall back to non-native Spark execution. This is because the Spark-side native converters explicitly assert filter.isEmpty, the protobuf schema has no field for filter predicates, and the Rust execution engine does not apply per-aggregate row filtering. This PR enables end-to-end native execution for filtered aggregates, improving performance for a commonly used SQL pattern. ## What changes are included in this PR? 1. Protobuf schema: Add filter field to PhysicalAggExprNode. 2. Spark converters (NativeConverters, ShimsImpl): Remove assert(filter.isEmpty) and convert the Spark AggregateExpression.filter expression to protobuf. 3. Rust planner: Parse the filter field from protobuf into AggExpr.filter. 4. Rust execution (AggContext::update_batch_slice_to_acc_table): Evaluate each aggregate's filter expression against the input batch; for aggregates with a filter, build filtered acc_idx / input_idx vectors and pass only matching rows to partial_update. 5. Tests: - Rust: test_agg_with_filter in agg_exec.rs - Scala: aggregate with filter clause in AuronQuerySuite ## Are there any user-facing changes? NO. ## How was this patch tested? - cargo test --workspace — all Rust tests pass - cargo test -p datafusion-ext-plans -- agg_exec — existing test_agg / fuzztest plus new test_agg_with_filter all pass - Added Scala integration test in AuronQuerySuite covering filtered aggregates with/without GROUP BY and multiple predicates

2026-05-20

[AURON #2183] Implement native support for ORC InsertIntoHiveTable writes (#2191) # Which issue does this PR close? Closes #2183 # Rationale for this change Auron already supports native Parquet InsertIntoHiveTable writes, but ORC Hive writes still fall back to Spark’s regular execution path. This leaves native write coverage incomplete for a common Hive storage format. This PR adds native support for ORC InsertIntoHiveTable writes so eligible Hive ORC write workloads can stay on the native path instead of falling back. # What changes are included in this PR? This PR: - adds native ORC sink support in the native engine - adds planner / proto support for ORC sink execution - adds Spark-side physical plan support for native ORC InsertIntoHiveTable - extends AuronConverters to convert supported Hive ORC write plans to the native path - adds ORC sink utilities for task output path generation and output completion - preserves dynamic partition write handling on the native ORC write path - adapts input batches to the expected ORC/Hive output schema before writing - records output row and byte metrics for native ORC writes - adds execution coverage in AuronExecSuite # Are there any user-facing changes? Yes. Hive table writes using ORC may now remain on the native execution path when they match the supported InsertIntoHiveTable write pattern, instead of falling back to Spark’s regular write execution. # How was this patch tested? CI. --------- Signed-off-by: weimingdiit <weimingdiit@gmail.com>

2026-05-11

[AURON #2240] Implement native function of make_date #2240 (#2241) # Which issue does this PR close? Closes #2240 # Rationale for this change Spark SQL provides the `make_date` function to construct a date from year, month, and day components. To improve query performance, this PR implements native support for `make_date` in Auron's execution engine, allowing this function to be executed natively instead of falling back to Spark's JVM-based implementation. # What changes are included in this PR? - Protocol Buffer definition - Planner mapping - Spark-to-Native converter - Test coverage # Are there any user-facing changes? No. # How was this patch tested? - Added unit test

2026-05-05

[AURON #2181] Implement native support for cume_dist window function (#2205) # Which issue does this PR close? Closes #2181 # Rationale for this change Auron does not currently support the `cume_dist()` window function in native execution, so queries using it fall back to Spark. `cume_dist()` must follow Spark semantics exactly: it returns the number of rows preceding or equal to the current row within the partition ordering, divided by the total number of rows in the partition, and rows in the same peer group must return the same value. Because this depends on the full partition size and peer group boundaries, it cannot be implemented correctly with the existing purely streaming window path. # What changes are included in this PR? This PR adds native support for `cume_dist()` window function end to end. The main changes are: - Add `CUME_DIST` to the window function protobuf and planner conversion path. - Extend `NativeWindowBase` to recognize Spark's `CumeDist` expression and serialize it into the native window plan. - Add a native `CumeDistProcessor` that computes `cume_dist()` with Spark-compatible semantics: - rows in the same peer group produce the same value - the result is `(peer_group_end_position) / (partition_size)` - single-row partitions return `1.0` - Extend native window execution with a full-partition processing path for window functions that require complete partition context. - Use that full-partition path for `cume_dist()` to ensure correctness. - Add regression tests on both the native execution side and the Spark SQL side. # Are there any user-facing changes? Yes. Queries using `cume_dist()` window function can now stay on the native execution path instead of falling back to Spark, as long as the rest of the plan is supported by Auron. No new user-facing configuration is introduced. # How was this patch tested? CI. Signed-off-by: weimingdiit <weimingdiit@gmail.com>

2026-04-27

[AURON #2182] Implement native support for percent_rank window function (#2204) # Which issue does this PR close? Closes #2182 # Rationale for this change Auron does not currently support `percent_rank()` in native window execution, so queries using this function fall back to Spark. This leaves a gap in native window function coverage. `percent_rank()` cannot be implemented with the existing streaming-style window processors alone, because its result depends on both the row rank and the total number of rows in the partition. To match Spark semantics, the native engine needs to evaluate it with full-partition context. # What changes are included in this PR? This PR adds native support for `percent_rank()` window function end to end. The main changes are: - Add `PERCENT_RANK` to the window function protobuf and planner conversion path so Spark plans can be serialized into native plans correctly. - Extend `NativeWindowBase` to recognize Spark's `PercentRank` expression and convert it to the native window function enum. - Introduce a native `PercentRankProcessor` that computes percent rank with Spark-compatible semantics: - rows in the same peer group share the same rank - the result is `(rank - 1) / (partition_size - 1)` - single-row partitions return `0.0` - Add a full-partition execution path in native window execution for functions that require complete partition context, and use that path for `percent_rank()`. - Add tests on both the native execution side and the Spark SQL side to verify correctness. # Are there any user-facing changes? Yes. Queries using `percent_rank()` window function can now stay on the native execution path instead of falling back to Spark, as long as the rest of the plan is supported by Auron. No user-facing configuration changes are introduced. # How was this patch tested? CI. Signed-off-by: weimingdiit <weimingdiit@gmail.com>

2026-04-23

[AURON #2180] Implement native support for nth_value window function (#2203) # Which issue does this PR close? Closes #2180 # Rationale for this change Auron did not previously support native execution for Spark nth_value window function, so queries using nth_value would fall back to Spark execution. Spark defines nth_value(input, offset) as returning the value at the offsetth row from the beginning of the current window frame, with offset starting from 1. When IGNORE NULLS is specified, null input rows should be skipped when counting toward the target position. To improve Spark compatibility, Auron needs native support for both the regular and IGNORE NULLS variants. The current native window executor only supports cumulative row-frame semantics, so this change scopes native conversion to the frame that is already supported correctly: ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW. # What changes are included in this PR? This PR adds end-to-end native support for nth_value window function across the Spark frontend, planner, and native engine. The changes include: - adding new protobuf/planner window function variants for NTH_VALUE and NTH_VALUE_IGNORE_NULLS - extending Spark-side window conversion in NativeWindowBase to recognize Spark NthValue - using reflection to access NthValue fields so the code remains compatible with Spark 3.0, where this expression class is not available - adding a native Rust processor to evaluate nth_value - implementing both standard counting semantics and IGNORE NULLS semantics - restricting native conversion to the supported cumulative row frame - adding Rust-side execution tests - adding Scala integration tests to verify result correctness and native operator conversion # Are there any user-facing changes? Yes. Queries using nth_value can now be executed natively when they use the supported frame: ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW. Both of the following forms are supported: - nth_value(input, offset) - nth_value(input, offset) IGNORE NULLS Queries using unsupported window frames will continue to fall back to Spark execution. # How was this patch tested? CI. Signed-off-by: weimingdiit <weimingdiit@gmail.com>

2026-04-18

[AURON #2176] Implement native support for lead window function (#2188) # Which issue does this PR close? Closes #2176 # Rationale for this change Auron’s native window support previously covered rank-like functions and a subset of aggregate window functions, but did not support offset-based window functions such as `lead(...)`. This PR extends native window coverage with a conservative first step: - support `lead(...)` - preserve Spark-compatible behavior for `input`, `offset`, and `default` - keep unsupported semantics out of the native path rather than approximating them incorrectly # What changes are included in this PR? This PR: - adds `Lead` handling in `NativeWindowBase` - extends the protobuf/planner window function enum with `LEAD` - adds native planner support to decode `LEAD` into the native window plan - introduces a native `LeadProcessor` in `datafusion-ext-plans` - evaluates `lead` using Spark-compatible offset/default/null behavior - adds a full-partition processing path for `lead` so that lookahead works correctly across input batches - adds Rust regression coverage for cross-batch `lead` - adds Scala regression tests for: - native `lead(...)` execution - Spark fallback for `lead(... ) IGNORE NULLS` The native implementation supports Spark semantics for: - `lead(input)` - default offset is `1` - default value is `null` - `lead(input, offset, default)` - returns the value of `input` at the `offset`th row after the current row in the same window partition - if the target row exists and `input` there is `null`, returns `null` - if the target row does not exist, returns `default` Supported scope in this PR: - standard `RESPECT NULLS` behavior Not supported natively in this PR: - `IGNORE NULLS` Unsupported `IGNORE NULLS` queries continue to fall back to Spark to preserve correctness. # Are there any user-facing changes? Yes. Queries using `lead(...)` can now remain on Auron’s native window execution path when they use supported semantics. Queries using unsupported `lead(... ) IGNORE NULLS` behavior will continue to fall back to Spark. # How was this patch tested? CI. --------- Signed-off-by: weimingdiit <weimingdiit@gmail.com>

2026-04-01

[AURON #2126] Add native support for acosh function (#2135) # Which issue does this PR close? Closes # https://github.com/apache/auron/issues/2126 # Rationale for this change Spark Acosh expressions were not wired into Auron’s standard builtin scalar function conversion path, so acosh(expr) could not be planned through the native backend. This change follows the existing ScalarFunction flow used by other builtin math functions such as acos, asin, and atan: Spark expression conversion in NativeConverters, protobuf enum registration in auron.proto, and planner mapping in planner.rs. This keeps acosh aligned with the current architecture instead of introducing a custom extension function path. # What changes are included in this PR? This PR: adds Spark Acosh expression conversion in NativeConverters introduces ScalarFunction::Acosh in auron.proto maps ScalarFunction::Acosh in planner.rs enables acosh(expr) through the standard builtin ScalarFunction chain # Are there any user-facing changes? No. # How was this patch tested? CI. --------- Signed-off-by: weimingdiit <weimingdiit@gmail.com>

2026-03-23

[AURON #2093] Introduce kafka mock source (#2106) # Which issue does this PR close? Closes #2093 # Rationale for this change * Since we don’t have a Kafka environment, testing is difficult; therefore, we’ve introduced a simulated Kafka source that allows us to specify the data to be sent, thereby achieving our testing objectives. # What changes are included in this PR? * add kafka_mock_scan_exec to send mock data * add support for specifying mock data in the Kafka Table Factory * add test AuronKafkaSourceITCase and AuronKafkaSourceTestBase # Are there any user-facing changes? * No # How was this patch tested? * test via UT

2026-03-06

[AURON #2058] Introduce KafkaScanExec (#2072) # Which issue does this PR close? Closes #2058 # Rationale for this change * add native kafka consumer # What changes are included in this PR? * add Protobuf Node : KafkaScanExecNode * add Native kafka_scan_exec.rs # Are there any user-facing changes? * No # How was this patch tested? * No need test for kafka_scan_exec.rs * kafka_scan_exec#test_flink_kafka_partition_assign Validate Kafka partition allocation strategy.

2026-02-03

[AURON #1889] Implement monotonically_increasing_id() function (#1955) <!-- - Start the PR title with the related issue ID, e.g. '[AURON #XXXX] Short summary...'. --> # Which issue does this PR close? Closes #1889 # Rationale for this change Adds's support for non-deterministic function, as part of #1833 # What changes are included in this PR? Implements native support for Spark's `monotonically_increasing_id()`` function in Auron. **Functionality TL;DR:** The `monotonically_increasing_id()` function generates unique, monotonically increasing 64-bit integers across all partitions. Each partition generates IDs using the formula: ```go id := (partition_id << 33) | row_number ``` - Each ID is globally unique across all partitions - IDs increase monotonically within each partition - The upper 31 bits encode the partition ID, while the lower 33 bits encode the row number within that partition # Are there any user-facing changes? N/A # How was this patch tested? Unit tests

2026-01-27

[AURON #1739] Support LIMIT with OFFSET (#1740) <!-- - Start the PR title with the related issue ID, e.g. '[AURON #XXXX] Short summary...'. --> # Which issue does this PR close? Closes #1739 # Rationale for this change # What changes are included in this PR? # Are there any user-facing changes? # How was this patch tested?

2026-01-21

[AURON #1888] Implement spark_partition_id() function (#1928) <!-- - Start the PR title with the related issue ID, e.g. '[AURON #XXXX] Short summary...'. --> # Which issue does this PR close? Closes ##1888 # Rationale for this change This is part of the effort to add non-deterministic expressions to Auron. # What changes are included in this PR? This PR adds support for spark_partition_id using native # Are there any user-facing changes? N/A # How was this patch tested? Unit tests

2026-01-13

[AURON #1833] Refactor from_proto.rs to planner.rs (#1843) <!-- - Start the PR title with the related issue ID, e.g. '[AURON #XXXX] Short summary...'. --> # Which issue does this PR close? Closes #1833 # Rationale for this change Refactor planning logic from `from_proto.rs` into `planner.rs` and introduce `PhysicalPlanner` for converting Spark query plans to DataFusion physical plans, adding support for passing a task-level partition_id during conversion. This partition_id will be used by upcoming implementations of nondeterministic functions. # What changes are included in this PR? # Are there any user-facing changes? # How was this patch tested?

2026-01-12

[AURON #1792][FOLLOWUP] Broadcast isNullAwareAntiJoin flag (#1866) # Which issue does this PR close? Closes #1792 # Rationale for this change # What changes are included in this PR? # Are there any user-facing changes? # How was this patch tested? Add UT

2025-12-16

[AURON #1680] initCap semantics are aligned with Spark (#1681) # Which issue does this PR close? Closes #1680 # Rationale for this change The current initcap implementation uses DataFusion's initcap, which does not match Spark's semantics. Spark uses space-only word boundaries and title-cases the first letter while lowercasing the rest. # What changes are included in this PR? + Implement a new initcap native function aligned with Spark, similar to Spark's implementation logic: ` string.asInstanceOf[UTF8String].toLowerCase.toTitleCase`. + Refactor and expand initcap unit tests, adding corner cases. # Are there any user-facing changes? Yes. initcap results will now match Spark's semantics. # How was this patch tested? Added unit tests covering ASCII/non-ASCII, punctuation, space-only boundaries, and edge cases.

2025-11-13

make MD5 output back to utf8 (#1605)

2025-11-10

[AURON #1613] Introduce AuronExtFunctions (#1614) * [AURON #1613] Introduce AuronExtFunctions * fix conflicts

2025-11-05

[AURON #1588]Implement native function of isnan(87),nv2(83) ,nvl(82), greatest(85), least(84), find_in_set(81) #1588 #1585 (#1585) Co-authored-by: guihuawen <guihuawen@didiglobal.com>

2025-11-03

[AURON #1571]Implement native function of levenshtein(80) #1571 (#1573) * [AURON #1571]Implement native function of Levenshtein(80) #1571 * [AURON #1571]Implement native function of levenshtein(80) #1571 --------- Co-authored-by: guihuawen <guihuawen@didiglobal.com>

2025-10-28

[AURON #1513] Implement native function of pow/power. (#1514) * [AURON #1513] Implement native function of pow/power. Signed-off-by: slfan1989 <slfan1989@apache.org> * [AURON #1513] Implement native function of pow/power. Signed-off-by: slfan1989 <slfan1989@apache.org> --------- Signed-off-by: slfan1989 <slfan1989@apache.org>

2025-08-26

Introduce datafusion-spark crate to support some math functions (#1215) * Introduce datafusion-spark crate to support some math functions * fix style * Update spark-extension/src/main/scala/org/apache/spark/sql/auron/NativeConverters.scala Co-authored-by: cxzl25 <3898450+cxzl25@users.noreply.github.com> --------- Co-authored-by: cxzl25 <3898450+cxzl25@users.noreply.github.com>

The documentation is generated from this commit.

2025-08-20

Auron project parent to ASF (#1188)

2025-08-19

Update apache license header (#1180)

2025-08-18

rename Blaze to Auron (#1174) support Blaze as a deprecated name in configuration add auron-build.sh for building the project

rename Blaze to Auron support Blaze as a deprecated name in configuration add auron-build.sh for building the project

2025-08-15

rename Blaze to Auron

2025-06-09

add TID to logging (#1014) Co-authored-by: zhangli20 <zhangli20@kuaishou.com>

add TID to logging

2025-04-27

add expr string to SparkUDFWrapper (#967) Co-authored-by: lihao29 <lihao29@kuaishou.com>

add expr string to SparkUDFWrapper

Expect sha2 function result will be consistent with spark (#966)

2025-04-21

supports WindowGroupLimitExec (#957) Co-authored-by: zhangli20 <zhangli20@kuaishou.com>

2025-04-18

supports WindowGroupLimitExec

2025-04-17

code refactoring and bug fixes (#952) * fix rss writer: do not commit when native shuffle writing is not succeeded. fix unsafe lifetime error in spark UDAF wrapper. fix hash join and aggregate exec metrics when failed back to sorted refactor UnionExec to support PartitionerAwareUnionRDD. refactor celeborn reader: remove hard copied code from celeborn. * fix EmptyRDD error with empty local table scan --------- Co-authored-by: zhangli20 <zhangli20@kuaishou.com>

2025-04-16

fix rss writer: do not commit when native shuffle writing is not succeeded. fix unsafe lifetime error in spark UDAF wrapper. fix hash join and aggregate exec metrics when failed back to sorted refactor UnionExec to support PartitionerAwareUnionRDD. refactor celeborn reader: remove hard copied code from celeborn.

refactor UnionExec to support PartitionerAwareUnionRDD. fix unsafe lifetime error in spark UDAF wrapper.

2025-04-13

refactor UnionExec to support PartitionerAwareUnionRDDPartition

2025-04-07

convert scalar value using arrow ipc, fixing unsupported map type (#938) fix GetIndexField with scalar value fix arrow writer with map type Co-authored-by: zhangli20 <zhangli20@kuaishou.com>

2025-04-03

convert scalar value using arrow ipc, fixing unsupported map type fix GetIndexField with scalar value fix arrow writer with map type

2025-03-31

NativeConverters adds aggregate function return type (#930) Coalesce adds params casting Co-authored-by: zhangli20 <zhangli20@kuaishou.com>

NativeConverters adds aggregate function return type Coalesce adds params casting

rewrite UnionExec and support auto type casting (#927) Co-authored-by: zhangli20 <zhangli20@kuaishou.com>

2025-03-30

rewrite UnionExec and support auto type casting

2025-03-26

ProjectExec adds cast automatically when data types not matched (#916) Co-authored-by: zhangli20 <zhangli20@kuaishou.com>

ProjectExec adds cast automatically when data types not matched

2025-02-25

Supports UDAF and other aggregate functions not implemented (#848)

formate name and schema

optimize

2025-02-20

Dev udaf new merge NativeConverters

init DeclarativeAggregate udaf Resolving conflicted_file ArrowFFIExporter

2025-01-08

Supports range partitioning (#734) Co-authored-by: guoying06 <guoying06@kuaishou.com>

2024-12-30

update range partition

2024-12-25

update range partition

2024-12-17

Dev repartitioning (#693) * add round robin shuffle * update roundrobin * update roundrobin * update roundrobin * update roundrobin * update roundrobin log info * update roundrobin * update roundrobin * update roundrobin * update roundrobin * update roundrobin * update roundrobin * update partition proto * update partition proto * update sort before round robin * update sort before round robin * update round robin * update round robin * update sort_batch_by_partition_id test * update sort_batch_by_partition_id test --------- Co-authored-by: guoying06 <guoying06@kuaishou.com>

2024-12-13

update partition proto

update partition proto

2024-10-22

optimize bloom filter (#620) Co-authored-by: zhangli20 <zhangli20@kuaishou.com>

optimize bloom filter

2024-10-09

update to datafusion-v42/arrow-v53/arrow-java-v16 (#574) Co-authored-by: zhangli20 <zhangli20@kuaishou.com>

update to datafusion-v42/arrow-v53/arrow-java-v16

2024-09-14

support native scan orc format (#544) Co-authored-by: Zhang Li <richselian@gmail.com>

2024-07-30

supports bloom filter join (#532) fix decimal issue supports spark351 Co-authored-by: zhangli20 <zhangli20@kuaishou.com>

2024-07-29

supports bloom filter join fix decimal issue supports spark351

2024-07-08

supports native shuffled hash join (#509) * supports native shuffled hash join * supports distinct hash map during semi/anti join --------- Co-authored-by: zhangli20 <zhangli20@kuaishou.com>

2024-07-04

supports native shuffled hash join

2024-07-01

release version 3.0.0 (#506) * debug * supports using spark.io.compression.codec for shuffle/broadcast compression * use cached parquet metadata * remove out-of-date codes * refactor native broadcast to avoid duplicated broadcast jobs * fix in_list conversion in from_proto.rs * supports date type casting * release version 3.0.0 refactor join implementations to support existence joins and BHJ building hash map on driver side. supports spark333 batch shuffle reading. update rust-toolchain to latest nightly version. other minor improvements. update docs. --------- Co-authored-by: zhangli20 <zhangli20@kuaishou.com>

release version 3.0.0 refactor join implementations to support existence joins and BHJ building hash map on driver side. supports spark333 batch shuffle reading. update rust-toolchain to latest nightly version. other minor improvements. update docs.

2024-06-29

release version 3.0.0

2024-06-26

joins refactoring

2024-06-25

joins refactoring

2024-06-19

supports BHJ in blaze

2024-06-15

refactor sort-merge join

2024-05-24

supports UDTF fallbacks to spark (#483) Co-authored-by: zhangli20 <zhangli20@kuaishou.com>

2024-05-23

supports UDTF fallbacks to spark

2024-05-21

supports brickhouse UDF/UDAFs: array_union, collect, combine_unique (#479) Co-authored-by: zhangli20 <zhangli20@kuaishou.com>

2024-05-20

supports brickhouse UDF/UDAFs: array_union, collect, combine_unique

2024-05-14

supports parquet sink with zstd compression level. (#471) use stable sorting for dynamic partitions. fix parquet sink metrics. code formatting. Co-authored-by: zhangli20 <zhangli20@kuaishou.com>

supports parquet sink with zstd compression level. use stable sorting for dynamic partitions. fix parquet sink metrics. code formatting.

2024-04-02

update datafusion/arrow version to v36/v50. (#428) delegate sort-merge join to datafusion implementation. Co-authored-by: zhangli20 <zhangli20@kuaishou.com>

update datafusion/arrow version to v36/v50. delegate sort-merge join to datafusion implementation.

2024-03-20

fix acc loader/saver for AggDynScalar (#413) Co-authored-by: zhangli20 <zhangli20@kuaishou.com>

fix acc loader/saver for AggDynScalar

2024-03-04

implement get_json udf with sonic-rs (with serde-json fail-backing). (#399) implement json_tuple udtf. get_json flats nested arrays (keep consistent with hive UDFJson). Co-authored-by: zhangli20 <zhangli20@kuaishou.com>

2024-03-01

implement get_json udf with sonic-rs (with serde-json fail-backing). implement json_tuple udtf. get_json flats nested arrays (keep consistent with hive UDFJson).

implement get_json udf with sonic-rs (with serde-json fail-backing). implement json_tuple udtf. get_json flats nested arrays (keep consistent with hive UDFJson).

2024-02-22

supports compressing multiple batches in ipc format. improved batch_serde format for better compression. implement radix tournament tree for linear time k-way merging. implement accumulator store. other minor refactoring and optimizating.

supports compressing multiple batches in ipc format. (#392) improved batch_serde format for better compression. implement radix tournament tree for linear time k-way merging. implement accumulator store. other minor refactoring and optimizating. Co-authored-by: zhangli20 <zhangli20@kuaishou.com>

supports compressing multiple batches in ipc format. improved batch_serde format for better compression. implement radix tournament tree for linear time k-way merging. implement accumulator store. other minor refactoring and optimizating.

supports compressing multiple batches in ipc format. improved batch_serde format for better compression. implement radix tournament tree for linear time k-way merging. implement accumulator store. other minor refactoring and optimizating.

2024-01-25

close FSDataOutputStream after writing finished (#363) supports writing parquet table with dynamic partitions Co-authored-by: zhangli20 <zhangli20@kuaishou.com>

2024-01-14

close FSDataOutputStream after writing finished supports writing parquet table with dynamic partitions

2023-11-28

minor fixes and reafcatoring (#333) * code refactoring supports partial aggregate skipping fix incorrect assertion when converting concat_ws function fix ffi "not all nodes and buffers were consumed" issues * supports nested type hashing supports nested type array() function * use arrow snapshot version --------- Co-authored-by: zhangli20 <zhangli20@kuaishou.com>

2023-11-24

code refactoring supports partial aggregate skipping fix incorrect assertion when converting concat_ws function fix ffi "not all nodes and buffers were consumed" issues

2023-11-01

supports native broadcast nested loop join (#282) Co-authored-by: zhangli20 <zhangli20@kuaishou.com>

2023-10-31

supports native broadcast nested loop join

2023-09-26

add shims spark-3.3.3

refactor SMJ to support post filters add BlazeConf