Proto commits in kwai/blaze

These commits are when the Protocol Buffers files have changed: (only the last 100 relevant commits are shown)

2025-04-27

add expr string to SparkUDFWrapper (#967) Co-authored-by: lihao29 <lihao29@kuaishou.com>

The documentation is generated from this commit.

add expr string to SparkUDFWrapper

The documentation is generated from this commit.

Expect sha2 function result will be consistent with spark (#966)

2025-04-21

supports WindowGroupLimitExec (#957) Co-authored-by: zhangli20 <zhangli20@kuaishou.com>

2025-04-18

supports WindowGroupLimitExec

2025-04-17

code refactoring and bug fixes (#952) * fix rss writer: do not commit when native shuffle writing is not succeeded. fix unsafe lifetime error in spark UDAF wrapper. fix hash join and aggregate exec metrics when failed back to sorted refactor UnionExec to support PartitionerAwareUnionRDD. refactor celeborn reader: remove hard copied code from celeborn. * fix EmptyRDD error with empty local table scan --------- Co-authored-by: zhangli20 <zhangli20@kuaishou.com>

2025-04-16

fix rss writer: do not commit when native shuffle writing is not succeeded. fix unsafe lifetime error in spark UDAF wrapper. fix hash join and aggregate exec metrics when failed back to sorted refactor UnionExec to support PartitionerAwareUnionRDD. refactor celeborn reader: remove hard copied code from celeborn.

refactor UnionExec to support PartitionerAwareUnionRDD. fix unsafe lifetime error in spark UDAF wrapper.

2025-04-13

refactor UnionExec to support PartitionerAwareUnionRDDPartition

2025-04-07

convert scalar value using arrow ipc, fixing unsupported map type (#938) fix GetIndexField with scalar value fix arrow writer with map type Co-authored-by: zhangli20 <zhangli20@kuaishou.com>

2025-04-03

convert scalar value using arrow ipc, fixing unsupported map type fix GetIndexField with scalar value fix arrow writer with map type

2025-03-31

NativeConverters adds aggregate function return type (#930) Coalesce adds params casting Co-authored-by: zhangli20 <zhangli20@kuaishou.com>

NativeConverters adds aggregate function return type Coalesce adds params casting

rewrite UnionExec and support auto type casting (#927) Co-authored-by: zhangli20 <zhangli20@kuaishou.com>

2025-03-30

rewrite UnionExec and support auto type casting

2025-03-26

ProjectExec adds cast automatically when data types not matched (#916) Co-authored-by: zhangli20 <zhangli20@kuaishou.com>

ProjectExec adds cast automatically when data types not matched

2025-02-25

Supports UDAF and other aggregate functions not implemented (#848)

formate name and schema

optimize

2025-02-20

Dev udaf new merge NativeConverters

init DeclarativeAggregate udaf Resolving conflicted_file ArrowFFIExporter

2025-01-08

Supports range partitioning (#734) Co-authored-by: guoying06 <guoying06@kuaishou.com>

2024-12-30

update range partition

2024-12-25

update range partition

2024-12-17

Dev repartitioning (#693) * add round robin shuffle * update roundrobin * update roundrobin * update roundrobin * update roundrobin * update roundrobin log info * update roundrobin * update roundrobin * update roundrobin * update roundrobin * update roundrobin * update roundrobin * update partition proto * update partition proto * update sort before round robin * update sort before round robin * update round robin * update round robin * update sort_batch_by_partition_id test * update sort_batch_by_partition_id test --------- Co-authored-by: guoying06 <guoying06@kuaishou.com>

2024-12-13

update partition proto

update partition proto

2024-10-22

optimize bloom filter (#620) Co-authored-by: zhangli20 <zhangli20@kuaishou.com>

optimize bloom filter

2024-10-09

update to datafusion-v42/arrow-v53/arrow-java-v16 (#574) Co-authored-by: zhangli20 <zhangli20@kuaishou.com>

update to datafusion-v42/arrow-v53/arrow-java-v16

2024-09-14

support native scan orc format (#544) Co-authored-by: Zhang Li <richselian@gmail.com>

2024-07-30

supports bloom filter join (#532) fix decimal issue supports spark351 Co-authored-by: zhangli20 <zhangli20@kuaishou.com>

2024-07-29

supports bloom filter join fix decimal issue supports spark351

2024-07-08

supports native shuffled hash join (#509) * supports native shuffled hash join * supports distinct hash map during semi/anti join --------- Co-authored-by: zhangli20 <zhangli20@kuaishou.com>

2024-07-04

supports native shuffled hash join

2024-07-01

release version 3.0.0 (#506) * debug * supports using spark.io.compression.codec for shuffle/broadcast compression * use cached parquet metadata * remove out-of-date codes * refactor native broadcast to avoid duplicated broadcast jobs * fix in_list conversion in from_proto.rs * supports date type casting * release version 3.0.0 refactor join implementations to support existence joins and BHJ building hash map on driver side. supports spark333 batch shuffle reading. update rust-toolchain to latest nightly version. other minor improvements. update docs. --------- Co-authored-by: zhangli20 <zhangli20@kuaishou.com>

release version 3.0.0 refactor join implementations to support existence joins and BHJ building hash map on driver side. supports spark333 batch shuffle reading. update rust-toolchain to latest nightly version. other minor improvements. update docs.

2024-06-29

release version 3.0.0

2024-06-26

joins refactoring

2024-06-25

joins refactoring

2024-06-19

supports BHJ in blaze

2024-06-15

refactor sort-merge join

2024-05-24

supports UDTF fallbacks to spark (#483) Co-authored-by: zhangli20 <zhangli20@kuaishou.com>

2024-05-23

supports UDTF fallbacks to spark

2024-05-21

supports brickhouse UDF/UDAFs: array_union, collect, combine_unique (#479) Co-authored-by: zhangli20 <zhangli20@kuaishou.com>

2024-05-20

supports brickhouse UDF/UDAFs: array_union, collect, combine_unique

2024-05-14

supports parquet sink with zstd compression level. (#471) use stable sorting for dynamic partitions. fix parquet sink metrics. code formatting. Co-authored-by: zhangli20 <zhangli20@kuaishou.com>

supports parquet sink with zstd compression level. use stable sorting for dynamic partitions. fix parquet sink metrics. code formatting.

2024-04-02

update datafusion/arrow version to v36/v50. (#428) delegate sort-merge join to datafusion implementation. Co-authored-by: zhangli20 <zhangli20@kuaishou.com>

update datafusion/arrow version to v36/v50. delegate sort-merge join to datafusion implementation.

2024-03-20

fix acc loader/saver for AggDynScalar (#413) Co-authored-by: zhangli20 <zhangli20@kuaishou.com>

fix acc loader/saver for AggDynScalar

2024-03-04

implement get_json udf with sonic-rs (with serde-json fail-backing). (#399) implement json_tuple udtf. get_json flats nested arrays (keep consistent with hive UDFJson). Co-authored-by: zhangli20 <zhangli20@kuaishou.com>

2024-03-01

implement get_json udf with sonic-rs (with serde-json fail-backing). implement json_tuple udtf. get_json flats nested arrays (keep consistent with hive UDFJson).

implement get_json udf with sonic-rs (with serde-json fail-backing). implement json_tuple udtf. get_json flats nested arrays (keep consistent with hive UDFJson).

2024-02-22

supports compressing multiple batches in ipc format. improved batch_serde format for better compression. implement radix tournament tree for linear time k-way merging. implement accumulator store. other minor refactoring and optimizating.

supports compressing multiple batches in ipc format. (#392) improved batch_serde format for better compression. implement radix tournament tree for linear time k-way merging. implement accumulator store. other minor refactoring and optimizating. Co-authored-by: zhangli20 <zhangli20@kuaishou.com>

supports compressing multiple batches in ipc format. improved batch_serde format for better compression. implement radix tournament tree for linear time k-way merging. implement accumulator store. other minor refactoring and optimizating.

supports compressing multiple batches in ipc format. improved batch_serde format for better compression. implement radix tournament tree for linear time k-way merging. implement accumulator store. other minor refactoring and optimizating.

2024-01-25

close FSDataOutputStream after writing finished (#363) supports writing parquet table with dynamic partitions Co-authored-by: zhangli20 <zhangli20@kuaishou.com>

2024-01-14

close FSDataOutputStream after writing finished supports writing parquet table with dynamic partitions

2023-11-28

minor fixes and reafcatoring (#333) * code refactoring supports partial aggregate skipping fix incorrect assertion when converting concat_ws function fix ffi "not all nodes and buffers were consumed" issues * supports nested type hashing supports nested type array() function * use arrow snapshot version --------- Co-authored-by: zhangli20 <zhangli20@kuaishou.com>

2023-11-24

code refactoring supports partial aggregate skipping fix incorrect assertion when converting concat_ws function fix ffi "not all nodes and buffers were consumed" issues

2023-11-01

supports native broadcast nested loop join (#282) Co-authored-by: zhangli20 <zhangli20@kuaishou.com>

2023-10-31

supports native broadcast nested loop join

2023-09-26

add shims spark-3.3.3

refactor SMJ to support post filters add BlazeConf

native parquet writer

supports combining filter and projection exec. supports cached exprs evaluator.

supports native first() aggregator. supports scalar map value. refactor agg_buf by adding AggDynScalar.

convert If to CaseWhen expr

use sc_and/or from datafusion v25-blaze

remove datafusion-ext-file-format and move parquet-exec to datafusion-ext-plans

supports converting literal array values

new broadcast join

supports name_batch in named_struct

Map entries

supports all timestamp units use postcard to serialize/deserialize data types

next KDev_MR_link：https://ksurl.cn/SMbgH7by

reduce partitions size in parquet scan

implements native GenerateExec

1.update datafusion version from 15.0.0 to 20.0.0 2.update arrow version from 28.0.0 to 34.0.0 3.update parquet version from 28.0.0 to 34.0.0 4.use new method to convert like expr 5.concat_batch method fix(skip name check) 6.decimal compute in add/minus/multipy/divide cast to double and then pass to datafusion

remove SortMergeJoin.null_equals_null (always false in spark)

supports native SortAggregate

support rss shuffle for blaze

supports spillable aggregation

update datafusion to 15.0.0

supports native expand exec

implement short-circuiting logical operators fix aggregation transforming with inconvertible result expressions

add native if expression

coalesce batches before shuffle write and ipc write

refactor: split out datafusion-ext-common, datafusion-ext-plans

refactor: rename package plan-serde to blaze-serde

update arrow/datafusion to latest revision

add string expr

refactor SparkFallbackToJvmExpr to SparkExpressionWrapperExpr

implement native limits exec

fix error for partition column data type does not match