Proto commits in apache/datafusion

These commits are when the Protocol Buffers files have changed: (only the last 100 relevant commits are shown)

2026-06-23

Add `ListingOptions::output_partitioning` and `FileScanConfig::output_partitioning` for pre-defined file partitioning (#22657) ## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> - Closes #22645. ## Rationale for this change <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> This follows up on #22607 by replacing range-partitioning sqllogictest boilerplate with a general file/listing scan API for declared output partitioning. Related: #21992, #22607, https://github.com/apache/datafusion/pull/22607#discussion_r3323904683 ## What changes are included in this PR? <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> - Add declared `output_partitioning` to file scan and listing table configuration. - Preserve declared partition counts during listing-table file grouping. - Serialize scan `output_partitioning` through physical plan proto. - Refactor `range_partitioning.slt` to use a CSV `ListingTable` instead of a custom test-only `TableProvider` / `DataSource`. Contract: - Declared partitioning expressions are written against the full table schema before scan projection. For example, `Range([range_key@0], [(10), (20)], 3)` remains valid if the scan projects `range_key` and falls back to `UnknownPartitioning(3)` if `range_key` is not projected. - Listing tables create one file group per declared output partition (which can exceed `target_partitions`). It is up to the user to plan their partitioning. For example, a 4-partition range declaration creates four scan file groups, adding empty trailing groups when fewer files are present. - File group index is part of the contract: file group `i` must contain rows for declared output partition `i`. DataFusion does not validate row placement, matching other user-declared properties such as sortedness. ## Are these changes tested? <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> Yes. ## Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> Yes. This adds public API for declaring file/listing scan output partitioning. No breaking API changes. <!-- If there are any breaking changes to public APIs, please add the `api change` label. --> --------- Co-authored-by: Gabriel <45515538+gabotechs@users.noreply.github.com>

The documentation is generated from this commit.

2026-06-22

feat: logical plan protobuf representation for range repartitioning (#23030) ## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> - Closes #22787 ## Rationale for this change <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> The range repartitioning scheme for logical plans does not currently have a protobuf representation. ## What changes are included in this PR? <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> A protobuf representation of the `RangeRepartition` struct was added to `datafusion.proto`, and the codegened Rust types were created. Added logic for serializing and deserializing to and from the protobuf representation, and a roundtrip test as well! ## Are these changes tested? <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> Yes! Added a test in `roundtrip_logical_plan` ## Are there any user-facing changes? No, adding internal protobuf serialization support for an existing logical plan variant <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> <!-- If there are any breaking changes to public APIs, please add the `api change` label. -->

2026-06-17

Remove redundant `collect_stat` and `target_partitions` on `ListingOptions` (#22969) ## Which issue does this PR close? - Closes #. ## Rationale for this change Something that was spotted during the review of: - https://github.com/apache/datafusion/pull/22657 `ListingOptions::target_partitions` and `ListingOptions::collect_stat` duplicate `SessionConfig`'s `execution.target_partitions` and `execution.collect_statistics`. After some investigation, I think they only live on `ListingOptions` for historical reasons: when the struct was added (#1010 5 years ago), `TableProvider::scan` had no access to the session, so the values had to be copied onto the table at build time. Once #2660 passed `SessionState` into `scan`, the fields became redundant (and had already drifted — `scan` read them from the session config while `list_files_for_scan` read the stale copy). This PR makes `SessionConfig` the single source of truth. ## What changes are included in this PR? - Remove `target_partitions`/`collect_stat` fields, their builders, and `with_session_config_options` from `ListingOptions`. - `ListingTable` now reads both values from the session config at scan time. - Reserve proto tags 8/9 in `ListingTableScanNode` and drop the related (de)serialization. - Update benchmarks, factory, and test call sites. ## Are these changes tested? Yes, by existing tests ## Are there any user-facing changes? Yes, breaking: the removed fields/builders require configuring `SessionConfig` instead, and the two proto fields no longer round-trip. --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>

2026-06-16

Add MERGE INTO types to datafusion-expr (#20763) ## Which issue does this PR close? - part of #20746 [EPIC] Complete DML Support (MERGE, INSERT OVERWRITE, TRUNCATE) #19617 As well as task 1 of #20746 ## Rationale for this change Lay the foundation for MERGE INTO support in DataFusion by adding the logical plan types and their proto serialization. Keeping types separate from execution lets reviewers reason about the data model independently of the planner and physical dispatch. ## What changes are included in this PR? **`datafusion/expr` — new types in `dml.rs`** - `MergeIntoOp` — carries the `ON` join condition and ordered list of `WHEN` clauses - `MergeIntoClause` — a single `WHEN` clause: kind + optional predicate + action - `MergeIntoClauseKind` — `Matched` / `NotMatched` / `NotMatchedByTarget` / `NotMatchedBySource`; includes `is_not_matched_by_target()` and `canonical()` helpers because `NotMatched` and `NotMatchedByTarget` are semantically identical and must be treated identically downstream - `MergeIntoAction` — `Update(Vec<(col, expr)>)` / `Insert { columns, values }` / `Delete` - `WriteOp::MergeInto(MergeIntoOp)` variant added to the existing `WriteOp` enum; `WriteOp` is now `#[non_exhaustive]` so future variant additions are not a SemVer break **`datafusion/proto-models` — proto schema** - Extended `DmlNode` with a `MERGE_INTO` type tag and a boxed `MergeIntoOpNode` payload field - Added `MergeIntoOpNode`, `MergeIntoClauseNode`, `MergeIntoActionNode` messages **`datafusion/proto` — serialization** - `from_proto`: `parse_write_op(&DmlNode, ...)` reads the payload when the type tag is `MergeInto`; defensive helpers `parse_merge_into_op/clause/action` with explicit errors for missing fields - `to_proto`: `serialize_merge_into_op/clause/action` helpers; encode path uses an explicit `match` over all `WriteOp` variants producing `(dml_type, merge_into)` pair — no silent payload loss - Cross-crate conversions use `FromProto` (the crate-local trait) rather than `From` to satisfy the Rust orphan rule after the upstream `datafusion-proto-models` refactor **Proto codegen** — after editing `.proto` files, regenerate with: ```bash PROTOC=/tmp/protoc cargo run --manifest-path datafusion/proto-models/gen/Cargo.toml ``` (Install `protoc` from https://github.com/protocolbuffers/protobuf/releases if not present; set `PROTOC` to its path.) ## Are these changes tested? - `datafusion-expr` unit tests: `WriteOp::MergeInto` display, `is_not_matched_by_target`, `canonical` - `datafusion-proto` round-trip test: exercises all four `MergeIntoClauseKind` variants and all three `MergeIntoAction` variants through encode → decode - `datafusion-proto` error-path tests: missing `merge_into` payload, missing `on` expression, unknown clause kind tag, missing clause action, missing action oneof ## Are there any user-facing changes? `WriteOp` gains a `MergeInto` variant and is now `#[non_exhaustive]`. Existing downstream `match` arms need a wildcard arm added (this is intentional and expected for a new DML operation). ## Follow-up A stacking PR that adds the SQL planner, physical planner dispatch, and `TableProvider::merge_into` hook is available at https://github.com/wirybeaver/datafusion/pull/2. If reviewers prefer to review both together in one pass, I'm happy to include that work here instead. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

2026-06-08

[branch-54] fix: preserve null_aware on logical JoinNode proto round-trip (backport #22104) (#22785) ## Which issue does this PR close? - Backport of #22104 to `branch-54` (for 54.1.0, tracked in #22547). This PR: - Backports #22104 to the `branch-54` line so the `null_aware` proto round-trip fix ships in 54.1.0, as requested in https://github.com/apache/datafusion/issues/22065#issuecomment-4634038807 Clean cherry-pick; `datafusion-proto` builds and both round-trip regression tests pass on `branch-54`.

2026-06-05

feat: add max_row_group_bytes option to ParquetOptions (#22649) ## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> - Closes https://github.com/apache/datafusion/issues/22650. ## Rationale for this change <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> arrow-rs 58.0 added WriterProperties::set_max_row_group_bytes (PR: apache/arrow-rs#9357 Issue: apache/arrow-rs#1213), which flushes a row group when either the row-count or the byte limit is reached, whichever comes first, matching parquet-mr's parquet.block.size. DataFusion already consumes atleast this version of arrow but does not yet expose this new byte-based setter through its config. ## What changes are included in this PR? <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> - Add `max_row_group_bytes: Option<usize>` (default None) to ParquetOptions in `datafusion/common/src/config.rs`. - Wire it through `ParquetOptions::into_writer_properties_builder` to `WriterPropertiesBuilder::set_max_row_group_bytes`, with a guard that rejects Some(0) as a configuration error (arrow-rs panics on a zero byte limit). - Plumb the field through protobuf serialization - add it to the ParquetOptions proto message and the proto-common/proto conversions, with regenerated bindings. - Exposed as the max_row_group_bytes COPY / CREATE EXTERNAL TABLE format option alongside max_row_group_size. - Update the generated config docs and the format options table doc. ## Are these changes tested? <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> Yes - run locally and passing: Unit (datafusion-common, parquet_writer.rs): - defaults to None, so no byte limit is propagated to WriterProperties. - a configured value propagates to WriterProperties. - Some(0) is rejected with a configuration error. - the existing table_parquet_opts_to_writer_props round-trip and test_defaults_match tests were extended to cover the new field. Protobuf round-trip (datafusion-proto-common): - new test_parquet_options_max_row_group_bytes_round_trip confirms the option survives serialization to protobuf and back. SLTs: - new test_files/parquet_max_row_group_bytes.slt writes Parquet with the option set (via both COPY ... OPTIONS and session config), reads it back, asserts the data round-trips, and asserts a zero value is rejected. - copy.slt exercises the option inside the existing "all supported statement overrides" COPY test. - information_schema.slt updated for the new option in SHOW ALL. Commands run locally (all pass): cargo test -p datafusion-common --features parquet cargo test -p datafusion-proto-common cargo test -p datafusion-proto cargo test --test sqllogictests -- parquet_max_row_group_bytes cargo test --test sqllogictests -- information_schema cargo test --test sqllogictests -- copy ## Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> Additive only, does not affect existing options. <!-- If there are any breaking changes to public APIs, please add the `api change` label. --> --------- Co-authored-by: Yongting You <2010youy01@gmail.com>

2026-06-03

[branch-54] refactor: give parquet CDC options an explicit `enabled` flag (backport #22632) (#22648) ## Which issue does this PR close? - Backport of #22632 to `branch-54`. ## Rationale for this change Content-defined chunking (CDC) write options were added in #21110 and are slated for the 54.0.0 release. This backports the refactor in #22632 so the config/proto surface ships in its final form, before the release goes out. The CDC options previously worked as `use_content_defined_chunking: Option<CdcOptions>` with a `ConfigField` impl that accepted a bare `use_content_defined_chunking = true|false` and otherwise enabled CDC implicitly when any sub-field was set. This has a few problems: - **Naming diverges from parquet-rs.** `WriterProperties` exposes `content_defined_chunking()` / `set_content_defined_chunking(Option<CdcOptions>)` with no `use_` prefix. - **Implicit / order-dependent on the SQL side.** Format options in `COPY ... OPTIONS` / `CREATE EXTERNAL TABLE ... OPTIONS` are applied from a `HashMap` (non-deterministic order). With the old bare-boolean form, mixing `... = false` with a sub-field could resolve to enabled or disabled depending on iteration order. - **Extra machinery.** Supporting the bare boolean required hand-written `ConfigField` impls and a `#[expect(clippy::should_implement_trait)]` workaround, plus a zero-sentinel fallback in the proto mapping. Since CDC is unreleased, the config/proto surface can still be changed freely. ## What changes are included in this PR? - Rename the `ParquetOptions` field `use_content_defined_chunking` -> `content_defined_chunking` (matches parquet-rs). - Make `CdcOptions` a plain `config_namespace!` with an explicit `enabled: bool` field alongside the chunking parameters; the field is a bare `CdcOptions` (no longer `Option<CdcOptions>`). CDC is on iff `content_defined_chunking.enabled` is true. Setting a parameter no longer implicitly enables CDC, and the result is independent of key order. - Add `CdcOptions::enabled()` / `CdcOptions::disabled()` shorthand constructors. - Drop the `ConfigField` impls and the `should_implement_trait` workaround — all generated by the macro now. - Add an `enabled` field to the proto `CdcOptions` message so the proto <-> config mapping is a plain field copy in both directions. - Update unit tests, regenerate config docs + the `information_schema` snapshot, and add `parquet_cdc_config.slt` documenting the resolution behavior. ## Are these changes tested? Yes — `datafusion-common` config + writer unit tests, `datafusion-proto-common` proto round-trip tests, `datafusion/core` parquet integration tests, and sqllogictest (`parquet_cdc.slt` + new `parquet_cdc_config.slt`). Cherry-pick applied cleanly onto `branch-54`; affected crates build and the CDC unit tests pass. ## Are there any user-facing changes? Yes, but only to the unreleased CDC options: - Config key `datafusion.execution.parquet.use_content_defined_chunking` -> `datafusion.execution.parquet.content_defined_chunking.enabled` (plus `.min_chunk_size` / `.max_chunk_size` / `.norm_level`). - The bare-boolean form is removed; enable/disable via `content_defined_chunking.enabled = true|false`. No released API is affected. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

refactor: give parquet CDC options an explicit `enabled` flag (#22632) ## Which issue does this PR close? - None ## Rationale for this change The CDC options currently work as `use_content_defined_chunking: Option<CdcOptions>` with a `ConfigField` impl that accepts a bare `use_content_defined_chunking = true|false` and otherwise enables CDC implicitly when any sub-field is set. This has a few problems: - **Naming diverges from parquet-rs.** `WriterProperties` exposes `content_defined_chunking()` / `set_content_defined_chunking(Option<CdcOptions>)` with no `use_` prefix. - **Implicit / order-dependent on the SQL side.** Format options in `COPY ... OPTIONS` / `CREATE EXTERNAL TABLE ... OPTIONS` are applied from a `HashMap` (non-deterministic order). With the old bare-boolean form, mixing `... = false` with a sub-field, or setting a sub-field after `= false`, could resolve to enabled or disabled depending on iteration order. - **Extra machinery.** Supporting the bare boolean required a hand-written `impl ConfigField for CdcOptions` + `impl ConfigField for Option<CdcOptions>` and a `#[expect(clippy::should_implement_trait)]` workaround, plus a zero-sentinel fallback in the proto mapping. Since CDC is unreleased, the config/proto surface can still be changed freely. ## What changes are included in this PR? - Rename the `ParquetOptions` field `use_content_defined_chunking` -> `content_defined_chunking` (matches parquet-rs). - Make `CdcOptions` a plain `config_namespace!` with an explicit `enabled: bool` field alongside the chunking parameters; the field is a bare `CdcOptions` (no longer `Option<CdcOptions>`). CDC is on if `content_defined_chunking.enabled` is true. Setting a parameter no longer implicitly enables CDC, and the result is independent of key order. - Add `CdcOptions::enabled()` / `CdcOptions::disabled()` shorthand constructors. - Drop the `ConfigField` impls and the `should_implement_trait` workaround — all generated by the macro now. - Add an `enabled` field to the proto `CdcOptions` message so the proto <-> config mapping is a plain field copy in both directions (removes the presence-encoding and the zero-sentinel fallback). - Update unit tests, regenerate config docs + the `information_schema` snapshot, and add `parquet_cdc_config.slt` documenting the resolution behavior. ## Are these changes tested? Yes: - `datafusion-common` config + writer unit tests (enable toggle, parameter-does-not-enable, validation, writer round-trip). - `datafusion-proto-common` proto round-trip tests (enabled / disabled / negative norm level). - `datafusion/core` parquet integration tests (data round-trip, page boundaries). - sqllogictest: `parquet_cdc.slt` (end-to-end) and a new `parquet_cdc_config.slt` (config resolution / order independence). ## Are there any user-facing changes? Yes, but only to the unreleased CDC options: - Config key `datafusion.execution.parquet.use_content_defined_chunking` -> `datafusion.execution.parquet.content_defined_chunking.enabled` (plus `.min_chunk_size` / `.max_chunk_size` / `.norm_level`). - The bare-boolean form is removed; enable/disable via `content_defined_chunking.enabled = true|false`. No released API is affected. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

2026-06-02

Allow specifying an arrow schema for PartitionedFile (#22360) ## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> - Closes https://github.com/apache/datafusion/issues/22200. ## Rationale for this change <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> As described in the linked issue, parsing the arrow schema from parquet metadata can be expensive for point lookups, relative to the rest of the query execution pipeline. If the user knows the arrow schema of the file, they should be able to specify it explicitly. ## What changes are included in this PR? <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> * Add a `arrow_schema: SchemaRef` field to `PartitionedFile` * Use the `arrow_schema` field in the parquet opener to bypass schema inference from the `ARROW:schema` metadata field. ## Are these changes tested? Added unit tests for both matching and mismatching schemas. <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> ## Are there any user-facing changes? There are no breaking changes, the new field is optional and is set to None by default. <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> <!-- If there are any breaking changes to public APIs, please add the `api change` label. -->

2026-05-30

feat: add pgjson format support for EXPLAIN ANALYZE (#21767) ## Which issue does this PR close? - Closes #. ## Rationale for this change DataFusion already emits PostgreSQL JSON (pgjson) for logical plans via `EXPLAIN (FORMAT pgjson) ...`. This PR extends that support to `EXPLAIN ANALYZE` so the physical plan, along with live execution metrics, can be fed into pgjson visualizers such as [Dalibo](https://explain.dalibo.com/) and PEV2. Today, `EXPLAIN ANALYZE FORMAT pgjson` is explicitly rejected in the planner with `"EXPLAIN ANALYZE with FORMAT is not supported"`. With this PR the restriction is lifted for pgjson. ## What changes are included in this PR? - Add a `format: ExplainFormat` field to the logical `Analyze` node and the physical `AnalyzeExec` operator, threaded through SQL parsing, logical planning, and physical planning. - Accept `EXPLAIN ANALYZE FORMAT pgjson <stmt>`. `Tree` and `Graphviz` with `ANALYZE` still error with a clear message (out of scope for this PR). - Add `DisplayableExecutionPlan::pgjson()` and a new `PgJsonExecutionPlanVisitor` that mirror the logical-plan `PgJsonVisitor`. Per-node output includes: - `Node Type` — `ExecutionPlan::name()` - `Details` — the one-line `DisplayAs::Default` rendering - `Actual Rows` / `Actual Total Time` — PG-canonical metric keys populated from `output_rows` / `elapsed_compute` (emitted as float milliseconds; note DataFusion records compute time, not wall time) - `Extras` — remaining DataFusion metrics keyed by their native name - `Plans` — child nodes - Add an optional `set_summary()` builder so `AnalyzeExec` can attach `Total Rows` and `Duration` at the root in verbose mode. - Honor existing `analyze_level` / `analyze_categories` config exactly as `indent()` does. - Update the `EXPLAIN` user-guide docs (`docs/source/user-guide/sql/explain.md` and `explain-usage.md`) to document pgjson support under `ANALYZE` and lead with the Postgres-style option-list spelling. ### Composes with the `EXPLAIN (...)` option list (#21768) This builds on the now-merged Postgres-style option list (#21768). Because both the keyword form and the parenthesized option list parse into a single `ExplainStatementOptions` that is threaded through `explain_to_plan`, pgjson works with **both** spellings, and the `METRICS` / `LEVEL` knobs from #21768 compose with it in one statement: ```sql EXPLAIN (ANALYZE, FORMAT pgjson) SELECT count(*) FROM t; EXPLAIN (ANALYZE, FORMAT pgjson, METRICS 'rows', LEVEL summary) SELECT count(*) FROM t; ``` The parenthesized form is the idiomatic spelling for pgjson workflows since it mirrors Postgres's `EXPLAIN (ANALYZE, FORMAT json)` — exactly what visualizers like Dalibo / PEV2 document. (Note: `ANALYZE` must go *inside* the parens; a bare `EXPLAIN ANALYZE (FORMAT pgjson)` is invalid, as it is in Postgres.) ## Are these changes tested? - Unit tests in `datafusion/physical-plan/src/display.rs`: - `pgjson_renders_plan_without_metrics` - `pgjson_includes_summary_when_set` - `pgjson_snapshot_of_sample_plan` (insta snapshot) - sqllogictest coverage in `datafusion/sqllogictest/test_files/explain_analyze.slt`: - Structural golden for `EXPLAIN (ANALYZE, FORMAT PGJSON, METRICS 'none')` (option-list form) - `EXPLAIN (ANALYZE, FORMAT PGJSON, METRICS 'rows')` showing `Actual Rows` surfacing - Keyword form `EXPLAIN ANALYZE FORMAT pgjson` still works - Negative tests for `EXPLAIN ANALYZE FORMAT tree` and `EXPLAIN ANALYZE FORMAT graphviz` - `cargo clippy --all-targets --all-features -- -D warnings` clean on the touched crates; `cargo fmt --all` clean. ## Are there any user-facing changes? Yes — `EXPLAIN ANALYZE` now accepts the `pgjson` format, in either spelling: ```sql -- Postgres-style option list (idiomatic; composes with METRICS / LEVEL) EXPLAIN (ANALYZE, FORMAT pgjson) SELECT count(*) FROM t; -- legacy keyword form EXPLAIN ANALYZE FORMAT pgjson SELECT count(*) FROM t; ``` No existing behavior changes: the default (`EXPLAIN ANALYZE ...` with no `FORMAT`) still emits the indent-format plan with metrics, and `EXPLAIN (FORMAT pgjson) ...` on the logical plan is unchanged. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>

2026-05-28

feat(sql): Postgres-style `EXPLAIN (...)` option list (#21768) ## Which issue does this PR close? - Closes #. (Follow-up to #21160, which introduced per-category metric filtering via session config. This PR lets users reach those knobs inline from the EXPLAIN statement.) ## Rationale for this change #21160 added metric categories (`Rows`, `Bytes`, `Timing`, `Uncategorized`) and a verbosity level (`Summary`, `Dev`) to DataFusion's metrics, exposed today only via session config: - `datafusion.explain.analyze_categories` - `datafusion.explain.analyze_level` Users have to `SET` these out-of-band before running `EXPLAIN ANALYZE`, which is awkward for ad-hoc debugging. Postgres solves this with its parenthesized option list: ```sql EXPLAIN (ANALYZE, BUFFERS, VERBOSE, SETTINGS, WAL) SELECT ... ; ``` This PR adds the same ergonomics to DataFusion, mapping option names to DataFusion's existing semantics rather than Postgres's buffer/WAL model. ## What changes are included in this PR? **Parser.** On dialects whose `supports_explain_with_utility_options()` returns true (the default `GenericDialect`, `PostgreSqlDialect`, `DuckDbDialect`, etc.), `DFParser::parse_explain` delegates to sqlparser's `pub fn parse_utility_options()` and feeds the result through a new `ExplainStatementOptions::from_utility_options`. The legacy keyword form (`EXPLAIN ANALYZE VERBOSE FORMAT tree ...`) is unchanged. **Normalized option type.** A new `ExplainStatementOptions` in `datafusion-common` captures the knobs parsed from either form. Argument parsing reuses existing `ExplainFormat::from_str`, `ExplainAnalyzeCategories::from_str`, and `MetricType::from_str`. **Options accepted:** | Option | Argument | Effect | | --------- | ---------------- | --------------------------------------------------------------------- | | `ANALYZE` | bool, default T | Same as keyword `ANALYZE` | | `VERBOSE` | bool, default T | Same as keyword `VERBOSE` | | `FORMAT` | ident/string | `indent` / `tree` / `pgjson` / `graphviz` | | `METRICS` | string | `'all'`, `'none'`, or comma-separated `rows,bytes,timing,uncategorized` | | `LEVEL` | ident/string | `summary` or `dev` | | `TIMING` | bool | Sugar: toggles inclusion of the `timing` category | | `SUMMARY` | bool | Sugar: TRUE → `summary`, FALSE → `dev` | | `COSTS` | bool | Per-statement `show_statistics` override (not valid with `ANALYZE`) | Postgres-only options (`BUFFERS`, `WAL`, `SETTINGS`, `GENERIC_PLAN`, `MEMORY`) return a helpful unsupported-option error. **Logical plan.** `Analyze` gains `analyze_level: Option<MetricType>` and `analyze_categories: Option<ExplainAnalyzeCategories>`. `Explain` gains `show_statistics: Option<bool>`. `None` means "fall back to session config" — existing callers are unchanged. **Physical planner.** `handle_analyze` and `handle_explain` prefer statement-level overrides over session config before constructing `AnalyzeExec` / `ExplainExec`. `AnalyzeExec` itself needs no change — it already accepts the filters from #21160. **Proto.** The new override fields round-trip through `datafusion-proto`: - `datafusion_common.proto` gains `MetricType`, `MetricCategory`, and an `ExplainAnalyzeCategoriesNode` wrapper (`bool all` + `repeated MetricCategory only`, mirroring the Rust enum's `All` / `Only(Vec<…>)` variants). - `AnalyzeNode` gains `optional MetricType analyze_level` and `optional ExplainAnalyzeCategoriesNode analyze_categories`; `ExplainNode` gains `optional bool show_statistics`. - `ExplainOption` is extended with `analyze_level` / `analyze_categories` setters so the proto decode arms construct `LogicalPlan::Analyze` / `LogicalPlan::Explain` through the same `LogicalPlanBuilder::explain_option_format` path as the SQL planner. ## Are these changes tested? Yes: - **Unit tests** in `datafusion/sql/src/parser.rs` cover legacy keyword form on PostgreSQL dialect, each option form (`bare`, `= val`, `ON/OFF`, quoted), unknown-option errors, dialect gating (the parenthesized form is rejected under a dialect that doesn't enable it), and the error path for unsupported Postgres-only options. - **Integration tests** in `datafusion/core/tests/sql/explain_analyze.rs` — `explain_analyze_paren_metrics_filtering`, `explain_analyze_paren_level_overrides_session_config`, `explain_analyze_paren_metrics_overrides_session_config`, `explain_paren_buffers_rejected`. - **sqllogictest** fixtures in `datafusion/sqllogictest/test_files/explain.slt` covering the parenthesized form, round-trip with the legacy form, and each error path. - **Proto round-trip tests** in `datafusion/proto/tests/cases/roundtrip_logical_plan.rs` — `roundtrip_explain_show_statistics_override`, `roundtrip_analyze_level_override`, `roundtrip_analyze_categories_override` — cover each field set and unset, including `All`, `Only(vec![])` (plan-only), and a fully populated `Only` list. Ran `cargo fmt --all` and `cargo clippy --all-targets --all-features -- -D warnings` (clean). Two pre-existing test failures on `main` (`test_display_pg_json` snapshot and a `pgjson` SLT case at `explain.slt:642`) are unrelated to this change — verified by running them against a clean checkout of the same base commit. ## Are there any user-facing changes? Yes — new syntax. User-facing docs updated at `docs/source/user-guide/explain-usage.md` with a new section describing the option list and the dialect gate. No breaking changes: the legacy keyword form continues to work exactly as before. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

2026-05-27

Add Physical `Partitioning::Range` enum variant (#22207) ## Which issue does this PR close? - First mechanical PR for `ExprPartitioning` as described in thread: #21992. ## Rationale for this change DataFusion currently cannot truthfully represent range-partitioned physical data. Some sources may be range partitioned, but have to advertise another partitioning shape or fall back to unknown partitioning. This PR introduces the metadata shape for range partitioning without implementing optimizer or execution behavior yet. The goal is to establish the public representation first, then implement planning, compatibility, and execution behavior incrementally in follow-up PRs. ## What changes are included in this PR? - Adds `Partitioning::Range(RangePartitioning)`. - Adds range metadata types: - `RangePartitioning` - `RangePartition` - `RangeInterval` - `RangeBound` - Adds proto serialization/deserialization. - Adds `not_impl_err!` handling for range partitioning at call sites. - Preserves range partitioning through projection only when all partition expressions can be projected, otherwise `UnknownPartitioning`. ## Are these changes tested? Yes. ## Are there any user-facing changes? Yes. This adds new public physical partitioning API and proto for range partitioning. --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>

2026-05-22

Fix missing field `partitioned_by_file_group` in serialization (#22365) I'm not super versed in the serialization machinery involved here, please review carefully. ## Which issue does this PR close? - Closes #22363. ## Rationale for this change The partitioned_by_file_group field was introduced in #21351 and #21342 but not added to the protobuf schema, breaking `datafusion-distributed`. ## What changes are included in this PR? - Add optional `bool partitioned_by_file_group = 14` to `FileScanExecConf` in `datafusion.proto` - Serialize the field in `to_proto.rs` - Deserialize the field in `from_proto.rs` - Regenerate prost/pbjson code ## Are these changes tested? Yes, added roundtrip_parquet_exec_partitioned_by_file_group test. ## Are there any user-facing changes? No

Split proto serialization to encapsulate private state (#21835) (#21929) ## Which issue does this PR close? - Closes #21835. ## Rationale for this change `datafusion-proto` serializes every built-in `PhysicalExpr` through a single ~300-line `downcast_ref` chain, with a mirror `match` on the decode side. That chain lives outside the crate where each expression is defined, so every field an expression wants to round-trip has to be made `pub`. #21807 is the cautionary tale: it had to add five `pub` "proto-only, not stable" items to `DynamicFilterPhysicalExpr` just to serialize an `RwLock`-wrapped inner. This PR adds the infrastructure so a `PhysicalExpr` can serialize itself and keep its state private. ## What changes are included in this PR? A `PhysicalExpr` can now opt into serializing itself, in both directions: ```rust fn try_to_proto(&self, ctx: &PhysicalExprEncodeCtx) -> Result<Option<PhysicalExprNode>> fn try_from_proto(node: &PhysicalExprNode, ctx: &PhysicalExprDecodeCtx) -> Result<Arc<dyn PhysicalExpr>> ``` `try_to_proto` returning `Ok(None)` (the default) means "fall through to the old downcast chain", so the change is purely additive — nothing is forced to migrate. `Column` and `BinaryExpr` are migrated as working demos; everything else stays on the old path and migrates later, one expression at a time, with no wire-format change. Five stacked commits, each builds green on its own and is independently reviewable (or splittable into its own PR): 1. **Extract `datafusion-proto-models` crate** — move the `.proto` file and prost-generated types into a lightweight crate (mirrors the existing `datafusion-proto-common` split). 2. **Add the `try_to_proto` hook** — feature-gated, off by default. 3. **Migrate `Column` encode.** 4. **Add the decode side and migrate `Column` decode.** 5. **Migrate `BinaryExpr`** (both directions). ## A few design decisions worth flagging - **`FromProto` / `TryFromProto` traits instead of plain `From` / `TryFrom`.** Once the prost types move into their own crate they are *foreign* to `datafusion-proto`, and the orphan rule forbids `impl From<&protobuf::X> for Y` when both `X` and `Y` are foreign. So those conversions become `FromProto` / `TryFromProto` traits in `datafusion_proto::convert`, and callers go from `(&x).into()` to `Y::from_proto(&x)`. This is a known workaround, not the end state — see Future work. - **The ctx is a concrete struct, not `&dyn`.** `PhysicalExprEncodeCtx` / `PhysicalExprDecodeCtx` wrap a sealed dispatch trait. Keeping them concrete keeps `&dyn` out of every expression's signature and gives a stable place to add helpers (UDF encoding, registry hooks) later without churning a public trait. - **`try_from_proto` takes the whole `PhysicalExprNode`**, not the pre-unwrapped variant payload, so every expression's decoder has the same signature and can still see outer-node fields like `expr_id`. ## Are these changes tested? No new behavior, so no new tests. `Column` and `BinaryExpr` produce and consume the same wire format as before; the existing `roundtrip_physical_plan` / `roundtrip_physical_expr` tests already cover both directions and now exercise the new path. ## Are there any user-facing changes? Small API breaks in `datafusion-proto`: - `try_from_physical_plan_with_converter` / `try_into_physical_plan_with_converter` move to a `PhysicalPlanNodeExt` trait — callers add `use datafusion_proto::physical_plan::PhysicalPlanNodeExt;`. - Foreign-foreign `From` / `TryFrom` conversions become `FromProto` / `TryFromProto` (see Design decisions above). - `datafusion_proto::generated::*` is deprecated in favor of `datafusion_proto::protobuf`; it still works. The new `proto` feature on `datafusion-physical-expr(-common)` is off by default, so crates that don't serialize plans pay nothing. ## Future work - Migrate the remaining built-in expressions — including `DynamicFilterPhysicalExpr`, the original motivation — one per follow-up PR. - Apply the same pattern to `ExecutionPlan` serialization. - Drop the `FromProto` / `TryFromProto` workaround: collapse `datafusion-proto-common` into `datafusion-proto-models` and push the conversion impls down to the target-type crates so callers use plain `From` / `TryFrom` again. Full dep-graph analysis and a step-by-step plan are in [#21835 (comment)](https://github.com/apache/datafusion/issues/21835#issuecomment-4348350257). 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

2026-05-21

fix: preserve null_aware on logical JoinNode proto round-trip (#22104) ## Summary Closes #22065. `null_aware` was missing from `JoinNode` in the logical proto (it was added to the physical `HashJoinExecNode` in #19635). The encoder dropped it via `..` destructuring and the decoder had no field to restore it from, so any `to_proto` -> `from_proto` round trip silently downgraded a null-aware LeftAnti (NOT IN semantics) to a plain LeftAnti and returned wrong rows. ## Changes - Add `bool null_aware = 9;` to `JoinNode`. - Decoder switches to `Join::try_new`, plumbing `null_aware` and `null_equality` (same bug, same path) from the wire. - Encoder destructure binds `schema: _` instead of `..`, so any future `Join` field is a compile error here instead of a silent drop. - Decoder rejects mismatched `left_join_key`/`right_join_key` lengths via `proto_error`. - Regression tests `roundtrip_join_null_aware` and `roundtrip_join_null_equality`, each exercising one non-default field. ## Test plan - `cargo test -p datafusion-proto --test proto_integration cases::roundtrip_logical_plan` passes. - Clippy clean.

2026-05-20

feat: optional timezone for coerce_int96 (#22318) ## Which issue does this PR close? N/A ## Rationale for this change `coerce_int96_to_resolution` currently produces `Timestamp(unit, None)` for every INT96-derived column. Some downstream readers need the resulting Arrow type to carry a timezone, because the *absence* of a timezone is itself meaningful. The motivating case is Apache DataFusion Comet (a Spark accelerator) trying to enforce [SPARK-36182\: pre-Spark-4 Spark rejects reading a Parquet TimestampLTZ column as TimestampNTZ](https://issues.apache.org/jira/browse/SPARK-36182). Comet's schema adapter pattern-matches `Timestamp(_, Some(_)) -> Timestamp(_, None)` to detect this case, but for INT96 columns the post-coerce type is `Timestamp(unit, None)` — indistinguishable from a true TimestampNTZ source. The LTZ signal is destroyed at the wrong layer. Spark and other systems write INT96 as UTC-adjusted instants, so a caller can ask for the column to surface as `Timestamp(unit, Some(\"UTC\"))`, preserving the LTZ semantic at the Arrow level. ## What changes are included in this PR? - New `TableParquetOptions.global.coerce_int96_tz: Option<String>` config field (defaults to `None`). - `coerce_int96_to_resolution` gains a `timezone: Option<Arc<str>>` parameter and threads it into the constructed `Timestamp` type. - The new option is plumbed through `ParquetSource` -> `ParquetOpener` / `ParquetMorselizer` -> `DFParquetMetadata`. - `with_coerce_int96_tz` builder method on `DFParquetMetadata`. - Default behavior is unchanged when the option is unset. ## Are these changes tested? Yes, see https://github.com/apache/datafusion-comet/pull/4357 ## Are there any user-facing changes? A new \`coerce_int96_tz\` config option. No change in behavior for the default value. --------- Co-authored-by: Oleks V <comphead@users.noreply.github.com>

2026-05-17

proto: serialize dynamic filters on Sort, Aggregate, HashJoin plan nodes (#22011) ## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> - Closes https://github.com/apache/datafusion/issues/20418 (Looks like this was accidentally closed early) - Informs: https://github.com/apache/datafusion/issues/21207#issuecomment-4254968115 ## Rationale for this change `SortExec`, `AggregateExec`, and `HashJoinExec` do not serialize their dynamic filters, so plans lose dynamic filtering when they are serialized and sent across network boundaries. ## What changes are included in this PR? This change adds `with_dynamic_filter_expr()` and `dynamic_filter_expr()` to `SortExec`, `AggregateExec`, and `HashJoinExec`. ``` pub fn with_dynamic_filter_expr( mut self, filter: Arc<DynamicFilterPhysicalExpr>, ) -> Result<Self> pub fn dynamic_filter_expr(&self) -> Option<&Arc<DynamicFilterPhysicalExpr>> { ``` This are used as getters and setters for the `proto` crate to get and set dynamic filters. ## Are these changes tested? Yes. See `datafusion/datafusion/proto/tests/cases/roundtrip_physical_plan.rs`. There are also tests for the plan nodes in the `physical-plan` crate. ## Are there any user-facing changes? `SortExec`, `AggregateExec`, and `HashJoinExec` now roundtrip serialize dynamic filter expressions. --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

2026-05-07

feat: Add Protobuf support for Explain node (#21994) ## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> - Closes #. ## Rationale for this change <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> `EXPLAIN FORMAT TREE` is supported in logical plans, but protobuf serialization did not preserve the explain format. In Datafusion Ballista, we need the format field to generate corresponding distributed plan. https://github.com/apache/datafusion-ballista/issues/1627#issuecomment-4355988101 ## What changes are included in this PR? <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> - Add `ExplainFormat` to protobuf common definitions. - Add the `format` field to protobuf `ExplainNode`. - Regenerate protobuf code. ## Are these changes tested? Yes, we add a roundtrip test for `EXPLAIN FORMAT TREE`. <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> ## Are there any user-facing changes? No <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> <!-- If there are any breaking changes to public APIs, please add the `api change` label. --> --------- Co-authored-by: Kumar Ujjawal <ujjawalpathak6@gmail.com>

2026-05-01

proto: serialize and dedupe dynamic filters v2 (#21807) ## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> Informs: https://github.com/datafusion-contrib/datafusion-distributed/issues/180 Closes: https://github.com/apache/datafusion/issues/20418 ## Rationale for this change <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> Consider you have a plan with a `HashJoinExec` and `DataSourceExec` ``` HashJoinExec(dynamic_filter_1 on a@0) (...left side of join) ProjectionExec(a := Column("a", source_index)) DataSourceExec ParquetSource(predicate = dynamic_filter_2) ``` You serialize the plan, deserialize it, and execute it. What should happen is that the dynamic filter should "work", meaning: 1. When you deserialize the plan, both the `HashJoinExec` and `DataSourceExec` should have pointers to the same `DynamicFilterPhysicalExpr` 2. The `DynamicFilterPhysicalExpr` should be updated during execution by the `HashJoinExec` and the `DataSourceExec` should filter out rows This does not happen today for a few reasons, a couple of which this PR aims to address 1. `DynamicFilterPhysicalExpr` is not survive round-tripping. The internal exprs get inlined (ex. it may be serialized as `Literal`) due to the `PhysicalExpr::snapshot()` API 2. Even if `DynamicFilterPhysicalExpr` survives round-tripping, the one pushed down to the `DataSourceExec` often has different children. In this case, you have two `DynamicFilterPhysicalExpr` which do not survive deduping, causing referential integrity to be lost. ## What changes are included in this PR? <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> This PR aims to fix those problems by: 1. Removing the `snapshot()` call from the serialization process 2. Adding protos for `DynamicFilterPhysicalExpr` so it can be serialized and deserialized 3. Removing `Arc`-based deduplication. We now only dedupe on `expression_id` if the `PhysicalExpr` reports a `expression_id`. After this change, only `DynamicFilterPhysicalExpr` reports an `expression_id` to be deduped. 4. `expression_id` is now just a random u64. Since a given query likely only has a few `DynamicFilterPhysicalExpr` instances, the odds of a collision are very low 5. There's no need for a `DedupingSerializer` anymore since the `expression_id` is already stored in the dynamic filter proto itself Future work: 1. Serialize dynamic filters in `HashJoinExec`, `AggregateExec` and `SortExec` 2. Add tests which actually execute plans after deserialization and assert that dynamic filtering is functional 3. Add proto converters to the `PhysicalExtensionCodec` trait so implementors can utilize deduping logic ## Are these changes tested? - adds tests which roundtrip dynamic filters and assert that referential integrity is maintained - removes tests that test `Arc`-based deduplication and session id rotation since we don't support that anymore ## Are there any user-facing changes? - The default codec does not call `snapshot()` on `PhysicalExpr` during serialization anymore. This means that `DynamicFilterPhysicalExpr` are now serialized and deserialized without snapshotting. - All `PhysicalExpr` are not deduped anymore. Only `DynamicFilterPhysicalExpr` is --------- Co-authored-by: Dmitrii Blaginin <dmitrii@blaginin.me>

2026-04-27

Add protobuf serialization/deserialization support for `EmptyTable` scans (#20844) ## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> I figured it will be easier to submit PR right away as change doesn't look controversial. I'm happy to create an issue and link it here if you'd prefer. ## Rationale for this change <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> So short story is: in another project we'd like to use DataFusion's to "build" operations on data and then submit resulting logical plan _somewhere_ to execute (likely not using DF to actually execute the query). Since those plans never meant to be executed by DF we use `EmptyTable` as a base to bring schema to DF without any actual data. `EmptyTable` scans not being serializable prevents us from sending those plans to Python or over the wire. I believe this change makes datafusion's LogicalPlan more portable and more usable outside of datafusion's query executor. Longer story: [VegaFusion](https://github.com/vega/vegafusion) does server-side aggregation for Vega charts and is powered by DataFusion. We recently added option to [use custom query/plan executors](https://github.com/vega/vegafusion/pull/573), which allows user to pass a schema (without data) to VegaFusion which will add all necessary aggregations (but not execute them) and return a logical plan to user. They can then outsource this plan to custom query executor (e.g. Spark). This is already implemented and works. However, since VegaFusion is most commonly used through Python bindings, we'd like to expose this API to Python too (and additionally as part of gPRC API too) , which requires serializing built plans to protobuf. Currently we use `EmptyTable` to bring schema without any data to DataFusion. But since it can't be converted to protobuf, we're unable to expose this API. We considered providing custom decoder/encoder, but that would work only for gRPC case, but not Python as datafusion-python doesn't allow to provide custom decoder as far as I understand. ## What changes are included in this PR? <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> * Moved `EmptyTable` from `datafusion-core` into `datafusion-catalog` and added backwards compatibility re-export (following pattern for other table providers moved earlier) * Added new `EmptyTableScanNode` to protobuf definitions * Added encoding and decoding for new entity into `AsLogicalPlan for LogicalPlanNode` implementation ## Are these changes tested? <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> I added two roundtrip tests for the new node ## Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> `EmptyTable` can be imported from `datafusion-catalog` crate now, but old crate (`datafusion-core`) still re-exports it, so this shouldn't be breaking change <!-- If there are any breaking changes to public APIs, please add the `api change` label. --> P.S. Just to be explicit, code itself was written mostly by LLM (as I'm not that proficient in Rust yet). I did review and test it though

perf: Implement physical execution of uncorrelated scalar subqueries (#21240) ## Which issue does this PR close? - Closes #3781. - Closes #18181. ## Rationale for this change Previously, DataFusion evaluated uncorrelated scalar subqueries by transforming them into joins. This has three shortcomings: 1. Scalar subqueries that return > 1 row were allowed, producing incorrect query results. Such queries should instead result in a runtime error. 2. Performance. Evaluating scalar subqueries as a join requires going through the join machinery. More importantly, it means that UDFs that have specialized handling of scalar inputs cannot use those code paths for scalar subqueries, which often results in significantly slower query execution (e.g., #18181). It also makes filter pushdown for scalar subquery filters more difficult (#21324) 3. Uncorrelated scalar subqueries previously did not work in `ORDER BY` or `JOIN ON`, or as arguments to an aggregate function. Those cases are now supported. This PR introduces physical execution of uncorrelated scalar subqueries: * Uncorrelated subqueries are left in the plan by the optimizer, not rewritten into joins * The physical planner collects uncorrelated scalar subqueries and plans them recursively (supporting nested subqueries). We add a `ScalarSubqueryExec` plan node to the top of any physical plan with uncorrelated subqueries: it has N+1 children, N subqueries and its "main" input, which is the rest of the query plan. The subquery expression in the parent plan is replaced with a `ScalarSubqueryExpr`. * `ScalarSubqueryExec` manages the execution of the subqueries. Subquery evaluation is done in parallel (for a given query level), but at present it happens strictly before evaluation of the parent query. This might be improved in the future (#21591). * `ScalarSubqueryExpr` reads its value from a shared slot that `ScalarSubqueryExec` populates when the subquery finishes; the physical planner assigns each subquery its slot index via `ExecutionProps`. This architecture makes it easy to avoid the shortcomings described above. Performance seems roughly unchanged (benchmarks added in this PR), but in situations like #18181, we can now leverage scalar fast-paths; in the case of #18181 specifically, this improves performance from ~800 ms to ~30 ms. ## What changes are included in this PR? * Modify subquery rewriter to not transform subqueries -> joins * Collect and plan uncorrelated scalar subqueries in the physical planner, and wire up `ScalarSubqueryExpr` * Support for subqueries in physical plan serialization/deserialization using `PhysicalProtoConverterExtension` to wire up `ScalarSubqueryExpr` correctly * Support for subqueries in logical plan serialization/deserialization * Add various SLT tests and update expected plan shapes for some tests ## Are these changes tested? Yes. New SLT coverage for cardinality errors, `ORDER BY` / `JOIN ON` / aggregate-arg contexts, nested uncorrelated subqueries, duplicate-subquery deduplication, and partition-pruning filters; new roundtrip tests for logical and physical plan serialization. ## Are there any user-facing changes? SQL: * Uncorrelated scalar subqueries that return more than one row now result in a runtime error, instead of silently producing incorrect results. * Uncorrelated scalar subqueries now work in `ORDER BY`, `JOIN ON`, and as aggregate function arguments. Rust APIs: * In `datafusion-proto`, breaking changes to `Serializeable::from_bytes_with_registry` (renamed to `from_bytes_with_ctx`), `parse_expr` / `parse_sorts` / `parse_exprs`, and the `PhysicalProtoConverterExtension` trait. Plan shape: * `LogicalPlan::Subquery` nodes will now be preserved in the logical plan * Physical plans can now contain `ScalarSubqueryExec` plan node and `ScalarSubqueryExpr` expressions The wire format has also changed to include scalar subqueries. --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>

2026-04-24

Add quote style and trimming to csv writier (#20813) ## Which issue does this PR close? - Closes https://github.com/apache/datafusion/issues/10669 Related arrow-rs PRs https://github.com/apache/arrow-rs/pull/8960 and https://github.com/apache/arrow-rs/pull/9004 ## Rationale for this change The CSV writer was missing support for `quote_style`, `ignore_leading_whitespace`, and `ignore_trailing_whitespace` options that are available on the underlying arrow `WriterBuilder`. This meant users couldn't control quoting behaviour or whitespace trimming when writing CSV files. ## What changes are included in this PR? Adds three new CSV writer options wired through the full stack: - `quote_style` — controls when fields are quoted (`Always`, `Necessary`, `NonNumeric`, `Never`). Modelled as a protobuf enum (`CsvQuoteStyle`). - `ignore_leading_whitespace` — trims leading whitespace from string values on write. - `ignore_trailing_whitespace` — trims trailing whitespace from string values on write. ## Are these changes tested? Yes — sqllogictest coverage added in `csv_files.slt` ## Are there any user-facing changes? Three new `format.*` options available in COPY TO and CREATE EXTERNAL TABLE for CSV: - `format.quote_style` (string: `Always`, `Necessary`, `NonNumeric`, `Never`) - `format.ignore_leading_whitespace` (boolean) - `format.ignore_trailing_whitespace` (boolean)

2026-04-22

feat: support `ListView` and `LargeListView` in `ScalarValue` (#21669) ## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> - Closes #18886 - Previous iteration: #18884 ## Rationale for this change <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> More support for listview types in the codebase ## What changes are included in this PR? <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> Added `ListView` and `LargeListView` to `ScalarValue` with all accompanying changes Support `ListView` and `LargeListView` in proto, both for the arrow datatype & the newly introduced scalarvalue variants. ## Are these changes tested? <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> Yes, added tests ## Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> No <!-- If there are any breaking changes to public APIs, please add the `api change` label. --> --------- Co-authored-by: Khanh Duong <dqkqdlot@gmail.com> Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>

2026-04-17

fix: linearized operands in physical binaryexpr protobuf to avoid recursion limit (#21031) ## Which issue does this PR close? - part of #18602. ## Rationale for this change When a SQL query contains many filter conditions (e.g., 40+ `AND`/`OR` clauses in a `WHERE`), serializing the physical plan to protobuf and deserializing it fails with `DecodeError: recursion limit reached`. [This is because prost has a default recursion limit of 100](https://docs.rs/prost/latest/src/prost/lib.rs.html#30), and each `BinaryExpr` nesting consumes ~2 levels of protobuf recursion depth, so a chain of ~50 AND conditions exceeds the limit. ## What changes are included in this PR? Applied the same **linearization** approach that [logical expressions already use](https://github.com/apache/datafusion/blob/b6b542e87b84f4744096106bea0de755b2e70cc5/datafusion/proto/src/logical_plan/to_proto.rs#L228-L256) that convert a left-deep tree to linearization list. Instead of encoding a chain of same-operator binary expressions as a deeply nested tree, we flatten it into a flat `operands` list: **Before (nested, O(n) recursion depth):** ``` BinaryExpr(AND) { l: BinaryExpr(AND) { l: BinaryExpr(AND) { l: a, r: b }, r: c }, r: d } ``` **After (flat, O(1) recursion depth for the chain):** ``` BinaryExpr(AND) { operands: [a, b, c, d] } ``` ## Are these changes tested? yes, add some test case ## Are there any user-facing changes?

2026-04-06

feat: add support for parquet content defined chunking options (#21110) ## Rationale for this change - closes https://github.com/apache/datafusion/pull/21110 Expose the new Content-Defined Chunking feature from parquet-rs https://github.com/apache/arrow-rs/pull/9450 ## What changes are included in this PR? New parquet writer options for enabling CDC. ## Are these changes tested? In-progress. ## Are there any user-facing changes? New config options. Depends on the 58.1 arrow-rs release.

2026-03-28

Add metric category filtering for EXPLAIN ANALYZE (#21160) ## Summary - Adds `MetricCategory` enum (`Rows`, `Bytes`, `Timing`) classifying metrics by what they measure and, critically, their **determinism**: rows/bytes are deterministic given the same plan+data; timing varies across runs. - Each `Metric` can now declare its category via `MetricBuilder::with_category()`. Well-known builder methods (`output_rows`, `elapsed_compute`, `output_bytes`, etc.) set the category automatically. Custom counters/gauges default to "always included". - New session config `datafusion.explain.analyze_categories` accepts `all` (default), `none`, or comma-separated `rows`, `bytes`, `timing`. - This is orthogonal to the existing `analyze_level` (summary/dev) which controls verbosity. ## Motivation Running `EXPLAIN ANALYZE` in `.slt` tests currently requires liberal use of `<slt:ignore>` for every non-deterministic timing metric. With this change, a test can simply: ```sql SET datafusion.explain.analyze_categories = 'rows'; EXPLAIN ANALYZE SELECT ...; -- output contains only row-count metrics — fully deterministic, no <slt:ignore> needed ``` In particular, for dynamic filters we have relatively complex integration tests that exist mostly to assert the plan shapes and state of the dynamic filters after the plan has been executed. For example #21059. With this change I think most of those can be moved to SLT tests. I've also wanted to e.g. make assertions about pruning effectiveness without having timing information included. ## Test plan - [x] New Rust integration test `explain_analyze_categories` covering all combos (rows, none, all, rows+bytes) - [x] New `.slt` tests in `explain_analyze.slt` for `rows`, `none`, `rows,bytes`, and `rows` with dev level - [x] Existing `explain_analyze` integration tests pass (24/24) - [x] Proto roundtrip test updated and passing - [x] `information_schema` slt updated for new config entry - [x] Full `core_integration` suite passes (918 tests) 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

2026-03-17

Replace ahash with foldhash for faster hashing in datafusion-common (#20958) ## Summary - Replace `ahash` with `foldhash`, hashing (`with_hashes`/`create_hashes`) - seems to auto-vectorize much better as it doesn't rely on special instructions. - Use `SeedableRandomState` for rehash paths: fold existing hash into hasher's initial state, eliminating the separate `combine_hashes` step - Add `hash_write` method to `HashValue` trait for writing values into an existing hasher - Use `valid_indices()` iterator for null paths instead of per-element `is_null()` checks - Update some code to be deterministic. Notably `RandomState::default()` now does create random seed every instance, with hash it just reused a single one. Also some group by results changed as the hash function is different, added rowsort. ## Benchmark results (int64, 8192 rows, Apple M1) | Benchmark | Before (ahash) | After (foldhash) | Improvement | |---|---|---|---| | single array, no nulls | 5.65 µs | 3.30 µs | **-42%** | | multiple arrays, no nulls | 22.15 µs | 11.19 µs | **-49%** | | single array, nulls | 11.94 µs | 9.47 µs | **-21%** | | multiple arrays, nulls | 36.92 µs | 29.80 µs | **-19%** | String view improvements (utf8_view, 8192 rows): | Benchmark | Improvement | |---|---| | single, no nulls | **-13%** | | multiple, no nulls | **-28%** | | small strings, single | **-55%** | | small strings, multiple | **-60%** | ## Test plan - [x] All 36 `hash_utils` unit tests pass - [x] Run full CI suite In some hash-heavy benchmarks (clickbench_extended) we clearly see that foldhash is faster! ``` │ QQuery 1 │ 227.74 / 228.63 ±0.76 / 229.71 ms │ 205.48 / 207.44 ±1.72 / 210.35 ms │ +1.10x faster │ │ QQuery 2 │ 541.63 / 543.65 ±1.11 / 544.82 ms │ 499.61 / 502.19 ±1.76 / 504.94 ms │ +1.08x faster │ │ QQuery 3 │ 334.85 / 336.04 ±1.16 / 337.89 ms │ 316.43 / 317.67 ±1.02 / 319.48 ms │ +1.06x faster │ ``` # Are there any user-facing changes? Yes `RandomState::with_seeds` was replaced as `RandomState::with_seeds` and there were protobuf changes for the same function. Also the function will generate different hashes, so in distributed environments it shouldn't use different versions of binaries to run the same query. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

2026-03-12

[branch-53] ser/de fetch in FilterExec (#20738) (#20883) - Part of https://github.com/apache/datafusion/issues/19692 - Closes https://github.com/apache/datafusion/issues/20737 on branch-53 This PR: - Backports https://github.com/apache/datafusion/pull/20738 from @haohuaijin to the branch-53 line Co-authored-by: Huaijin <haohuaijin@gmail.com>

2026-03-11

impl ser/de for preserve_order in RepartitionExec (#20798) ## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> - Closes #20797 ## Rationale for this change - see #20797 ## What changes are included in this PR? impl ser/de for preserve_order in RepartitionExec ## Are these changes tested? add one test case ## Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> <!-- If there are any breaking changes to public APIs, please add the `api change` label. -->

2026-03-09

feat: make DefaultLogicalExtensionCodec support serialisation of buil… (#20638) ## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> - Closes #16944. ## Rationale for this change Currently, the `LogicalExtensionCodec` implementation for `DefaultLogicalExtensionCodec` leaves `try_decode_file_format` / `try_encode_file_format` unimplemented (returning "not implemented" errors). However, the actual serialization logic for built-in file formats — arrow, parquet, csv, and json— already exists in their respective codec implementations. All we need to do is tag which format is being used, and delegate to the corresponding format-specific codec to handle the data. <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> ## What changes are included in this PR? Added a `FileFormatKind` enum and a `FileFormatProto` message to `datafusion.proto` to identify the file format type during transmission. Implemented `try_decode_file_format` and `try_encode_file_format` for `DefaultLogicalExtensionCodec`, which dispatch serialization/deserialization to the corresponding format-specific codec based on the format kind. Note that Avro is not covered because the upstream repository has not yet implemented the corresponding Avro codec, so Avro support is not functional at this time. ## Are these changes tested? Yes. Roundtrip tests are included for csv, json, parquet, and arrow. <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> ## Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> <!-- If there are any breaking changes to public APIs, please add the `api change` label. -->

2026-03-08

ser/de fetch in FilterExec (#20738) ## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> - Closes https://github.com/apache/datafusion/issues/20737 ## Rationale for this change FilterExec have fetch filed but not impl the ser/de in proto ## What changes are included in this PR? add ser/de for fetch in FilterExec ## Are these changes tested? add one test case ## Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> <!-- If there are any breaking changes to public APIs, please add the `api change` label. -->

2026-03-03

Add `Field` to `Expr::Cast` -- allow logical expressions to express a cast to an extension type (#18136) ## Which issue does this PR close? - Closes #18060. I am sorry that I missed the previous PR implementing this ( https://github.com/apache/datafusion/pull/18120 ) and I'm also happy to review that one instead of updating this! ## Rationale for this change Other systems that interact with the logical plan (e.g., SQL, Substrait) can express types that are not strictly within the arrow DataType enum. ## What changes are included in this PR? For the Cast and TryCast structs, the destination data type was changed from a DataType to a FieldRef. ## Are these changes tested? Yes. ## Are there any user-facing changes? Yes, any code using `Cast { .. }` to create an expression would need to use `Cast::new()` instead (or pass on field metadata if it has it). Existing matches will need to be upated for the `data_type` -> `field` member rename. --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>

2026-02-12

Add protoc support for ArrowScanExecNode (#20280) (#20284) ## Which issue does this PR close? - Closes #20280. ## Rationale for this change Physical plans that read Arrow files (.arrow / IPC) could not be serialized or deserialized via the proto layer. PhysicalPlanNode already had scan nodes for Parquet, CSV, JSON, Avro, and in-memory sources, but not for Arrow, so a DataSourceExec using ArrowSource was not round-trippable. That blocked use cases like distributing plans that scan Arrow files (e.g. Ballista). This change adds Arrow scan to the proto layer so those plans can be serialized and deserialized like the other file formats. ## What changes are included in this PR? Proto: Added ArrowScanExecNode (with FileScanExecConf base_conf) and arrow_scan = 38 to the PhysicalPlanNode oneof in datafusion.proto. Generated code: Updated prost.rs and pbjson.rs to include ArrowScanExecNode and the ArrowScan variant (manual edits; protoc was not run). To-proto: In try_from_data_source_exec, when the data source is a FileScanConfig whose file source is ArrowSource, it is now serialized as ArrowScanExecNode. From-proto: Implemented try_into_arrow_scan_physical_plan to deserialize ArrowScanExecNode into DataSourceExec with ArrowSource; missing base_conf returns an explicit error (no .unwrap()). Test: Added roundtrip_arrow_scan in roundtrip_physical_plan.rs to assert Arrow scan plans round-trip correctly. ## Are these changes tested? Yes. A new test roundtrip_arrow_scan builds a physical plan that scans Arrow files, serializes it to bytes and deserializes it back, and asserts the round-tripped plan matches the original. The full cargo test -p datafusion-proto suite (150 tests: unit, integration, and doc tests) passes, including all existing roundtrip and serialization tests. ## Are there any user-facing changes? No. This only extends the existing physical-plan proto support to Arrow scan. Callers that already serialize/deserialize physical plans (e.g. for distributed execution) can now round-trip plans that read Arrow files in addition to Parquet, CSV, JSON, and Avro, with no API or behavioral changes for existing usage.

2026-02-11

Support JSON arrays reader/parse for datafusion (#19924) ## Which issue does this PR close? Closes #19920 ## Rationale for this change DataFusion currently only supports line-delimited JSON (NDJSON) format. Many data sources provide JSON in array format `[{...}, {...}]`, which cannot be parsed by the existing implementation. ## What changes are included in this PR? - Add `newline_delimited` option to `JsonOptions` (default `true` for backward compatibility) - Implement streaming JSON array to NDJSON conversion via `JsonArrayToNdjsonReader` - Support both file-based and stream-based (e.g., S3) reading with memory-efficient streaming - Add `ChannelReader` for async-to-sync byte transfer in object store streaming scenarios - Add protobuf serialization support for the new option - Rename `NdJsonReadOptions` to `JsonReadOptions` (with deprecation alias) - SQL support via `OPTIONS ('format.newline_delimited' 'false')` ### Architecture ```text JSON Array File (e.g., 33GB) │ ▼ read chunks via ChannelReader (for streams) or BufReader (for files) ┌───────────────────┐ │ JsonArrayToNdjson │ ← streaming character substitution: │ Reader │ '[' skip, ',' → '\n', ']' stop └───────────────────┘ │ ▼ outputs NDJSON format ┌───────────────────┐ │ Arrow Reader │ ← batch parsing └───────────────────┘ │ ▼ RecordBatch ``` ### Memory Efficiency | Approach | Memory for 33GB file | Parse count | |----------|---------------------|-------------| | Load entire file + serde_json | ~100GB+ | 3x | | Streaming with JsonArrayToNdjsonReader | ~32MB | 1x | ## Are these changes tested? Yes: - Unit tests for `JsonArrayToNdjsonReader` (nested objects, escaped strings, empty arrays, buffer boundaries) - Unit tests for `ChannelReader` - Integration tests for `JsonOpener` (file-based, stream-based, large files, cancellation) - Schema inference tests (normal, empty, nested struct, list types) - End-to-end query tests with SQL - SQLLogicTest for SQL validation ## Are there any user-facing changes? Yes. Users can now read JSON array format files: **Via SQL:** ```sql CREATE EXTERNAL TABLE my_table STORED AS JSON OPTIONS ('format.newline_delimited' 'false') LOCATION 'path/to/array.json'; ``` **Via API:** ```rust let options = JsonReadOptions::default().newline_delimited(false); ctx.register_json("my_table", "path/to/array.json", options).await?; ``` **Note:** `NdJsonReadOptions` is deprecated in favor of `JsonReadOptions`. **Limitation:** JSON array format does not support range-based file scanning (`repartition_file_scans`). Users will see a clear error message if this is attempted.

2026-02-03

Add BufferExec execution plan (#19760) ## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> - Closes #. ## Rationale for this change This is a PR from a batch of PRs that attempt to improve performance in hash joins: - https://github.com/apache/datafusion/pull/19759 - This PR - https://github.com/apache/datafusion/pull/19761 It adds a building block that allows eagerly collecting data on the probe side of a hash join before the build side is finished. Even if the intended use case is for hash joins, the new execution node is generic and is designed to work anywhere in the plan. ## What changes are included in this PR? > [!NOTE] > The new BufferExec node introduced in this PR is still not wired up automatically <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> Adds a new `BufferExec` node that can buffer up to a certain size in bytes for each partition eagerly performing work that otherwise would be delayed. Schematically, it looks like this: ``` ┌───────────────────────────┐ │ BufferExec │ │ │ │┌────── Partition 0 ──────┐│ ││ ┌────┐ ┌────┐││ ┌────┐ ──background poll────────▶│ │ │ ├┼┼───────▶ │ ││ └────┘ └────┘││ └────┘ │└─────────────────────────┘│ │┌────── Partition 1 ──────┐│ ││ ┌────┐ ┌────┐ ┌────┐││ ┌────┐ ──background poll─▶│ │ │ │ │ ├┼┼───────▶ │ ││ └────┘ └────┘ └────┘││ └────┘ │└─────────────────────────┘│ │ │ │ ... │ │ │ │┌────── Partition N ──────┐│ ││ ┌────┐││ ┌────┐ ──background poll───────────────▶│ ├┼┼───────▶ │ ││ └────┘││ └────┘ │└─────────────────────────┘│ └───────────────────────────┘ ``` ## Are these changes tested? <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> yes, by new unit tests ## Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> users can import a new `BufferExec` execution plan in their codebase, but no internal usage is shipped yet in this PR. <!-- If there are any breaking changes to public APIs, please add the `api change` label. -->

2026-02-02

Add `ScalarValue::RunEndEncoded` variant (#19895) ## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> - Closes #18563 ## Rationale for this change <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> Support RunEndEncoded scalar values, similar to how we support for Dictionary. ## What changes are included in this PR? <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> - Add new `ScalarValue::RunEndEncoded` enum variant - Fix `ScalarValue::new_default` to support `Decimal32` and `Decimal64` - Support RunEndEncoded type in proto for both `ScalarValue` message and `ArrowType` message ## Are these changes tested? <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> Added tests. ## Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> New variant for `ScalarValue` Protobuf changes to support RunEndEncoded type <!-- If there are any breaking changes to public APIs, please add the `api change` label. --> --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>

Preserve PhysicalExpr graph in proto round trip using Arc pointers as unique identifiers (#20037) Replaces #18192 using the APIs in #19437. Similar to #18192 the end goal here is specifically to enable deduplication of `DynamicFilterPhysicalExpr` so that distributed query engines can get one step closer to using dynamic filters. Because it's actually simpler we apply this deduplication to all `PhysicalExpr`s with the added benefit that we more faithfully preserve the original expression tree (instead of adding new duplicate branches) which will have the immediate impact of e.g. not duplicating large `InListExpr`s.

2026-01-30

feat: add AggregateMode::PartialReduce for tree-reduce aggregation (#20019) DataFusion's current `AggregateMode` enum has four variants covering three of the four cells in the input/output matrix: | | Input: raw data | Input: partial state | | - | - | - | | Output: final values | `Single` / `SinglePartitioned` | `Final` / `FinalPartitioned` | | Output: partial state | `Partial` | ??? | This PR adds `AggregateMode::PartialReduce` to fill in the missing cell: it takes partially-reduced values as input, and reduces them further, but without finalizing. This is useful because it's the key component needed to implement distributed tree-reduction (as seen in e.g. the Scuba or Honeycomb papers): a set of worker nodes each perform multithreaded `Partial` aggregations, feed those into a `PartialReduce` to reduce all of this node's values into a single row, and then a head node collects the outputs from all nodes' `PartialReduce` to feed into a `Final` reduction. PR can be reviewed commit by commit: first commit is pure refactor/simplification; most places we were matching on `AggregateMode` we were actually just trying to either check which row of the above table we were in, or else which column. So now we have `is_first_stage` (tells you which column) and `is_last_stage` (tells you which row) and we use them everywhere. Second commit adds `PartialReduce`, and is pretty small because `is_first_stage`/`is_last_stage` do most of the heavy lifting. It also adds a test demonstrating a minimal Partial -> PartialReduce -> Final tree-reduction.

2026-01-28

fix: respect DataFrameWriteOptions::with_single_file_output for paths without extensions (#19931) ## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> - Closes #13323. ## Rationale for this change When using `DataFrameWriteOptions::with_single_file_output(true)`, the setting was being ignored if the output path didn't have a file extension. For example: ```rust df.write_parquet("/path/to/output", DataFrameWriteOptions::new().with_single_file_output(true), None).await?; ``` Would create a directory /path/to/output/ with files inside instead of a single file at /path/to/output. This happened because the demuxer used a heuristic based solely on file extension, ignoring the explicit user setting. <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> ## What changes are included in this PR? - New FileOutputMode enum: Uses explicit modes (Automatic, SingleFile, Directory) in FileSinkConfig for clearer output path handling. - The demuxer now uses the user's explicit setting instead of always relying on extension-based heuristics. <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> ## Are these changes tested? - New unit test test_single_file_output_without_extension tests the fixed behavior - All sqllogictest pass <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> ## Are there any user-facing changes? Breaking for direct FileSinkConfig construction: The struct now requires file_output_mode: FileOutputMode field. Use FileOutputMode::Automatic to preserve existing behavior. <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> <!-- If there are any breaking changes to public APIs, please add the `api change` label. --> --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>

2026-01-24

preserve FilterExec batch size during ser/de (#19960) ## Rationale for this change Noticed that `FilterExec` batch size is not preserved so it is set to default one after plan serialization + de-serialization. This patch fixes it. ## What changes are included in this PR? <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. -->

2026-01-21

Feat : added truncate table support (#19633) ## Which issue does this PR close? - Related to #19617 ## Rationale for this change DataFusion recently added TableProvider hooks for row-level DML operations such as DELETE and UPDATE, but TRUNCATE TABLE was still unsupported. ## What changes are included in this PR? This PR adds planning and integration support for TRUNCATE TABLE in DataFusion, completing another part of the DML surface alongside existing DELETE and UPDATE support. Specifically, it includes: - SQL parsing support for TRUNCATE TABLE - Logical plan support via a new WriteOp::Truncate DML operation - Physical planner routing for TRUNCATE statements - A new TableProvider::truncate() hook for storage-native implementations - Protobuf / DML node support for serializing and deserializing TRUNCATE operations - SQL logic tests validating logical and physical planning behavior The implementation follows the same structure and conventions as the existing DELETE and UPDATE DML support. Execution semantics are delegated to individual TableProvider implementations via the new hook. ## Are these changes tested? Yes. The PR includes: SQL logic tests that verify: - Parsing of TRUNCATE TABLE - Correct logical plan generation - Correct physical planner routing - Clear and consistent errors for providers that do not yet support TRUNCATE These tests mirror the existing testing strategy used for unsupported DELETE and UPDATE operations. ## Are there any user-facing changes? Yes. Users can now execute TRUNCATE TABLE statements in DataFusion for tables whose TableProvider supports the new truncate() hook. Tables that do not support TRUNCATE will return a clear NotImplemented error.

2026-01-18

feat: support `SELECT DISTINCT id FROM t ORDER BY id LIMIT n` query use GroupedTopKAggregateStream (#19653) ## Which issue does this PR close? - close https://github.com/apache/datafusion/issues/19638 ## Rationale for this change see issue #19638 ## What changes are included in this PR? 1. Introduced `LimitOptions` struct limit field with both `limit` and optional `descending` ordering direction 2. Extended `TopKAggregation` optimizer rule to DISTINCT queries by recognizing `GROUP BY` queries without aggregates and setting the `descending` flag based on ordering direction 3. Enhanced `GroupedTopKAggregateStream` to handle DISTINCT by using group key as both priority queue key and value for DISTINCT operations 4. Updated Proto definitions to add optional `descending` field to `AggLimit` message for serialization/deserialization ## benchmark result <img width="731" height="475" alt="image" src="https://github.com/user-attachments/assets/05b6eb8c-186d-4b17-84a9-a2897dbcb095" /> ## Are these changes tested? yes, add test case in aggregates_topk.slt ## Are there any user-facing changes? no

2026-01-13

feat: Add null-aware anti join support (#19635) ## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> - Closes #10583. ## Rationale for this change <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> ## What changes are included in this PR? <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> This patch implements null-aware anti join support for HashJoin LeftAnti operations, enabling correct SQL NOT IN subquery semantics with NULL values. ## Are these changes tested? <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> ## Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> <!-- If there are any breaking changes to public APIs, please add the `api change` label. --> --------- Co-authored-by: Claude Sonnet 4.5 <noreply@anthropic.com>

2025-12-19

feat(proto): Add protobuf serialization for HashExpr (#19379) ## Summary This PR adds protobuf serialization/deserialization support for `HashExpr`, enabling distributed query execution to serialize hash expressions used in hash joins and repartitioning. This is a followup to #18393 which introduced `HashExpr` but did not add serialization support. This causes errors when serialization is triggered on a query that pushes down dynamic filters from a `HashJoinExec`. As of #18393 `HashJoinExec` produces filters of the form: ```sql CASE (hash_repartition % 2) WHEN 0 THEN a >= ab AND a <= ab AND b >= bb AND b <= bb AND hash_lookup(a,b) WHEN 1 THEN a >= aa AND a <= aa AND b >= ba AND b <= ba AND hash_lookup(a,b) ELSE FALSE END ``` Where `hash_lookup` is an expression that holds a reference to a given partitions hash join hash table and will check for membership. Since we created these new expressions but didn't make any of them serializable any attempt to do a distributed query or similar would run into errors. In https://github.com/apache/datafusion/pull/19300 we fixed `hash_lookup` by replacing it with `true` since it can't be serialized across the wire (we'd have to send the entire hash table). The logic was that this preserves the bounds checks, which as still valuable. This PR handles `hash_repartition` which determines which partition (and hence which branch of the `CASE` expression) the row belongs to. For this expression we *can* serialize it, so that's what I'm doing in this PR. ### Key Changes - **SeededRandomState wrapper**: Added a `SeededRandomState` struct that wraps `ahash::RandomState` while preserving the seeds used to create it. This is necessary because `RandomState` doesn't expose seeds after creation, but we need them for serialization. - **Updated seed constants**: Changed `HASH_JOIN_SEED` and `REPARTITION_RANDOM_STATE` constants to use `SeededRandomState` instead of raw `RandomState`. - **HashExpr enhancements**: - Changed `HashExpr` to use `SeededRandomState` - Added getter methods: `on_columns()`, `seeds()`, `description()` - Exported `HashExpr` and `SeededRandomState` from the joins module - **Protobuf support**: - Added `PhysicalHashExprNode` message to `datafusion.proto` with fields for `on_columns`, seeds (4 `u64` values), and `description` - Implemented serialization in `to_proto.rs` - Implemented deserialization in `from_proto.rs` ## Test plan - [x] Added roundtrip test in `roundtrip_physical_plan.rs` that creates a `HashExpr`, serializes it, deserializes it, and verifies the result - [x] All existing hash join tests pass (583 tests) - [x] All proto roundtrip tests pass 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>

2025-12-17

feat: add compression level configuration for JSON/CSV writers (#18954) ## Which issue does this PR close? Closes #18947 ## Rationale for this change Currently, DataFusion uses default compression levels when writing compressed JSON and CSV files. For ZSTD, this means level 3, which prioritizes speed over compression ratio. Users working with large datasets who want to optimize for storage costs or network transfer have no way to increase the compression level. This is particularly important for cloud data lake scenarios where storage and egress costs can be significant. ## What changes are included in this PR? - Add `compression_level: Option<u32>` field to `JsonOptions` and `CsvOptions` in `config.rs` - Add `convert_async_writer_with_level()` method to `FileCompressionType` (non-breaking API extension) - Keep original `convert_async_writer()` as a convenience wrapper for backward compatibility - Update `JsonWriterOptions` and `CsvWriterOptions` with `compression_level` field - Update `ObjectWriterBuilder` to support compression level - Update JSON and CSV sinks to pass compression level through the write pipeline - Update proto definitions and conversions for serialization support - Fix unrelated unused import warning in `udf.rs` (conditional compilation for debug-only imports) ## Are these changes tested? The changes follow the existing patterns used throughout the codebase. The implementation was verified by: - Building successfully with `cargo build` - Running existing tests with `cargo test --package datafusion-proto` - All 131 proto integration tests pass ## Are there any user-facing changes? Yes, users can now specify compression level when writing JSON/CSV files: ```rust use datafusion::common::config::JsonOptions; use datafusion::common::parsers::CompressionTypeVariant; let json_opts = JsonOptions { compression: CompressionTypeVariant::ZSTD, compression_level: Some(9), // Higher compression ..Default::default() }; ``` **Supported compression levels:** - ZSTD: 1-22 (default: 3) - GZIP: 0-9 (default: 6) - BZIP2: 1-9 (default: 9) - XZ: 0-9 (default: 6) **This is a non-breaking change** - the original `convert_async_writer()` method signature is preserved for backward compatibility. Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>

2025-12-16

Fix panic for `GROUPING SETS(())` and handle empty-grouping aggregates (#19252) ## Which issue does this PR close? * Closes #18974. ## Rationale for this change The DataFusion CLI currently panics with an "index out of bounds" error when executing queries that use `GROUP BY GROUPING SETS(())`, such as: ```sql SELECT SUM(v1) FROM generate_series(10) AS t1(v1) GROUP BY GROUPING SETS(()) ``` This panic originates in the physical aggregation code, which assumes that an empty list of grouping expressions always corresponds to "no grouping". That assumption breaks down in the presence of `GROUPING SETS`, where an empty set is a valid grouping set that should still produce a result row (and `__grouping_id`) rather than crashing. This PR fixes the panic by explicitly distinguishing: * true "no GROUP BY" aggregations, and * `GROUPING SETS`/`CUBE`/`ROLLUP` plans that may have empty grouping expressions but still require grouping-set semantics and a valid `__grouping_id`. The change restores robustness of the CLI and ensures standards-compliant behavior for grouping sets with empty sets. ## What changes are included in this PR? Summary of the main changes: * **Track grouping-set usage explicitly in `PhysicalGroupBy`:** * Add a `has_grouping_set: bool` field to `PhysicalGroupBy`. * Extend `PhysicalGroupBy::new` to accept the `has_grouping_set` flag. * Add helper methods: * `has_grouping_set(&self) -> bool` to expose the flag, and * `is_true_no_grouping(&self) -> bool` to represent the case of genuinely no grouping (no GROUP BY and no grouping sets). * **Correct group state construction for empty grouping with grouping sets:** * Update `PhysicalGroupBy::from_pre_group` so that it only treats `expr.is_empty()` as "no groups" when `has_grouping_set` is `false`. * For `GROUPING SETS(())`, we now build at least one group, avoiding the previous out-of-bounds access on `groups[0]`. * **Clarify when `__grouping_id` should be present:** * Replace the previous `is_single` logic with a clearer distinction based on `has_grouping_set`. * `num_output_exprs`, `output_exprs`, `num_group_exprs`, and `group_schema` now add the `__grouping_id` column only when `has_grouping_set` is `true`. * `is_single` is redefined as "simple GROUP BY" (no grouping sets), i.e. `!self.has_grouping_set`. * **Integrate the new semantics into `AggregateExec`:** * Use `group_by.is_true_no_grouping()` instead of `group_by.expr.is_empty()` when choosing between the specialized no-grouping aggregation path and grouped aggregation. * Ensure that `is_unordered_unfiltered_group_by_distinct` only treats plans as grouped when there are grouping expressions **and** no grouping sets (`!has_grouping_set`). * Preserve existing behavior for regular `GROUP BY` while correctly handling `GROUPING SETS` and related constructs. * **Support `__grouping_id` with the no-grouping aggregation stream:** * Extend `AggregateStreamInner` with an optional `grouping_id: Option<ScalarValue>` field. * Change `AggregateStream::new` to accept a `grouping_id` argument. * Introduce `prepend_grouping_id_column` to prepend a `__grouping_id` column to the finalized accumulator output when needed. * Wire this up so that no-grouping aggregations can still match a schema that includes `__grouping_id` in grouping-set scenarios. * **Planner and execution wiring updates:** * Update all `PhysicalGroupBy::new` call sites to pass the correct `has_grouping_set` value: * `false` for: * ordinary `GROUP BY` or truly no-grouping aggregates. * `true` for: * `GROUPING SETS`, * `CUBE`, and * `ROLLUP` physical planning paths. * Ensure `merge_grouping_set_physical_expr`, `create_cube_physical_expr`, and `create_rollup_physical_expr` correctly mark grouping-set plans. * **Protobuf / physical plan round-trip support:** * Extend `AggregateExecNode` in `datafusion.proto` with a new `bool has_grouping_set = 12;` field. * Update the generated `pbjson` and `prost` code to serialize and deserialize the new field. * When constructing `AggregateExec` from protobuf, pass the decoded `has_grouping_set` into `PhysicalGroupBy::new`. * When serializing an `AggregateExec` back to protobuf, set `has_grouping_set` based on `exec.group_expr().has_grouping_set()`. * Update round-trip physical plan tests to include the new field in their expectations. * **Tests and SQL logic coverage:** * Add sqllogictests for the previously failing cases in `grouping.slt`: * `SELECT COUNT(*) FROM test GROUP BY GROUPING SETS (());` * `SELECT SUM(v1) FROM generate_series(10) AS t1(v1) GROUP BY GROUPING SETS(())` (the original panic case). * Extend or adjust unit tests in `aggregates`, `physical_planner`, `filter_pushdown`, and `coop` modules to account for the `has_grouping_set` flag in `PhysicalGroupBy` and expected debug output. * Update proto round-trip tests to validate `has_grouping_set` is preserved. ## Are these changes tested? Yes. * New sqllogictests covering `GROUPING SETS(())` for both a regular table and `generate_series(10)`: * `grouping.slt` now asserts the expected scalar results (e.g. `2` and `55`), preventing regressions on this edge case. * Updated and existing Rust unit tests: * `physical-plan/src/aggregates` tests updated to include `has_grouping_set` in `PhysicalGroupBy` expectations. * Planner and optimizer tests (e.g. `physical_planner.rs`, `filter_pushdown`) updated to construct `PhysicalGroupBy` with the new flag. * Execution tests in `core/tests/execution/coop.rs` updated to reflect the new constructor and continue to exercise the no-grouping aggregation path. * Protobuf round-trip tests extended to verify that `has_grouping_set` is correctly serialized and deserialized. These tests collectively ensure that: * the panic is fixed, * the aggregation semantics for `GROUPING SETS(())` are correct, and * existing aggregate behavior remains unchanged for non-grouping-set queries. ## Are there any user-facing changes? Yes, but they are bug fixes and behavior clarifications rather than breaking changes: * Queries using `GROUP BY GROUPING SETS(())` no longer cause a runtime panic in the DataFusion CLI. * Instead, they return the expected single aggregate row (e.g. `COUNT(*)` or `SUM(v1)`), consistent with SQL semantics. * For plans using `GROUPING SETS`, `CUBE`, or `ROLLUP`, the internal `__grouping_id` column is now present consistently whenever grouping sets are in use, even when the grouping expressions are empty. * For ordinary `GROUP BY` queries that do not use grouping sets, behavior is unchanged: no unexpected `__grouping_id` column is added. No API signatures were changed in a breaking way for downstream users; the additions are internal flags and protobuf fields to accurately represent the physical plan. ## LLM-generated code disclosure This PR includes LLM-generated code and comments. All LLM-generated content has been manually reviewed and tested.

2025-12-10

Track column sizes in Statistics; propagate through projections (#19113) Closes #19098, follow up to #19094. Related to #14936

[Proto]: Serialization support for `AsyncFuncExec` (#19118) ## Which issue does this PR close? Closes #19112 ## Rationale for this change Use async functions with Ballista ## What changes are included in this PR? - New `AsyncFuncExecNode` proto definition - to/from glue - A roundtrip test ## Are these changes tested? n/a ## Are there any user-facing changes? n/a

2025-12-03

Add `force_filter_selections` to restore `pushdown_filters` behavior prior to parquet 57.1.0 upgrade (#19003) ~Draft until https://github.com/apache/datafusion/pull/18820 is merged~ ## Which issue does this PR close? - Follow on to https://github.com/apache/datafusion/pull/18820 ## Rationale for this change The parquet 57.1.0 upgrade includes a new adaptive filter from @hhhizzz : - https://github.com/apache/arrow-rs/pull/8733 Our testing shows this is faster in all cases, but I want to have an escape valve for people to turn it off if they hit some issue. I had originally included this in #18820 but @rluvaton suggested it would be easier to understand as its own PR in https://github.com/apache/datafusion/pull/18820#pullrequestreview-3509993052 ## What changes are included in this PR? 1. Add a `force_filter_selections` config setting 2. Add configuration guide 3. Add tests ## Are these changes tested? Yes ## Are there any user-facing changes? A new boolean flag

2025-11-27

move projection handling into FileSource (#18627) - Part of https://github.com/apache/datafusion/issues/14993 This moves ownership of projections from `FileScanConfig` into `FileSource`. Notably we do *not* do anything special with this in Parquet just yet: I leave it for a followup to actually use the projection expressions instead of column indices to e.g. generate the Parquet `ProjectionMask` directly from expressions (in particular to select leaves instead of roots for struct and variant access).

2025-10-24

Enable placeholders with extension types (#17986) ## Which issue does this PR close? - Closes #17862 ## Rationale for this change Most logical plan expressions now propagate metadata; however, parameters with extension types or other field metadata cannot participate in placeholder/parameter binding. ## What changes are included in this PR? The DataType in the Placeholder struct was replaced with a FieldRef along with anything that stored the "DataType" of a parameter. Strictly speaking one could bind parameters with an extension type by copy/pasting the placeholder replacer, which I figured out towards the end of this change. I still think this change makes sense and opens up the door for things like handling UUID in SQL with full parameter binding support. ## Are these changes tested? Yes ## Are there any user-facing changes? Yes, one new function was added to extract the placeholder fields from a plan. This is a breaking change for code that specifically interacts with the pub fields of the modified structs (ParamValues, Placeholder, and Prepare are the main ones). --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>

2025-10-17

feat: support `null_treatment`, `distinct`, and `filter` for window functions in proto (#18024) ## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> - Closes #17417. ## Rationale for this change <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> - Support `null_treatment`, `distinct`, and `filter` for window function in proto. - Support `null_treatment` for aggregate udf in proto. ## What changes are included in this PR? <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> - [x] Add `null_treatment`, `distinct`, `filter` fields to `WindowExprNode` message and handle them in `to/from_proto.rs`. - [x] Add `null_treatment` field to `AggregateUDFExprNode` message and handle them in `to/from_proto.rs`. - [ ] Docs update: I'm not sure where to add docs as declared in the issue description. ## Are these changes tested? <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> - Add tests to `roundtrip_window` for respectnulls, ignorenulls, distinct, filter. - Add tests to `roundtrip_aggregate_udf` for respectnulls, ignorenulls. ## Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> <!-- If there are any breaking changes to public APIs, please add the `api change` label. --> N/A --------- Co-authored-by: Jeffrey Vo <jeffrey.vo.australia@gmail.com>

2025-09-18

Upgrade to arrow 56.1.0 (#17275) * Update to arrow/parquet 56.1.0 * Adjust for new parquet sizes, update for deprecated API * Thread through max_predicate_cache_size, add test

2025-09-17

feat: Add `OR REPLACE` to creating external tables (#17580) * feat: Add `OR REPLACE` to creating external tables * regen * fmt * make more explicit + add tests * clipy fix --------- Co-authored-by: Dmitrii Blaginin <dmitrii@blaginin.me>

2025-09-09

Support csv truncated rows in datafusion (#17465)

2025-09-05

Memory datasource protobuf support (#17290) * Add proto * fix proto * gen proto code * exec to proto * gen proto code * impl (de)serialization * Add test * Update submodules * gen proto * Set parquet-testing back to main --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> Co-authored-by: Tim Saucer <timsaucer@gmail.com>

2025-08-24

feat: Support SortMergeJoin proto serde (#17296) * Implement ser/de part of SortMergeJoin * add round trip tests for sort merge join * add filter test to roundtrip

2025-08-19

Support `distinct` and `ignore_nulls` in window expressions (#17235)

2025-08-18

Support serializing `generate_series` in `datafusion-proto` (#17200) * Allow `generate_series` to be serialized via protobuf * Add breaking change to the upgrade guide

2025-08-07

fix: Remove `datafusion.execution.parquet.cache_metadata` config (#17062) * fix: Remove `datafusion.execution.parquet.cache_metadata` config * prettier * fix prettier? * fix * fix config * fix test behaviour * fix

2025-08-04

Upgrade arrow/parquet to 56.0.0 (#16690)

2025-08-02

feat: Cache Parquet metadata in built in parquet reader (#16971) * feat: Cache Parquet metadata * Convert FileMetadata and FileMetadataCache to traits * Use as_any to respect MSRV * Use ObjectMeta as the key of FileMetadataCache

2025-07-26

fix(datafusion-proto): support serializing/deserilizing ArrowFormat tables (#16875) Fixes #16874

2025-07-25

Fixes 3 bugs during serialization and deserialization of physical plans (#16858)

2025-06-20

Use Tokio's task budget consistently, better APIs to support task cancellation (#16398) * Use Tokio's task budget consistently * Rework `ensure_coop` to base itself on evaluation and scheduling properties * Iterating on documentation * Improve robustness of cooperative yielding test cases * Reorganize tests by operator a bit better * Coop documentation * More coop documentation * Avoid Box in temporary CooperativeStream::poll_next implementation * Adapt interleave test cases for range generator * Add temporary `tokio_coop` feature to unblock merging * Extract magic number to constant * Fix documentation error * Push scheduling type down from DataSourceExec to DataSource * Use custom configuration instead of feature to avoid exposing internal cooperation variants * Use dedicated enum for yield results * Documentation improvements from review * More documentation * Change default coop strategy to 'tokio_fallback' * Documentation refinement * Re-enable interleave test cases * fix logical merge conflict --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>

2025-06-17

Use dedicated NullEquality enum instead of null_equals_null boolean (#16419) * Use dedicated NullEquality enum instead of null_equals_null boolean * Fix wrong operator mapping in hash_join * Add an example to the documentation

2025-06-16

feat: Support RightMark join for NestedLoop and Hash join (#16083) * feat: Support RightMark join for NestedLoop and Hash join * fixes * producer fix * fmt * update * fix * rem file * fix * fmt * Update datafusion/physical-plan/src/joins/utils.rs Co-authored-by: Christian <9384305+ctsk@users.noreply.github.com> * fixes * clippy * refactor --------- Co-authored-by: Christian <9384305+ctsk@users.noreply.github.com> Co-authored-by: Oleks V <comphead@users.noreply.github.com>

2025-06-09

feat: Allow cancelling of grouping operations which are CPU bound (#16196) * feat: support inability to yeild cpu for loop when it's not using Tokio MPSC (RecordBatchReceiverStream) * Fix fuzz test * polish code * add comments * fix corner case when huge data * Also add grouping case * Address comments * fmt * Move YieldStream into physical-plan crate * Use existing RecordBatchStreamAdapter * Add timeout testing for cancellation * fmt * add license * Support sort exec for cancellation * poc: unified yield exec for leaf node * polish code phase 1 * Add license * Fix testing * Support final path * fix test * polish code * fix testing and address suggestions * fix * remove buffer * address comments * fmt * Fix test * fix * fix * fix slt * fix tpch sql * Add flag for yield insert and disable default * recover testing * fix * Update doc * Address comments * fix fmt * Support config for yield frequency * add built-in yield support * Add LazyMemoryExec built-in Yield * Update datafusion/datasource/src/source.rs Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com> * Update datafusion/core/tests/physical_optimizer/enforce_distribution.rs Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com> * Update datafusion/physical-optimizer/src/optimizer.rs Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com> * Update datafusion/physical-plan/src/memory.rs Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com> * Update datafusion/physical-plan/src/memory.rs Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com> * Update datafusion/proto/src/physical_plan/mod.rs Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com> * Address comments * Add interleave reproducer * remove unused config * Add join with aggr case * fix clippy * add reproducer for filter and add workaround in rule * Add reproducer and comment it * adjust test * Harden tests, add failing tests for TDD, minor code refactors * Add remaining test * Add sort merge case * change rule solution * fix test * add more built-in case * fix * fix user defined exec * Reduce test diff * Format imports * Update documentation * Update information schema per docs * Only retain the with_cooperative_yields API for now * Format imports * Remove unnecessary clones * Fix logical conflict * Exercise DRY, refactor common code to a helper function. Use period instead of frequency for config parameter terminology * Update datafusion/physical-plan/src/yield_stream.rs Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> * Address new comments * Address more comments. --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com>

2025-06-03

Remove use of deprecated dict_ordered in datafusion-proto (#16218) (#16220) Co-authored-by: Sergey Zhukov <szhukov@aligntech.com>

2025-04-25

Fix `CoalescePartitionsExec` proto serialization (#15824) * add fetch to CoalescePartitionsExecNode * gen proto code * Add test * fix * fix build * Fix test build * remove comments

2025-04-17

fix: serialize listing table without partition column (#15737) * fix: serialize listing table without partition column * remove unwrap * format * clippy

2025-04-15

Add coerce int96 option for Parquet to support different TimeUnits, test int96_from_spark.parquet from parquet-testing (#15537)

2025-04-09

Rename protobuf Java package (#15658)

2025-03-21

feat: Support serde for FileScanConfig `batch_size` (#15335)

2025-03-20

feat: Support serde for JsonSource PhysicalPlan (#15311)

2025-03-13

feat: support customize metadata in alias for dataframe api (#15120) * feat: support customize metadata in alias for dataframe api * update doc * remove clone

2025-02-25

Add `statistics_truncate_length` parquet writer config (#14782) * Add parquet writer config * test fixes --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>

2025-02-14

feat: add resolved `target` to `DmlStatement` (to eliminate need for table lookup after deserialization) (#14631) * feat: serialize table source to DML proto * Update datafusion/core/src/dataframe/mod.rs Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> * remove redundant comment --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>

2025-02-11

Remove use of deprecated dict_id in datafusion-proto (#14173) (#14227) * Remove use of deprecated dict_id in datafusion-proto (#14173) * Fix issues causing GitHub checks to fail * Fix issues causing GitHub checks to fail * Fix issues causing GitHub checks to fail * Fix issues causing GitHub checks to fail * Fix issues causing GitHub checks to fail * Fix issues causing GitHub checks to fail * Fix issues causing GitHub checks to fail * Fix issues causing GitHub checks to fail * Fix issues causing GitHub checks to fail * Fix issues causing GitHub checks to fail * Fix issues causing GitHub checks to fail * Fix issues causing GitHub checks to fail * Fix issues causing GitHub checks to fail * Fix issues causing GitHub checks to fail * Fix issues causing GitHub checks to fail * Fix issues causing GitHub checks to fail * Fix issues causing GitHub checks to fail * remove accidental file * undo deletion of test in copy.slt * Fix issues causing GitHub checks to fail --------- Co-authored-by: Sergey Zhukov <szhukov@aligntech.com> Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>

2025-02-05

Serialize `parquet_options` in `datafusion-proto` (#14465) * Serialize `parquet_options` * Fix format

2025-01-28

Add `ColumnStatistics::Sum` (#14074) * Add sum statistic * Add sum statistic * Add sum statistic * Add sum statistic * Add sum statistic * Add sum statistic * Add tests and Cargo fmt * fix up --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>

2025-01-23

Update Logical Types Branch (#14241) * Handle alias when parsing sql(parse_sql_expr) (#12939) * fix: Fix parse_sql_expr not handling alias * cargo fmt * fix parse_sql_expr example(remove alias) * add testing * add SUM udaf to TestContextProvider and modify test_sql_to_expr_with_alias for function * revert change on example `parse_sql_expr` * Improve documentation for TableProvider (#13724) * Reveal implementing type and return type in simple UDF implementations (#13730) Debug trait is useful for understanding what something is and how it's configured, especially if the implementation is behind dyn trait. * minor: Extract tests for `EXTRACT` AND `date_part` to their own file (#13731) * Support unparsing `UNNEST` plan to `UNNEST` table factor SQL (#13660) * add `unnest_as_table_factor` and `UnnestRelationBuilder` * unparse unnest as table factor * fix typo * add tests for the default configs * add a static const for unnest_placeholder * fix tests * fix tests * Update to apache-avro 0.17, fix compatibility changes schema handling (#13727) * Update apache-avro requirement from 0.16 to 0.17 --- updated-dependencies: - dependency-name: apache-avro dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> * Fix compatibility changes schema handling apache-avro 0.17 - Handle ArraySchema struct - Handle MapSchema struct - Map BigDecimal => LargeBinary - Map TimestampNanos => Timestamp(TimeUnit::Nanosecond, None) - Map LocalTimestampNanos => todo!() - Add Default to FixedSchema test * Update Cargo.lock file for apache-avro 0.17 --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Marc Droogh <marc.droogh@imc.com> Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> * Minor: Add doc example to RecordBatchStreamAdapter (#13725) * Minor: Add doc example to RecordBatchStreamAdapter * Update datafusion/physical-plan/src/stream.rs Co-authored-by: Berkay Şahin <124376117+berkaysynnada@users.noreply.github.com> --------- Co-authored-by: Berkay Şahin <124376117+berkaysynnada@users.noreply.github.com> * Implement GroupsAccumulator for corr(x,y) aggregate function (#13581) * Implement GroupsAccumulator for corr(x,y) * feedbacks * fix CI MSRV * review * avoid collect in accumulation * add back cast * fix union serialisation order in proto (#13709) * fix union serialisation order in proto * clippy * address comments * Minor: make unsupported `nanosecond` part a real (not internal) error (#13733) * Minor: make unsupported `nanosecond` part a real (not internal) error * fmt * Improve wording to refer to date part * Add tests for date_part on columns + timestamps with / without timezones (#13732) * Add tests for date_part on columns + timestamps with / without timezones * Add tests from https://github.com/apache/datafusion/pull/13372 * remove trailing whitespace * Optimize performance of `initcap` function (~2x faster) (#13691) * Optimize performance of initcap (~2x faster) Signed-off-by: Tai Le Manh <manhtai.lmt@gmail.com> * format --------- Signed-off-by: Tai Le Manh <manhtai.lmt@gmail.com> * Minor: Add documentation explaining that initcap oly works for ASCII (#13749) * Support sqllogictest --complete with postgres (#13746) Before the change, the request to use PostgreSQL was simply ignored when `--complete` flag was present. * doc-gen: migrate window functions documentation to attribute based (#13739) * doc-gen: migrate window functions documentation Signed-off-by: zjregee <zjregee@gmail.com> * fix: update Cargo.lock --------- Signed-off-by: zjregee <zjregee@gmail.com> * Minor: Remove memory reservation in `JoinLeftData` used in HashJoin (#13751) * Refactor JoinLeftData structure by removing unused memory reservation field in hash join implementation * Add Debug and Clone derives for HashJoinStreamState and ProcessProbeBatchState enums This commit enhances the HashJoinStreamState and ProcessProbeBatchState structures by implementing the Debug and Clone traits, allowing for easier debugging and cloning of these state representations in the hash join implementation. * Update to bigdecimal 0.4.7 (#13747) * Add big decimal formatting test cases with potential trailing zeros * Rename and simplify decimal rendering functions - add `decimal` to function name - drop `precision` parameter as it is not supposed to affect the result * Update to bigdecimal 0.4.7 Utilize new `to_plain_string` function * chore: clean up dependencies (#13728) * CI: Warn on unused crates * CI: Warn on unused crates * CI: Warn on unused crates * CI: Warn on unused crates * CI: Clean up dependencies * CI: Clean up dependencies * fix: Implicitly plan `UNNEST` as lateral (#13695) * plan implicit lateral if table factor is UNNEST * check for outer references in `create_relation_subquery` * add sqllogictest * fix lateral constant test to not expect a subquery node * replace sqllogictest in favor of logical plan test * update lateral join sqllogictests * add sqllogictests * fix logical plan test * Minor: improve the Deprecation / API health guidelines (#13701) * Minor: improve the Deprecation / API health policy * prettier * Update docs/source/library-user-guide/api-health.md Co-authored-by: Jonah Gao <jonahgao@msn.com> * Add version guidance and make more copy/paste friendly * prettier * better * rename to guidelines --------- Co-authored-by: Jonah Gao <jonahgao@msn.com> * fix: specify roottype in substrait fieldreference (#13647) * fix: specify roottype in fieldreference Signed-off-by: MBWhite <whitemat@uk.ibm.com> * Fix formatting Signed-off-by: MBWhite <whitemat@uk.ibm.com> * review suggestion Signed-off-by: MBWhite <whitemat@uk.ibm.com> --------- Signed-off-by: MBWhite <whitemat@uk.ibm.com> * Simplify type signatures using `TypeSignatureClass` for mixed type function signature (#13372) * add type sig class Signed-off-by: jayzhan211 <jayzhan211@gmail.com> * timestamp Signed-off-by: jayzhan211 <jayzhan211@gmail.com> * date part Signed-off-by: jayzhan211 <jayzhan211@gmail.com> * fmt Signed-off-by: jayzhan211 <jayzhan211@gmail.com> * taplo format Signed-off-by: jayzhan211 <jayzhan211@gmail.com> * tpch test Signed-off-by: jayzhan211 <jayzhan211@gmail.com> * msrc issue Signed-off-by: jayzhan211 <jayzhan211@gmail.com> * msrc issue Signed-off-by: jayzhan211 <jayzhan211@gmail.com> * explicit hash Signed-off-by: jayzhan211 <jayzhan211@gmail.com> * Enhance type coercion and function signatures - Added logic to prevent unnecessary casting of string types in `native.rs`. - Introduced `Comparable` variant in `TypeSignature` to define coercion rules for comparisons. - Updated imports in `functions.rs` and `signature.rs` for better organization. - Modified `date_part.rs` to improve handling of timestamp extraction and fixed query tests in `expr.slt`. - Added `datafusion-macros` dependency in `Cargo.toml` and `Cargo.lock`. These changes improve type handling and ensure more accurate function behavior in SQL expressions. * fix comment Signed-off-by: Jay Zhan <jayzhan211@gmail.com> * fix signature Signed-off-by: Jay Zhan <jayzhan211@gmail.com> * fix test Signed-off-by: Jay Zhan <jayzhan211@gmail.com> * Enhance type coercion for timestamps to allow implicit casting from strings. Update SQL logic tests to reflect changes in timestamp handling, including expected outputs for queries involving nanoseconds and seconds. * Refactor type coercion logic for timestamps to improve readability and maintainability. Update the `TypeSignatureClass` documentation to clarify its purpose in function signatures, particularly regarding coercible types. This change enhances the handling of implicit casting from strings to timestamps. * Fix SQL logic tests to correct query error handling for timestamp functions. Updated expected outputs for `date_part` and `extract` functions to reflect proper behavior with nanoseconds and seconds. This change improves the accuracy of test cases in the `expr.slt` file. * Enhance timestamp handling in TypeSignature to support timezone specification. Updated the logic to include an additional DataType for timestamps with a timezone wildcard, improving flexibility in timestamp operations. * Refactor date_part function: remove redundant imports and add missing not_impl_err import for better error handling --------- Signed-off-by: jayzhan211 <jayzhan211@gmail.com> Signed-off-by: Jay Zhan <jayzhan211@gmail.com> * Minor: Add some more blog posts to the readings page (#13761) * Minor: Add some more blog posts to the readings page * prettier * prettier * Update docs/source/user-guide/concepts-readings-events.md --------- Co-authored-by: Oleks V <comphead@users.noreply.github.com> * docs: update GroupsAccumulator instead of GroupAccumulator (#13787) Fixing `GroupsAccumulator` trait name in its docs * Improve Deprecation Guidelines more (#13776) * Improve deprecation guidelines more * prettier * fix: add `null_buffer` length check to `StringArrayBuilder`/`LargeStringArrayBuilder` (#13758) * fix: add `null_buffer` check for `LargeStringArray` Add a safety check to ensure that the alignment of buffers cannot be overflowed. This introduces a panic if they are not aligned through a runtime assertion. * fix: remove value_buffer assertion These buffers can be misaligned and it is not problematic, it is the `null_buffer` which we care about being of the same length. * feat: add `null_buffer` check to `StringArray` This is in a similar vein to `LargeStringArray`, as the code is the same, except for `i32`'s instead of `i64`. * feat: use `row_count` var to avoid drift * Revert the removal of reservation in HashJoin (#13792) * fix: restore memory reservation in JoinLeftData for accurate memory accounting in HashJoin This commit reintroduces the `_reservation` field in the `JoinLeftData` structure to ensure proper tracking of memory resources during join operations. The absence of this field could lead to inconsistent memory usage reporting and potential out-of-memory issues as upstream operators increase their memory consumption. * fmt Signed-off-by: Jay Zhan <jay.zhan@synnada.ai> --------- Signed-off-by: Jay Zhan <jay.zhan@synnada.ai> * added count aggregate slt (#13790) * Update documentation guidelines for contribution content (#13703) * Update documentation guidelines for contribution content * Apply suggestions from code review Co-authored-by: Piotr Findeisen <piotr.findeisen@gmail.com> Co-authored-by: Oleks V <comphead@users.noreply.github.com> * clarify discussions and remove requirements note * prettier * Update docs/source/contributor-guide/index.md Co-authored-by: Piotr Findeisen <piotr.findeisen@gmail.com> --------- Co-authored-by: Piotr Findeisen <piotr.findeisen@gmail.com> Co-authored-by: Oleks V <comphead@users.noreply.github.com> * Add Round trip tests for Array <--> ScalarValue (#13777) * Add Round trip tests for Array <--> ScalarValue * String dictionary test * remove unecessary value * Improve comments * fix: Limit together with pushdown_filters (#13788) * fix: Limit together with pushdown_filters * Fix format * Address new comments * Fix testing case to hit the problem * Minor: improve Analyzer docs (#13798) * Minor: cargo update in datafusion-cli (#13801) * Update datafusion-cli toml to pin home=0.5.9 * update Cargo.lock * Fix `ScalarValue::to_array_of_size` for DenseUnion (#13797) * fix: enable pruning by bloom filters for dictionary columns (#13768) * Handle empty rows for `array_distinct` (#13810) * handle empty array distinct * ignore * fix --------- Co-authored-by: Cyprien Huet <chuet@palantir.com> * Fix get_type for higher-order array functions (#13756) * Fix get_type for higher-order array functions * Fix recursive flatten The fix is covered by recursive flatten test case in array.slt * Restore "keep LargeList" in Array signature * clarify naming in the test * Chore: Do not return empty record batches from streams (#13794) * do not emit empty record batches in plans * change function signatures to Option<RecordBatch> if empty batches are possible * format code * shorten code * change list_unnest_at_level for returning Option value * add documentation take concat_batches into compute_aggregates function again * create unit test for row_hash.rs * add test for unnest * add test for unnest * add test for partial sort * add test for bounded window agg * add test for window agg * apply simplifications and fix typo * apply simplifications and fix typo * Handle possible overflows in StringArrayBuilder / LargeStringArrayBuilder (#13802) * test(13796): reproducer of overflow on capacity * fix(13796): handle overflows with proper max capacity number which is valid for MutableBuffer * refactor: use simple solution and provide panic * fix: Ignore empty files in ListingTable when listing files with or without partition filters, as well as when inferring schema (#13750) * fix: Ignore empty files in ListingTable when listing files with or without partition filters, as well as when inferring schema * clippy * fix csv and json tests * add testing for parquet * cleanup * fix parquet tests * document describe_partition, add back repartition options to one of the csv empty files tests * Support Null regex override in csv parser options. (#13228) Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> * Minor: Extend ScalarValue::new_zero() (#13828) * Update mod.rs * Update mod.rs * Update mod.rs * Update mod.rs * chore: temporarily disable windows flow (#13833) * feat: `parse_float_as_decimal` supports scientific notation and Decimal256 (#13806) * feat: `parse_float_as_decimal` supports scientific notation and Decimal256 * Fix test * Add test * Add test * Refine negative scales * Update comment * Refine bigint_to_i256 * UT for bigint_to_i256 * Add ut for parse_decimal * Replace `BooleanArray::extend` with `append_n` (#13832) * Rename `TypeSignature::NullAry` --> `TypeSignature::Nullary` and improve comments (#13817) * Rename `TypeSignature::NullAry` --> `TypeSignature::Nullary` and improve comments * Apply suggestions from code review Co-authored-by: Piotr Findeisen <piotr.findeisen@gmail.com> * improve docs --------- Co-authored-by: Piotr Findeisen <piotr.findeisen@gmail.com> * [bugfix] ScalarFunctionExpr does not preserve the nullable flag on roundtrip (#13830) * [test] coalesce round trip schema mismatch * [proto] added the nullable flag in PhysicalScalarUdfNode * [bugfix] propagate the nullable flag for serialized scalar UDFS * Add example of interacting with a remote catalog (#13722) * Add example of interacting with a remote catalog * Update datafusion/core/src/execution/session_state.rs Co-authored-by: Berkay Şahin <124376117+berkaysynnada@users.noreply.github.com> * Apply suggestions from code review Co-authored-by: Jonah Gao <jonahgao@msn.com> Co-authored-by: Weston Pace <weston.pace@gmail.com> * Use HashMap to hold tables --------- Co-authored-by: Berkay Şahin <124376117+berkaysynnada@users.noreply.github.com> Co-authored-by: Jonah Gao <jonahgao@msn.com> Co-authored-by: Weston Pace <weston.pace@gmail.com> * Update substrait requirement from 0.49 to 0.50 (#13808) * Update substrait requirement from 0.49 to 0.50 Updates the requirements on [substrait](https://github.com/substrait-io/substrait-rs) to permit the latest version. - [Release notes](https://github.com/substrait-io/substrait-rs/releases) - [Changelog](https://github.com/substrait-io/substrait-rs/blob/main/CHANGELOG.md) - [Commits](https://github.com/substrait-io/substrait-rs/compare/v0.49.0...v0.50.0) --- updated-dependencies: - dependency-name: substrait dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> * Fix compilation * Add expr test --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: jonahgao <jonahgao@msn.com> * typo: remove extraneous "`" in doc comment, fix header (#13848) * typo: extraneous "`" in doc comment * Update datafusion/execution/src/runtime_env.rs * Update datafusion/execution/src/runtime_env.rs --------- Co-authored-by: Oleks V <comphead@users.noreply.github.com> * typo: remove extra "`" interfering with doc formatting (#13847) * Support n-ary monotonic functions in ordering equivalence (#13841) * Support n-ary monotonic functions in `discover_new_orderings` * Add tests for n-ary monotonic functions in `discover_new_orderings` * Fix tests * Fix non-monotonic test case * Fix unintended simplification * Minor comment changes * Fix tests * Add `preserves_lex_ordering` field * Use `preserves_lex_ordering` on `discover_new_orderings()` * Add `output_ordering` and `output_preserves_lex_ordering` implementations for `ConcatFunc` * Update tests * Move logic to UDF * Cargo fmt * Refactor * Cargo fmt * Simply use false value on default implementation * Remove unnecessary import * Clippy fix * Update Cargo.lock * Move dep to dev-dependencies * Rename output_preserves_lex_ordering to preserves_lex_ordering * minor --------- Co-authored-by: berkaysynnada <berkay.sahin@synnada.ai> * Replace `execution_mode` with `emission_type` and `boundedness` (#13823) * feat: update execution modes and add bitflags dependency - Introduced `Incremental` execution mode alongside existing modes in the DataFusion execution plan. - Updated various execution plans to utilize the new `Incremental` mode where applicable, enhancing streaming capabilities. - Added `bitflags` dependency to `Cargo.toml` for better management of execution modes. - Adjusted execution mode handling in multiple files to ensure compatibility with the new structure. * add exec API Signed-off-by: Jay Zhan <jay.zhan@synnada.ai> * replace done but has stackoverflow Signed-off-by: Jay Zhan <jay.zhan@synnada.ai> * exec API done Signed-off-by: Jay Zhan <jay.zhan@synnada.ai> * Refactor execution plan properties to remove execution mode - Removed the `ExecutionMode` parameter from `PlanProperties` across multiple physical plan implementations. - Updated related functions to utilize the new structure, ensuring compatibility with the changes. - Adjusted comments and cleaned up imports to reflect the removal of execution mode handling. This refactor simplifies the execution plan properties and enhances maintainability. * Refactor execution plan to remove `ExecutionMode` and introduce `EmissionType` - Removed the `ExecutionMode` parameter from `PlanProperties` and related implementations across multiple files. - Introduced `EmissionType` to better represent the output characteristics of execution plans. - Updated functions and tests to reflect the new structure, ensuring compatibility and enhancing maintainability. - Cleaned up imports and adjusted comments accordingly. This refactor simplifies the execution plan properties and improves the clarity of memory handling in execution plans. * fix test Signed-off-by: Jay Zhan <jay.zhan@synnada.ai> * Refactor join handling and emission type logic - Updated test cases in `sanity_checker.rs` to reflect changes in expected outcomes for bounded and unbounded joins, ensuring accurate test coverage. - Simplified the `is_pipeline_breaking` method in `execution_plan.rs` to clarify the conditions under which a plan is considered pipeline-breaking. - Enhanced the emission type determination logic in `execution_plan.rs` to prioritize `Final` over `Both` and `Incremental`, improving clarity in execution plan behavior. - Adjusted join type handling in `hash_join.rs` to classify `Right` joins as `Incremental`, allowing for immediate row emission. These changes improve the accuracy of tests and the clarity of execution plan properties. * Implement emission type for execution plans - Updated multiple execution plan implementations to replace `unimplemented!()` with `EmissionType::Incremental`, ensuring that the emission type is correctly defined for various plans. - This change enhances the clarity and functionality of the execution plans by explicitly specifying their emission behavior. These updates contribute to a more robust execution plan framework within the DataFusion project. * Enhance join type documentation and refine emission type logic - Updated the `JoinType` enum in `join_type.rs` to include detailed descriptions for each join type, improving clarity on their behavior and expected results. - Modified the emission type logic in `hash_join.rs` to ensure that `Right` and `RightAnti` joins are classified as `Incremental`, allowing for immediate row emission when applicable. These changes improve the documentation and functionality of join operations within the DataFusion project. * Refactor emission type logic in join and sort execution plans - Updated the emission type determination in `SortMergeJoinExec` and `SymmetricHashJoinExec` to utilize the `emission_type_from_children` function, enhancing the accuracy of emission behavior based on input characteristics. - Clarified comments in `sort.rs` regarding the conditions under which results are emitted, emphasizing the relationship between input sorting and emission type. - These changes improve the clarity and functionality of the execution plans within the DataFusion project, ensuring more robust handling of emission types. * Refactor emission type handling in execution plans - Updated the `emission_type_from_children` function to accept an iterator instead of a slice, enhancing flexibility in how child execution plans are passed. - Modified the `SymmetricHashJoinExec` implementation to utilize the new function signature, improving code clarity and maintainability. These changes streamline the emission type determination process within the DataFusion project, contributing to a more robust execution plan framework. * Enhance execution plan properties with boundedness and emission type - Introduced `boundedness` and `pipeline_behavior` methods to the `ExecutionPlanProperties` trait, improving the handling of execution plan characteristics. - Updated the `CsvExec`, `SortExec`, and related implementations to utilize the new methods for determining boundedness and emission behavior. - Refactored the `ensure_distribution` function to use the new boundedness logic, enhancing clarity in distribution decisions. - These changes contribute to a more robust and maintainable execution plan framework within the DataFusion project. * Refactor execution plans to enhance boundedness and emission type handling - Updated multiple execution plan implementations to incorporate `Boundedness` and `EmissionType`, improving the clarity and functionality of execution plans. - Replaced instances of `unimplemented!()` with appropriate emission types, ensuring that plans correctly define their output behavior. - Refactored the `PlanProperties` structure to utilize the new boundedness logic, enhancing decision-making in execution plans. - These changes contribute to a more robust and maintainable execution plan framework within the DataFusion project. * Refactor memory handling in execution plans - Updated the condition for checking memory requirements in execution plans from `has_finite_memory()` to `boundedness().requires_finite_memory()`, improving clarity in memory management. - This change enhances the robustness of execution plans within the DataFusion project by ensuring more accurate assessments of memory constraints. * Refactor boundedness checks in execution plans - Updated conditions for checking boundedness in various execution plans to use `is_unbounded()` instead of `requires_finite_memory()`, enhancing clarity in memory management. - Adjusted the `PlanProperties` structure to reflect these changes, ensuring more accurate assessments of memory constraints across the DataFusion project. - These modifications contribute to a more robust and maintainable execution plan framework, improving the handling of boundedness in execution strategies. * Remove TODO comment regarding unbounded execution plans in `UnboundedExec` implementation - Eliminated the outdated comment suggesting a switch to unbounded execution with finite memory, streamlining the code and improving clarity. - This change contributes to a cleaner and more maintainable codebase within the DataFusion project. * Refactor execution plan boundedness and emission type handling - Updated the `is_pipeline_breaking` method to use `requires_finite_memory()` for improved clarity in determining pipeline behavior. - Enhanced the `Boundedness` enum to include detailed documentation on memory requirements for unbounded streams. - Refactored `compute_properties` methods in `GlobalLimitExec` and `LocalLimitExec` to directly use the input's boundedness, simplifying the logic. - Adjusted emission type determination in `NestedLoopJoinExec` to utilize the `emission_type_from_children` function, ensuring accurate output behavior based on input characteristics. These changes contribute to a more robust and maintainable execution plan framework within the DataFusion project, improving clarity and functionality in handling boundedness and emission types. * Refactor emission type and boundedness handling in execution plans - Removed the `OptionalEmissionType` struct from `plan_properties.rs`, simplifying the codebase. - Updated the `is_pipeline_breaking` function in `execution_plan.rs` for improved readability by formatting the condition across multiple lines. - Adjusted the `GlobalLimitExec` implementation in `limit.rs` to directly use the input's boundedness, enhancing clarity in memory management. These changes contribute to a more streamlined and maintainable execution plan framework within the DataFusion project, improving the handling of emission types and boundedness. * Refactor GlobalLimitExec and LocalLimitExec to enhance boundedness handling - Updated the `compute_properties` methods in both `GlobalLimitExec` and `LocalLimitExec` to replace `EmissionType::Final` with `Boundedness::Bounded`, reflecting that limit operations always produce a finite number of rows. - Changed the input's boundedness reference to `pipeline_behavior()` for improved clarity in execution plan properties. These changes contribute to a more streamlined and maintainable execution plan framework within the DataFusion project, enhancing the handling of boundedness in limit operations. * Review Part1 * Update sanity_checker.rs * addressing reviews * Review Part 1 * Update datafusion/physical-plan/src/execution_plan.rs * Update datafusion/physical-plan/src/execution_plan.rs * Shorten imports * Enhance documentation for JoinType and Boundedness enums - Improved descriptions for the Inner and Full join types in join_type.rs to clarify their behavior and examples. - Added explanations regarding the boundedness of output streams and memory requirements in execution_plan.rs, including specific examples for operators like Median and Min/Max. --------- Signed-off-by: Jay Zhan <jay.zhan@synnada.ai> Co-authored-by: berkaysynnada <berkay.sahin@synnada.ai> Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com> * Preserve ordering equivalencies on `with_reorder` (#13770) * Preserve ordering equivalencies on `with_reorder` * Add assertions * Return early if filtered_exprs is empty * Add clarify comment * Refactor * Add comprehensive test case * Add comment for exprs_equal * Cargo fmt * Clippy fix * Update properties.rs * Update exprs_equal and add tests * Update properties.rs --------- Co-authored-by: berkaysynnada <berkay.sahin@synnada.ai> * replace CASE expressions in predicate pruning with boolean algebra (#13795) * replace CASE expressions in predicate pruning with boolean algebra * fix merge * update tests * add some more tests * add some more tests * remove duplicate test case * Update datafusion/physical-optimizer/src/pruning.rs * swap NOT for != * replace comments, update docstrings * fix example * update tests * update tests * Apply suggestions from code review Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> * Update pruning.rs Co-authored-by: Chunchun Ye <14298407+appletreeisyellow@users.noreply.github.com> * Update pruning.rs Co-authored-by: Chunchun Ye <14298407+appletreeisyellow@users.noreply.github.com> --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> Co-authored-by: Chunchun Ye <14298407+appletreeisyellow@users.noreply.github.com> * enable DF's nested_expressions feature by in datafusion-substrait tests to make them pass (#13857) fixes #13854 Co-authored-by: Arttu Voutilainen <avo@iki.fi> * Add configurable normalization for configuration options and preserve case for S3 paths (#13576) * Do not normalize values * Fix tests & update docs * Prettier * Lowercase config params * Unify transform and parse * Fix tests * Rename `default_transform` and relax boundaries * Make `compression` case-insensitive * Comment to new line * Deprecate and ignore `enable_options_value_normalization` * Update datafusion/common/src/config.rs * fix typo --------- Co-authored-by: Oleks V <comphead@users.noreply.github.com> * Improve`Signature` and `comparison_coercion` documentation (#13840) * Improve Signature documentation more * Apply suggestions from code review Co-authored-by: Piotr Findeisen <piotr.findeisen@gmail.com> --------- Co-authored-by: Piotr Findeisen <piotr.findeisen@gmail.com> * feat: support normalized expr in CSE (#13315) * feat: support normalized expr in CSE * feat: support normalize_eq in cse optimization * feat: support cumulative binary expr result in normalize_eq --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> * Upgrade to sqlparser `0.53.0` (#13767) * chore: Udpate to sqlparser 0.53.0 * Update for new sqlparser API * more api updates * Avoid serializing query to SQL string unless it is necessary * Box wildcard options * chore: update datafusion-cli Cargo.lock * Minor: Use `resize` instead of `extend` for adding static values in SortMergeJoin logic (#13861) Thanks @Dandandan * feat(function): add `least` function (#13786) * start adding least fn * feat(function): add least function * update function name * fix scalar smaller function * add tests * run Clippy and Fmt * Generated docs using `./dev/update_function_docs.sh` * add comment why `descending: false` * update comment * Update least.rs Co-authored-by: Bruce Ritchie <bruce.ritchie@veeva.com> * Update scalar_functions.md * run ./dev/update_function_docs.sh to update docs * merge greatest and least implementation to one * add header --------- Co-authored-by: Bruce Ritchie <bruce.ritchie@veeva.com> Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> * Improve SortPreservingMerge::enable_round_robin_repartition docs (#13826) * Clarify SortPreservingMerge::enable_round_robin_repartition docs * tweaks * Improve comments more * clippy * fix doc link * Minor: Unify `downcast_arg` method (#13865) * Implement `SHOW FUNCTIONS` (#13799) * introduce rid for different signature * implement show functions syntax * add syntax example * avoid duplicate join * fix clippy * show function_type instead of routine_type * add some doc and comments * Update bzip2 requirement from 0.4.3 to 0.5.0 (#13740) * Update bzip2 requirement from 0.4.3 to 0.5.0 Updates the requirements on [bzip2](https://github.com/trifectatechfoundation/bzip2-rs) to permit the latest version. - [Release notes](https://github.com/trifectatechfoundation/bzip2-rs/releases) - [Commits](https://github.com/trifectatechfoundation/bzip2-rs/compare/0.4.4...v0.5.0) --- updated-dependencies: - dependency-name: bzip2 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> * Fix test * Fix CLI cargo.lock --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: jonahgao <jonahgao@msn.com> * Fix build (#13869) * feat(substrait): modular substrait consumer (#13803) * feat(substrait): modular substrait consumer * feat(substrait): include Extension Rel handlers in default consumer Include SerializerRegistry based handlers for Extension Relations in the DefaultSubstraitConsumer * refactor(substrait) _selection -> _field_reference * refactor(substrait): remove SubstraitPlannerState usage from consumer * refactor: get_state() -> get_function_registry() * docs: elide imports from example * test: simplify test * refactor: remove Arc from DefaultSubstraitConsumer * doc: add ticket for API improvements * doc: link DefaultSubstraitConsumer to from_subtrait_plan * refactor: remove redundant Extensions parsing * Minor: fix: Include FetchRel when producing LogicalPlan from Sort (#13862) * include FetchRel when producing LogicalPlan from Sort * add suggested test * address review feedback * Minor: improve error message when ARRAY literals can not be planned (#13859) * Minor: improve error message when ARRAY literals can not be planned * fmt * Update datafusion/sql/src/expr/value.rs Co-authored-by: Oleks V <comphead@users.noreply.github.com> --------- Co-authored-by: Oleks V <comphead@users.noreply.github.com> * Add documentation for `SHOW FUNCTIONS` (#13868) * Support unicode character for `initcap` function (#13752) * Support unicode character for 'initcap' function Signed-off-by: Tai Le Manh <manhtai.lmt@gmail.com> * Update unit tests * Fix clippy warning * Update sqllogictests - initcap * Update scalar_functions.md docs * Add suggestions change Signed-off-by: Tai Le Manh <manhtai.lmt@gmail.com> --------- Signed-off-by: Tai Le Manh <manhtai.lmt@gmail.com> * [minor] make recursive package dependency optional (#13778) * make recursive optional * add to default for common package * cargo update * added to readme * make test conditional * reviews * cargo update --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> * Minor: remove unused async-compression `futures-io` feature (#13875) * Minor: remove unused async-compression feature * Fix cli cargo lock * Consolidate Example: dataframe_output.rs into dataframe.rs (#13877) * Restore `DocBuilder::new()` to avoid breaking API change (#13870) * Fix build * Restore DocBuilder::new(), deprecate * cmt * clippy * Improve error messages for incorrect zero argument signatures (#13881) * Improve error messages for incorrect zero argument signatures * fix errors * fix fmt * Consolidate Example: simplify_udwf_expression.rs into advanced_udwf.rs (#13883) * minor: fix typos in comments / structure names (#13879) * minor: fix typo error in datafusion * fix: fix rebase error * fix: format HashJoinExec doc * doc: recover thiserror/preemptively * fix: other typo error fixed * fix: directories to dir_entries in catalog example * Support 1 or 3 arg in generate_series() UDTF (#13856) * Support 1 or 3 args in generate_series() UDTF * address comment * Support (order by / sort) for DataFrameWriteOptions (#13874) * Support (order by / sort) for DataFrameWriteOptions * Fix fmt * Fix import * Add insert into example * Update sort_merge_join.rs (#13894) * Update join_selection.rs (#13893) * Fix `recursive-protection` feature flag (#13887) * Fix recursive-protection feature flag * rename feature flag to be consistent * Make default * taplo format * Fix visibility of swap_hash_join (#13899) * Minor: Avoid emitting empty batches in partial sort (#13895) * Update partial_sort.rs * Update partial_sort.rs * Update partial_sort.rs * Prepare for 44.0.0 release: version and changelog (#13882) * Prepare for 44.0.0 release: version and changelog * update changelog * update configs * update before release * Support unparsing implicit lateral `UNNEST` plan to SQL text (#13824) * support unparsing the implicit lateral unnest plan * cargo clippy and fmt * refactor for `check_unnest_placeholder_with_outer_ref` * add const for the prefix string of unnest and outer refernece column * fix case_column_or_null with nullable when conditions (#13886) * fix case_column_or_null with nullable when conditions * improve sqllogictests for case_column_or_null --------- Co-authored-by: zhangli20 <zhangli20@kuaishou.com> * Fixed Issue #13896 (#13903) The URL to the external website was returning a 404. Presuming recent changes in the external website's structure, the required data has been moved to a different URL. The commit ensures the new URL is used. * Introduce `UserDefinedLogicalNodeUnparser` for User-defined Logical Plan unparsing (#13880) * make ast builder public * introduce udlp unparser * add documents * add examples * add negative tests and fmt * fix the doc * rename udlp to extension * apply the first unparsing result only * improve the doc * seperate the enum for the unparsing result * fix the doc --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> * Preserve constant values across union operations (#13805) * Add value tracking to ConstExpr for improved union optimization * Update PartialEq impl * Minor change * Add docstring for ConstExpr value * Improve constant propagation across union partitions * Add assertion for across_partitions * fix fmt * Update properties.rs * Remove redundant constant removal loop * Remove unnecessary mut * Set across_partitions=true when both sides are constant * Extract and use constant values in filter expressions * Add initial SLT for constant value tracking across UNION ALL * Assign values to ConstExpr where possible * Revert "Set across_partitions=true when both sides are constant" This reverts commit 3051cd470b0ad4a70cd8bd3518813f5ce0b3a449. * Temporarily take value from literal * Lint fixes * Cargo fmt * Add get_expr_constant_value * Make `with_value()` accept optional value * Add todo * Move test to union.slt * Fix changed slt after merge * Simplify constexpr * Update properties.rs --------- Co-authored-by: berkaysynnada <berkay.sahin@synnada.ai> * chore(deps): update sqllogictest requirement from 0.23.0 to 0.24.0 (#13902) * fix RecordBatch size in topK (#13906) * ci improvements, update protoc (#13876) * Fix md5 return_type to only return Utf8 as per current code impl. * ci improvements * Lock taiki-e/install-action to a githash for apache action policy - Release 2.46.19 in the case of this hash. * Lock taiki-e/install-action to a githash for apache action policy - Release 2.46.19 in the case of this hash. * Revert nextest change until action is approved. * Exclude requires workspace * Fixing minor typo to verify ci caching of builds is working as expected. * Updates from PR review. * Adding issue link for disabling intel mac build * improve performance of running examples * remove cargo check * Introduce LogicalPlan invariants, begin automatically checking them (#13651) * minor(13525): perform LP validation before and after each possible mutation * minor(13525): validate unique field names on query and subquery schemas, after each optimizer pass * minor(13525): validate union after each optimizer passes * refactor: make explicit what is an invariant of the logical plan, versus assertions made after a given analyzer or optimizer pass * chore: add link to invariant docs * fix: add new invariants module * refactor: move all LP invariant checking into LP, delineate executable (valid semantic plan) vs basic LP invariants * test: update test for slight error message change * fix: push_down_filter optimization pass can push a IN(<subquery>) into a TableScan's filter clause * refactor: move collect_subquery_cols() to common utils crate * refactor: clarify the purpose of assert_valid_optimization(), runs after all optimizer passes, except in debug mode it runs after each pass. * refactor: based upon performance tests, run the maximum number of checks without impa ct: * assert_valid_optimization can run each optimizer pass * remove the recursive cehck_fields, which caused the performance regression * the full LP Invariants::Executable can only run in debug * chore: update error naming and terminology used in code comments * refactor: use proper error methods * chore: more cleanup of error messages * chore: handle option trailer to error message * test: update sqllogictests tests to not use multiline * Correct return type for initcap scalar function with utf8view (#13909) * Set utf8view as return type when input type is the same * Verify that the returned type from call to scalar function matches the return type specified in the return_type function * Match return type to utf8view * Consolidate example: simplify_udaf_expression.rs into advanced_udaf.rs (#13905) * Implement maintains_input_order for AggregateExec (#13897) * Implement maintains_input_order for AggregateExec * Update mod.rs * Improve comments --------- Co-authored-by: berkaysynnada <berkay.sahin@synnada.ai> Co-authored-by: mertak-synnada <mertak67+synaada@gmail.com> Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com> * Move join type input swapping to pub methods on Joins (#13910) * doc-gen: migrate scalar functions (string) documentation 3/4 (#13926) Co-authored-by: Cheng-Yuan-Lai <a186235@g,ail.com> * Update sqllogictest requirement from 0.24.0 to 0.25.0 (#13917) * Update sqllogictest requirement from 0.24.0 to 0.25.0 Updates the requirements on [sqllogictest](https://github.com/risinglightdb/sqllogictest-rs) to permit the latest version. - [Release notes](https://github.com/risinglightdb/sqllogictest-rs/releases) - [Changelog](https://github.com/risinglightdb/sqllogictest-rs/blob/main/CHANGELOG.md) - [Commits](https://github.com/risinglightdb/sqllogictest-rs/compare/v0.24.0...v0.25.0) --- updated-dependencies: - dependency-name: sqllogictest dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> * Remove labels --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: jonahgao <jonahgao@msn.com> * Consolidate Examples: memtable.rs and parquet_multiple_files.rs (#13913) * doc-gen: migrate scalar functions (crypto) documentation (#13918) * doc-gen: migrate scalar functions (crypto) documentation * doc-gen: fix typo and update function docs --------- Co-authored-by: Cheng-Yuan-Lai <a186235@g,ail.com> * doc-gen: migrate scalar functions (datetime) documentation 1/2 (#13920) * doc-gen: migrate scalar functions (datetime) documentation 1/2 * fix: fix typo and update function docs --------- Co-authored-by: Cheng-Yuan-Lai <a186235@g,ail.com> * fix RecordBatch size in hash join (#13916) * doc-gen: migrate scalar functions (array) documentation 1/3 (#13928) * doc-gen: migrate scalar functions (array) documentation 1/3 * fix: remove unsed import, fix typo and update function docs --------- Co-authored-by: Cheng-Yuan-Lai <a186235@g,ail.com> * doc-gen: migrate scalar functions (math) documentation 1/2 (#13922) * doc-gen: migrate scalar functions (math) documentation 1/2 * fix: fix typo --------- Co-authored-by: Cheng-Yuan-Lai <a186235@g,ail.com> * doc-gen: migrate scalar functions (math) documentation 2/2 (#13923) * doc-gen: migrate scalar functions (math) documentation 2/2 * fix: fix typo --------- Co-authored-by: Cheng-Yuan-Lai <a186235@g,ail.com> * doc-gen: migrate scalar functions (array) documentation 3/3 (#13930) * doc-gen: migrate scalar functions (array) documentation 3/3 * fix: import doc and macro, fix typo and update function docs --------- Co-authored-by: Cheng-Yuan-Lai <a186235@g,ail.com> * doc-gen: migrate scalar functions (array) documentation 2/3 (#13929) * doc-gen: migrate scalar functions (array) documentation 2/3 * fix: import doc and macro, fix typo and update function docs --------- Co-authored-by: Cheng-Yuan-Lai <a186235@g,ail.com> * doc-gen: migrate scalar functions (string) documentation 4/4 (#13927) * doc-gen: migrate scalar functions (string) documentation 4/4 * fix: fix typo and update function docs --------- Co-authored-by: Cheng-Yuan-Lai <a186235@g,ail.com> * Support explain query when running dfbench with clickbench (#13942) * Support explain query when running dfbench * Address comments * Consolidate example to_date.rs into dateframe.rs (#13939) * Consolidate example to_date.rs into dateframe.rs * Assert results using assert_batches_eq * clippy * Revert "Update sqllogictest requirement from 0.24.0 to 0.25.0 (#13917)" (#13945) * Revert "Update sqllogictest requirement from 0.24.0 to 0.25.0 (#13917)" This reverts commit 0989649214a6fe69ffb33ed38c42a8d3df94d6bf. * add comment * Implement predicate pruning for `like` expressions (prefix matching) (#12978) * Implement predicate pruning for like expressions * add function docstring * re-order bounds calculations * fmt * add fuzz tests * fix clippy * Update datafusion/core/tests/fuzz_cases/pruning.rs Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> * doc-gen: migrate scalar functions (string) documentation 1/4 (#13924) Co-authored-by: Cheng-Yuan-Lai <a186235@g,ail.com> * consolidate dataframe_subquery.rs into dataframe.rs (#13950) * migrate btrim to user_doc macro (#13952) * doc-gen: migrate scalar functions (datetime) documentation 2/2 (#13921) * doc-gen: migrate scalar functions (datetime) documentation 2/2 * fix: fix typo and update function docs * doc: update function docs * doc-gen: remove slash --------- Co-authored-by: Cheng-Yuan-Lai <a186235@g,ail.com> * Add sqlite test files, progress bar, and automatic postgres container management into sqllogictests (#13936) * Fix md5 return_type to only return Utf8 as per current code impl. * Add support for sqlite test files to sqllogictest * Force version 0.24.0 of sqllogictest dependency until issue with labels is fixed. * Removed workaround for bug that was fixed. * Git submodule update ... err update, link to sqlite tests. * Git submodule update * Readd submodule --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> * Supporting writing schema metadata when writing Parquet in parallel (#13866) * refactor: make ParquetSink tests a bit more readable * chore(11770): add new ParquetOptions.skip_arrow_metadata * test(11770): demonstrate that the single threaded ParquetSink is already writing the arrow schema in the kv_meta, and allow disablement * refactor(11770): replace with new method, since the kv_metadata is inherent to TableParquetOptions and therefore we should explicitly make the API apparant that you have to include the arrow schema or not * fix(11770): fix parallel ParquetSink to encode arrow schema into the file metadata, based on the ParquetOptions * refactor(11770): provide deprecation warning for TryFrom * test(11770): update tests with new default to include arrow schema * refactor: including partitioning of arrow schema inserted into kv_metdata * test: update tests for new config prop, as well as the new file partition offsets based upon larger metadata * chore: avoid cloning in tests, and update code docs * refactor: return to the WriterPropertiesBuilder::TryFrom<TableParquetOptions>, and separately add the arrow_schema to the kv_metadata on the TableParquetOptions * refactor: require the arrow_schema key to be present in the kv_metadata, if is required by the configuration * chore: update configs.md * test: update tests to handle the (default) required arrow schema in the kv_metadata * chore: add reference to arrow-rs upstream PR * chore: Create devcontainer.json (#13520) * Create devcontainer.json * update devcontainer * remove useless features * Minor: consolidate ConfigExtension example into API docs (#13954) * Update examples README.md * Minor: consolidate ConfigExtension example into API docs * more docs * Remove update * clippy * Fix issue with ExtensionsOptions docs * Parallelize pruning utf8 fuzz test (#13947) * Add swap_inputs to SMJ (#13984) * fix(datafusion-functions-nested): `arrow-distinct` now work with null rows (#13966) * added failing test * fix(datafusion-functions-nested): `arrow-distinct` now work with null rows * Update datafusion/functions-nested/src/set_ops.rs Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> * Update set_ops.rs --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> * Update release instructions for 44.0.0 (#13959) * Update release instructions for 44.0.0 * update macros and order * add functions-table * Add datafusion python 43.1.0 blog post to doc. (#13974) * Include license and notice files in more crates (#13985) * Extract postgres container from sqllogictest, update datafusion-testing pin (#13971) * Add support for sqlite test files to sqllogictest * Removed workaround for bug that was fixed. * Refactor sqllogictest to extract postgres functionality into a separate file. Removed dependency on once_cell in favour of LazyLock. * Add missing license header. * Update rstest requirement from 0.23.0 to 0.24.0 (#13977) Updates the requirements on [rstest](https://github.com/la10736/rstest) to permit the latest version. - [Release notes](https://github.com/la10736/rstest/releases) - [Changelog](https://github.com/la10736/rstest/blob/master/CHANGELOG.md) - [Commits](https://github.com/la10736/rstest/compare/v0.23.0...v0.23.0) --- updated-dependencies: - dependency-name: rstest dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Move hash collision test to run only when merging to main. (#13973) * Update itertools requirement from 0.13 to 0.14 (#13965) * Update itertools requirement from 0.13 to 0.14 Updates the requirements on [itertools](https://github.com/rust-itertools/itertools) to permit the latest version. - [Changelog](https://github.com/rust-itertools/itertools/blob/master/CHANGELOG.md) - [Commits](https://github.com/rust-itertools/itertools/compare/v0.13.0...v0.13.0) --- updated-dependencies: - dependency-name: itertools dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> * Fix build * Simplify * Update CLI lock --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: jonahgao <jonahgao@msn.com> * Change trigger, rename `hash_collision.yml` to `extended.yml` and add comments (#13988) * Rename hash_collision.yml to extended.yml and add comments * Adjust schedule, add comments * Update job, rerun * doc-gen: migrate scalar functions (string) documentation 2/4 (#13925) * doc-gen: migrate scalar functions (string) documentation 2/4 * doc-gen: update function docs * doc: fix related udf order for upper function in documentation * Update datafusion/functions/src/string/concat_ws.rs * Update datafusion/functions/src/string/concat_ws.rs * Update datafusion/functions/src/string/concat_ws.rs * doc-gen: update function docs --------- Co-authored-by: Cheng-Yuan-Lai <a186235@g,ail.com> Co-authored-by: Oleks V <comphead@users.noreply.github.com> * Update substrait requirement from 0.50 to 0.51 (#13978) Updates the requirements on [substrait](https://github.com/substrait-io/substrait-rs) to permit the latest version. - [Release notes](https://github.com/substrait-io/substrait-rs/releases) - [Changelog](https://github.com/substrait-io/substrait-rs/blob/main/CHANGELOG.md) - [Commits](https://github.com/substrait-io/substrait-rs/compare/v0.50.0...v0.51.0) --- updated-dependencies: - dependency-name: substrait dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Update release README for datafusion-cli publishing (#13982) * Enhance LastValueAccumulator logic and add SQL logic tests for last_value function (#13980) - Updated LastValueAccumulator to include requirement satisfaction check before updating the last value. - Added SQL logic tests to verify the behavior of the last_value function with merge batches and ensure correct aggregation in various scenarios. * Improve deserialize_to_struct example (#13958) * Cleanup deserialize_to_struct example * prettier * Apply suggestions from code review Co-authored-by: Jonah Gao <jonahgao@msn.com> --------- Co-authored-by: Jonah Gao <jonahgao@msn.com> * Update docs (#14002) * Optimize CASE expression for "expr or expr" usage. (#13953) * Apply optimization for ExprOrExpr. * Implement optimization similar to existing code. * Add sqllogictest. * feat(substrait): introduce consume_rel and consume_expression (#13963) * feat(substrait): introduce consume_rel and consume_expression Route calls to from_substrait_rel and from_substrait_rex through the SubstraitConsumer in order to allow users to provide their own behaviour * feat(substrait): consume nulls of user-defined types * docs(substrait): consume_rel and consume_expression docstrings * Consolidate csv_opener.rs and json_opener.rs into a single example (#… (#13981) * Consolidate csv_opener.rs and json_opener.rs into a single example (#13955) * Update datafusion-examples/examples/csv_json_opener.rs Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> * Update datafusion-examples/README.md Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> * Apply code formatting with cargo fmt --------- Co-authored-by: Sergey Zhukov <szhukov@aligntech.com> Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> * FIX : Incorrect NULL handling in BETWEEN expression (#14007) * submodule update * FIX : Incorrect NULL handling in BETWEEN expression * Revert "submodule update" This reverts commit 72431aadeaf33a27775a88c41931572a0b66bae3. * fix incorrect unit test * move sqllogictest to expr * feat(substrait): modular substrait producer (#13931) * feat(substrait): modular substrait producer * refactor(substrait): simplify col_ref_offset handling in producer * refactor(substrait): remove column offset tracking from producer * docs(substrait): document SubstraitProducer * refactor: minor cleanup * feature: remove unused SubstraitPlanningState BREAKING CHANGE: SubstraitPlanningState is no longer available * refactor: cargo fmt * refactor(substrait): consume_ -> handle_ * refactor(substrait): expand match blocks * refactor: DefaultSubstraitProducer only needs serializer_registry * refactor: remove unnecessary warning suppression * fix(substrait): route expr conversion through handle_expr * cargo fmt * fix: Avoid re-wrapping planning errors Err(DataFusionError::Plan) for use in plan_datafusion_err (#14000) * fix: unwrapping Err(DataFusionError::Plan) for use in plan_datafusion_err * test: add tests for error formatting during planning * feat: support `RightAnti` for `SortMergeJoin` (#13680) * feat: support `RightAnti` for `SortMergeJoin` * feat: preserve session id when using cxt.enable_url_table() (#14004) * Return error message during planning when inserting into a MemTable with zero partitions. (#14011) * Minor: Rewrite LogicalPlan::max_rows for Join and Union, made it easier to understand (#14012) * Refactor max_rows for join plan, made it easier to understand * Simplified max_rows for Union * Chore: update wasm-supported crates, add tests (#14005) * Chore: update wasm-supported crates * format * Use workspace rust-version for all workspace crates (#14009) * [Minor] refactor: make ArraySort public for broader access (#14006) * refactor: make ArraySort public for broader access Changes the visibility of the ArraySort struct fromsuper to public. allows broader access to the struct, enabling its use in other modules and promoting better code reuse. * clippy and docs --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> * Update sqllogictest requirement from =0.24.0 to =0.26.0 (#14017) * Update sqllogictest requirement from =0.24.0 to =0.26.0 Updates the requirements on [sqllogictest](https://github.com/risinglightdb/sqllogictest-rs) to permit the latest version. - [Release notes](https://github.com/risinglightdb/sqllogictest-rs/releases) - [Changelog](https://github.com/risinglightdb/sqllogictest-rs/blob/main/CHANGELOG.md) - [Commits](https://github.com/risinglightdb/sqllogictest-rs/compare/v0.24.0...v0.26.0) --- updated-dependencies: - dependency-name: sqllogictest dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> * remove version pin and note --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Eduard Karacharov <eduard.karacharov@gmail.com> * `url` dependancy update (#14019) * `url` dependancy update * `url` version update for datafusion-cli * Minor: Improve zero partition check when inserting into `MemTable` (#14024) * Improve zero partition check when inserting into `MemTable` * update err msg * refactor: make structs public and implement Default trait (#14030) * Minor: Remove redundant implementation of `StringArrayType` (#14023) * Minor: Remove redundant implementation of StringArrayType Signed-off-by: Tai Le Manh <manhtai.lmt@gmail.com> * Deprecate rather than remove StringArrayType --------- Signed-off-by: Tai Le Manh <manhtai.lmt@gmail.com> Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> * Added references to IDE documentation for dev containers along with a small note about why one may choose to do development using a dev container. (#14014) * Use partial aggregation schema for spilling to avoid column mismatch in GroupedHashAggregateStream (#13995) * Refactor spill handling in GroupedHashAggregateStream to use partial aggregate schema * Implement aggregate functions with spill handling in tests * Add tests for aggregate functions with and without spill handling * Move test related imports into mod test * Rename spill pool test functions for clarity and consistency * Refactor aggregate function imports to use fully qualified paths * Remove outdated comments regarding input batch schema for spilling in GroupedHashAggregateStream * Update aggregate test to use AVG instead of MAX * assert spill count * Refactor partial aggregate schema creation to use create_schema function * Refactor partial aggregation schema creation and remove redundant function * Remove unused import of Schema from arrow::datatypes in row_hash.rs * move spill pool testing for aggregate functions to physical-plan/src/aggregates * Use Arc::clone for schema references in aggregate functions * Encapsulate fields of `EquivalenceProperties` (#14040) * Encapsulate fields of `EquivalenceGroup` (#14039) * Fix error on `array_distinct` when input is empty #13810 (#14034) * fix * add test * oops --------- Co-authored-by: Cyprien Huet <chuet@palantir.com> * Update petgraph requirement from 0.6.2 to 0.7.1 (#14045) * Update petgraph requirement from 0.6.2 to 0.7.1 Updates the requirements on [petgraph](https://github.com/petgraph/petgraph) to permit the latest version. - [Changelog](https://github.com/petgraph/petgraph/blob/master/RELEASES.rst) - [Commits](https://github.com/petgraph/petgraph/compare/petgraph@v0.6.2...petgraph@v0.7.1) --- updated-dependencies: - dependency-name: petgraph dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> * Update datafusion-cli/Cargo.lock --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> * Encapsulate fields of `OrderingEquivalenceClass` (make field non pub) (#14037) * Complete encapsulatug `OrderingEquivalenceClass` (make fields non pub) * fix doc * Fix: ensure that compression type is also taken into consideration during ListingTableConfig infer_options (#14021) * chore: add test to verify that schema is inferred as expected * chore: add comment to method as suggested * chore: restructure to avoid need to clone * chore: fix flaw in rewrite * feat(optimizer): Enable filter pushdown on window functions (#14026) * feat(optimizer): Enable filter pushdown on window functions Ensures selections can be pushed past window functions similarly to what is already done with aggregations, when possible. * fix: Add missing dependency * minor(optimizer): Use 'datafusion-functions-window' as a dev dependency * docs(optimizer): Add example to filter pushdown on LogicalPlan::Window * Unparsing optimized (> 2 inputs) unions (#14031) * tests and optimizer in testing queries * unparse optimized unions * format Cargo.toml * format Cargo.toml * revert test * rewrite test to avoid cyclic dep * remove old test * cleanup * comments and error handling * handle union with lt 2 inputs * Minor: Document output schema of LogicalPlan::Aggregate and LogicalPlan::Window (#14047) * Simplify error handling in case.rs (#13990) (#14033) * Simplify error handling in case.rs (#13990) * Fix issues causing GitHub checks to fail * Update datafusion/physical-expr/src/expressions/case.rs Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> --------- Co-authored-by: Sergey Zhukov <szhukov@aligntech.com> Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> * feat: add `AsyncCatalogProvider` helpers for asynchronous catalogs (#13800) * Add asynchronous catalog traits to help users that have asynchronous catalogs * Apply clippy suggestions * Address PR reviews * Remove allow_unused exceptions * Update remote catalog example to demonstrate new helper structs * Move schema_name / catalog_name parameters into resolve function and out of trait * Custom scalar to sql overrides support for DuckDB Unparser dialect (#13915) * Allow adding custom scalar to sql overrides for DuckDB (#68) * Add unit test: custom_scalar_overrides_duckdb * Move `with_custom_scalar_overrides` definition on `Dialect` trait level * Improve perfomance of `reverse` function (#14025) * Improve perfomance of 'reverse' function Signed-off-by: Tai Le Manh <manhtai.lmt@gmail.com> * Apply sugestion change * Fix typo --------- Signed-off-by: Tai Le Manh <manhtai.lmt@gmail.com> * docs(ci): use up-to-date protoc with docs.rs (#14048) * fix (#14042) Co-authored-by: Cyprien Huet <chuet@palantir.com> * Re-export TypeSignatureClass from the datafusion-expr package (#14051) * Fix clippy for Rust 1.84 (#14065) * fix: incorrect error message of function_length_check (#14056) * minor fix * add ut * remove check for 0 arg * test: Add plan execution during tests for bounded source (#14013) * Bump `ctor` to `0.2.9` (#14069) * Refactor into `LexOrdering::collapse`, `LexRequirement::collapse` avoid clone (#14038) * Move collapse_lex_ordering to Lexordering::collapse * reduce diff * avoid clone, cleanup * Introduce LexRequirement::collapse * Improve performance of collapse, from @akurmustafa https://github.com/alamb/datafusion/pull/26 fix formatting * Revert "Improve performance of collapse, from @akurmustafa" This reverts commit a44acfdb3af5bf0082c277de6ee7e09e92251a49. * remove incorrect comment --------- Co-authored-by: Mustafa Akur <akurmustafa@gmail.com> * Bump `wasm-bindgen` and `wasm-bindgen-futures` (#14068) * update (#14070) * fix: make get_valid_types handle TypeSignature::Numeric correctly (#14060) * fix get_valid_types with TypeSignature::Numeric * fix sqllogictest * Minor: Make `group_schema` as `PhysicalGroupBy` method (#14064) * group shema as method Signed-off-by: Jay Zhan <jayzhan211@gmail.com> * fmt Signed-off-by: Jay Zhan <jayzhan211@gmail.com> --------- Signed-off-by: Jay Zhan <jayzhan211@gmail.com> * Minor: Move `LimitPushdown` tests to be in the same file as the code (#14076) * Minor: move limit_pushdown tests to be with their pass * Fix clippy * cleaup use * fmt * Add comments to physical optimizer tests (#14075) * added "DEFAULT_CLI_FORMAT_OPTIONS" for cli and sqllogic test (#14052) * added "DEFAULT_CLI_FORMAT_OPTIONS" for cli and sqllotic test * cargo fmt fix * fixed few errors * Add H2O.ai Database-like Ops benchmark to dfbench (groupby support) (#13996) * Add H2O.ai Database-like Ops benchmark to dfbench * Fix query and fmt * Change venv * Make sure venv version support falsa * Fix default path * Support groupby only now * fix * Address comments * fix * support python version higher * support higer python such as python 3.13 * Addressed new comments * Add specific query example * Add telemetry.sh to list of use cases (#14090) * chore: deprecate `ValuesExec` in favour of `MemoryExec` (#14032) * chore: deprecate `ValuesExec` in favour of `MemoryExec` * clippy fix * Update datafusion/physical-plan/src/values.rs Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> * change to memoryexec * Update datafusion/physical-plan/src/memory.rs Co-authored-by: Jay Zhan <jayzhan211@gmail.com> * use compute properties * clippy fix --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> Co-authored-by: Jay Zhan <jayzhan211@gmail.com> * Improve performance of `find_in_set` function (#14020) * Improve performance of 'find_in_set' function Signed-off-by: Tai Le Manh <manhtai.lmt@gmail.com> * Remove clippy warnings * Support scalar args for 'find_in_set' function Signed-off-by: Tai Le Manh <manhtai.lmt@gmail.com> --------- Signed-off-by: Tai Le Manh <manhtai.lmt@gmail.com> * minor: Add link to example in catalog (#14062) * fix set (#14081) Signed-off-by: Jay Zhan <jayzhan211@gmail.com> * Simplify the return type of `sql_select_to_rex()` (#14088) * Minor: Add a link to RecordBatchStreamAdapter to `SendableRecordBatchStream` (#14084) * Update substrait requirement from 0.51 to 0.52 (#14107) * Update substrait requirement from 0.51 to 0.52 Updates the requirements on [substrait](https://github.com/substrait-io/substrait-rs) to permit the latest version. - [Release notes](https://github.com/substrait-io/substrait-rs/releases) - [Changelog](https://github.com/substrait-io/substrait-rs/blob/main/CHANGELOG.md) - [Commits](https://github.com/substrait-io/substrait-rs/compare/v0.51.0...v0.52.0) --- updated-dependencies: - dependency-name: substrait dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> * Handle addition of `ReadType::IcebergTable` --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * fix null date args in range (#14093) * fix null dates in range * fix --------- Co-authored-by: Cyprien Huet <chuet@palantir.com> * feat: add support for `LogicalPlan::DML(...)` serde (#14079) * Add support for DML serialization to proto closes: #13616 * add round trip test for DML serde * cover all cases in round trip test * minor: change ordering of enum type * Avoid Aliased Window Expr Enter Unreachable Code (#14109) * clarify logic in nth_value window function (#14104) * Move JoinSelection into datafusion-physical-optimizer crate (#14073) (#14085) * Move JoinSelection into datafusion-physical-optimizer crate (#14073) * Fix issues causing GitHub checks to fail * Fix issues causing GitHub checks to fail * Fix issues causing GitHub checks to fail * Lock aws-sdk crates to fix MSRV check * fix comment * fix compilation --------- Co-authored-by: Sergey Zhukov <szhukov@aligntech.com> Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> * distinguish none and empty projection (#14116) * Add a hint about normalization in error message (#14089) (#14113) * Add a hint about normalization in error message (#14089) * normalization suggestion is only shown when a column name matches schema --------- Co-authored-by: Sergey Zhukov <szhukov@aligntech.com> * fix: incorrect NATURAL/USING JOIN schema (#14102) * fix: incorrect NATURAL/USING JOIN schema * Add test * Simplify exclude_using_columns * Add more tests * Chore: refactor DataSink traits to avoid duplication (#14121) * add some abstractions to file sinkers and centralize FileSinkConfig based behaviors * satisfy clippy * typo fix * move start_demuxer_task back into demux.rs add file_extension to FileSinkConfig * fix errors * merge get_writer_schema functions add schema() function to DataSink trait make FileSink a subtrait for DataSink * Unify write_all for all FileSink implementers * DRY builder/header fetch * Remove more duplication * enrich documentation for spawn_writer_tasks_and_join * fix cargo doc --------- Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com> * Add sqlite sqllogictest run to extended.yml for running sqlite test suite against every push to main. (#14101) * Fix duplicated SharedBitmapBuilder definitions (#14122) * feat: add `alias()` method for DataFrame (#14127) * feat: add `alias()` method for DataFrame * doc-gen: make user_doc to work with predefined consts (#14086) * Minor: Document the rationale for the lack of Cargo.lock (#14071) * Minor: Document the rationale for the lack of Cargo.lock * Update README.md --------- Co-authored-by: Oleks V <comphead@users.noreply.github.com> * Return err if wildcard is not expanded before type coercion (#14130) * Return err if wildcard is not expanded before type coercion * fix test * fix clippy * improve test --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> * Fix combine with session config (#14139) * fix combine with session config * must use it * Minor: move resolve_overlap a method on OrderingEquivalenceClas (#14138) * doc-gen: migrate scalar functions (encoding & regex) documentation (#13919) * doc-gen: migrate scalar functions (encoding & regex) documentation * fix: fix typo * doc: fix typo --------- Co-authored-by: Cheng-Yuan-Lai <a186235@g,ail.com> * bugfix: create view with multi union may get wrong schema (#14132) (#14133) create view with analyzer rule of TypeCoercion Co-authored-by: chenmch <chenmch@diit.cn> * Deduplicate function `get_final_indices_from_shared_bitmap` (#14145) * Deduplicate function get_final_indices_from_shared_bitmap * update * Add tests for PR #14133 (view with multi unions) (#14152) * bugfix: create view with multi union may get wrong schema (#14132) create view with analyzer rule of TypeCoercion * test: Add tests for PR #14133 (create view with multi unions) --------- Co-authored-by: chenmch <chenmch@diit.cn> * Update datafusion-testing git hash (#14137) * Reuse `on` expressions values in HashJoinExec (#14131) * Reduce duplicated build side experssions evaluations in HashJoinExec * Reuse probe side on expressions values * fix: encode should work with non-UTF-8 binaries (#14087) * fix: encode function should work with strings and binary closes #14055 * chore: address comments, add test * chore: move `SanityChecker` into `physical-optimizer` crate (#14083) * chore: move into crate * chore: move SanityChecker tests out to datafusion/core/tests * chore: update datafusion-cli/Cargo.lock * fix cargo doc --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> * Propagate table constraints through physical plans to optimize sort operations (#14111) * Add projection to `Constraints` * Add constraints support to `EquivalenceProperties` * Pass constraints to physical plan * Add slt test for primary key sort optimization * Pass constraints to MemoryExec * Update properties.rs * Simplify MemoryExec instantiation * Rename EquivalenceProperties method name for clarity * Refactor projection handling in FileScanConfig * Bug fix * Display constraints on data sources * Bug fix and test improvements * Use different schemas for tests * Lint and visibility fix * Fixes after merge * Review part 1 * Update memory.rs * update dep * update proto * add aggregate distinct * minor * Update order.slt * undo proto * Update properties.rs * Move reserved entry * Update `FileScanConfig` to return a single projected configuration object * Improve constraint based ordering satisfaction logic * Update datafusion/physical-plan/src/aggregates/mod.rs Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com> * Revert "Update `FileScanConfig` to return a single projected configuration object" This reverts commit bbe35d48fb5c4af573fdf0ef81375ea0c72c0327. * Refactor MemoryExec constraints display Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com> * Avoid unnecessary clone * Refactor constraint based ordering satisfaction logic * Cargo fmt * Revert "Avoid unnecessary clone" This reverts commit ab93279287311e4f6b5239fde2e8f98a22141c54. * Avoid unnecessary clone * Update properties.rs * Bug fix * Make `update_elements_with_matching_indices` take iterators for proj_indices * Revert "Make `update_elements_with_matching_indices` take iterators for proj_indices" This reverts commit d136860e2eb5054bd0a57588ea62337aa2712035. --------- Co-authored-by: berkaysynnada <berkay.sahin@synnada.ai> Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com> * NestedLoopJoin Projection Pushdown (#14120) * nlj proj pushdown Signed-off-by: Jay Zhan <jay.zhan@synnada.ai> * fmt Signed-off-by: Jay Zhan <jay.zhan@synnada.ai> * move swap proj to util Signed-off-by: Jay Zhan <jay.zhan@synnada.ai> * fmt Signed-off-by: Jay Zhan <jay.zhan@synnada.ai> * fix proto Signed-off-by: Jay Zhan <jay.zhan@synnada.ai> * fmt Signed-off-by: Jay Zhan <jay.zhan@synnada.ai> * use none Signed-off-by: Jay Zhan <jay.zhan@synnada.ai> * proto fix Signed-off-by: Jay Zhan <jay.zhan@synnada.ai> * fix slt Signed-off-by: Jay Zhan <jay.zhan@synnada.ai> * Update projection_pushdown.rs * refactor: streamline projection pushdown logic for join operations * minor * fmt Signed-off-by: Jay Zhan <jay.zhan@synnada.ai> --------- Signed-off-by: Jay Zhan <jay.zhan@synnada.ai> Co-authored-by: berkaysynnada <berkay.sahin@synnada.ai> * Fix: regularize order bys when consuming from substrait (#14125) * Fix: regularize order bys when consuming from substrait * Add window_function_with_range_unit_and_no_order_by test * Fix typo in comment * Remove dependency on physical-optimizer on functions-aggregates (#14134) * Remove dependency on physical-optimizer on functions-aggregates * update lock * doc-gen: migrate scalar functions (other, conditional, and struct) documentation (#14163) * chore: fix flaky tests (#14170) * Upgrade arrow-rs, parquet to `54.0.0` and pyo3 to `0.23.3` (#14153) * Upgrade arrow-rs, parquet and pyo3 * Fix fmt CI * Simplify Bloom Filter Check (#14165) * Make `LexOrdering::inner` non pub, add comments, update usages (#14155) * Fix doctests in ScalarValue (#14164) (#14178) Co-authored-by: Sergey Zhukov <szhukov@aligntech.com> * Add `ScalarValue::try_as_str` to get str value from logical strings (#14167) * fix: handle scalar predicates in CASE expressions to prevent internal errors for InfallibleExprOrNull eval method (#14156) * fix: handle scalar predicates in CASE expressions to prevent internal errors for InfallibleExprOrNull eval method * Update to latest datafusion-testing commit --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> * Minor: Consolidate dataframe tests into core_integration (#14169) * Cut/paste dataframe tests to integration * Fix test issues * clippy * Add a hint about expected extension in error message in register_csv,… (#14168) * Add a hint about expected extension in error message in register_csv, register_parquet, register_json, register_avro (#14144) * Add tests for error * fix test * fmt * Fix issues causing GitHub checks to fail * revert datafusion-testing change --------- Co-authored-by: Sergey Zhukov <szhukov@aligntech.com> Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> * Test: Validate memory limit for sort queries to extended test (#14142) * External memory limit validation for sort * add bug tracker * cleanup * Update submodule * reviews * fix CI * move feature to module level * refactor: switch BooleanBufferBuilder to NullBufferBuilder in sort function (#14183) Co-authored-by: Cheng-Yuan-Lai <a186235@g,ail.com> * Add section to howtos.md (#14171) * refactor: switch BooleanBufferBuilder to NullBufferBuilder in correlation function (#14181) Co-authored-by: Cheng-Yuan-Lai <a186235@g,ail.com> * Rename extended test job name (#14199) * Added job board as a separate header in the documentation (#14191) * Added job board as a separate header in the documentation * Update docs/source/contributor-guide/communication.md Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> * Update docs/source/contributor-guide/communication.md Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> * prettier --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> * Support spaceship operator (`<=>`) support (alias for `IS NOT DISTINCT FROM` (#14187) * Mapped the Spaceship operator with IsNotDistinctFrom * Added tests for Spaceship Operator <=> * Added sanity test for Spaceship Operator <=> * Add benchmark for planning sorted unions (#14157) * feat: Use `SchemaRef` in `JoinFilter` (#14182) * feat: Use `SchemaRef` in `JoinFilter` * Update datafusion/core/src/physical_optimizer/projection_pushdown.rs Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> * Update datafusion/physical-plan/src/joins/join_filter.rs Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> * Update datafusion/physical-plan/src/joins/join_filter.rs Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> * Update datafusion/physical-plan/src/joins/join_filter.rs Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> * fix --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> * refactor: switch BooleanBufferBuilder to NullBufferBuilder in functions-nested functions (#14201) Co-authored-by: Cheng-Yuan-Lai <a186235@g,ail.com> * Improve case expr constant handling, Add .slt test (#14159) * Minor add ticket references to deprecated code (#14174) --------- Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: Tai Le Manh <manhtai.lmt@gmail.com> Signed-off-by: zjregee <zjregee@gmail.com> Signed-off-by: MBWhite <whitemat@uk.ibm.com> Signed-off-by: jayzhan211 <jayzhan211@gmail.com> Signed-off-by: Jay Zhan <jayzhan211@gmail.com> Signed-off-by: Jay Zhan <jay.zhan@synnada.ai> Co-authored-by: Eason <30045503+Eason0729@users.noreply.github.com> Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> Co-authored-by: Piotr Findeisen <piotr.findeisen@gmail.com> Co-authored-by: Jax Liu <liugs963@gmail.com> Co-authored-by: Marc Droogh <33723117+mdroogh@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Marc Droogh <marc.droogh@imc.com> Co-authored-by: Berkay Şahin <124376117+berkaysynnada@users.noreply.github.com> Co-authored-by: Yongting You <2010youy01@gmail.com> Co-authored-by: Onur Satici <onursatici@users.noreply.github.com> Co-authored-by: Tai Le Manh <manhtai.lmt@gmail.com> Co-authored-by: zjregee <zjregee@gmail.com> Co-authored-by: Jay Zhan <jay.zhan@synnada.ai> Co-authored-by: Oleks V <comphead@users.noreply.github.com> Co-authored-by: Rohan Krishnaswamy <47869999+rkrishn7@users.noreply.github.com> Co-authored-by: Jonah Gao <jonahgao@msn.com> Co-authored-by: Matthew B White <matthew@mh-white.com> Co-authored-by: Jay Zhan <jayzhan211@gmail.com> Co-authored-by: Raz Luvaton <16746759+rluvaton@users.noreply.github.com> Co-authored-by: Jack <56563911+jdockerty@users.noreply.github.com> Co-authored-by: Burak Şen <buraksenb@gmail.com> Co-authored-by: Qi Zhu <821684824@qq.com> Co-authored-by: Kyle Barron <kylebarron2@gmail.com> Co-authored-by: Eduard Karacharov <eduard.karacharov@gmail.com> Co-authored-by: cht42 <42912042+cht42@users.noreply.github.com> Co-authored-by: Cyprien Huet <chuet@palantir.com> Co-authored-by: mertak-synnada <mertak67+synaada@gmail.com> Co-authored-by: wiedld <wiedld@users.noreply.github.com> Co-authored-by: Arttu <Blizzara@users.noreply.github.com> Co-authored-by: Daniel Hegberg <daniel.hegberg@gmail.com> Co-authored-by: Costi Ciudatu <ccciudatu@gmail.com> Co-authored-by: Weston Pace <weston.pace@gmail.com> Co-authored-by: Alex Kesling <alex@kesling.co> Co-authored-by: Goksel Kabadayi <45314116+gokselk@users.noreply.github.com> Co-authored-by: berkaysynnada <berkay.sahin@synnada.ai> Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com> Co-authored-by: Adrian Garcia Badaracco <1755071+adriangb@users.noreply.github.com> Co-authored-by: Chunchun Ye <14298407+appletreeisyellow@users.noreply.github.com> Co-authored-by: Arttu Voutilainen <avo@iki.fi> Co-authored-by: Dmitrii Blaginin <dmitrii@blaginin.me> Co-authored-by: zhuliquan <zlqlovecode@foxmail.com> Co-authored-by: Bruce Ritchie <bruce.ritchie@veeva.com> Co-authored-by: Victor Barua <victor.barua@datadoghq.com> Co-authored-by: robtandy <rob.tandy@gmail.com> Co-authored-by: Jack Park <xarus01@gmail.com> Co-authored-by: UBarney <UBarney@users.noreply.github.com> Co-authored-by: Zhang Li <richselian@gmail.com> Co-authored-by: zhangli20 <zhangli20@kuaishou.com> Co-authored-by: Spaarsh <67336892+Spaarsh@users.noreply.github.com> Co-authored-by: xudong.w <wxd963996380@gmail.com> Co-authored-by: Namgung Chan <33323415+getChan@users.noreply.github.com> Co-authored-by: Tim Saucer <timsaucer@gmail.com> Co-authored-by: Takahiro Ebato <takahiro.ebato@gmail.com> Co-authored-by: Alihan Çelikcan <alihan.celikcan@synnada.ai> Co-authored-by: Ian Lai <108986288+Chen-Yuan-Lai@users.noreply.github.com> Co-authored-by: Cheng-Yuan-Lai <a186235@g,ail.com> Co-authored-by: delamarch3 <68732277+delamarch3@users.noreply.github.com> Co-authored-by: Andrew Kane <andrew@ankane.org> Co-authored-by: Matthew Turner <matthew.m.turner@outlook.com> Co-authored-by: Andre Weltsch <aweltsch@users.noreply.github.com> Co-authored-by: Sergey Zhukov <62326549+cj-zhukov@users.noreply.github.com> Co-authored-by: Sergey Zhukov <szhukov@aligntech.com> Co-authored-by: Aleksey Kirilishin <54231417+avkirilishin@users.noreply.github.com> Co-authored-by: irenjj <renj.jiang@gmail.com> Co-authored-by: Marko Milenković <milenkovicm@users.noreply.github.com> Co-authored-by: Eugene Marushchenko <maruschin@gmail.com> Co-authored-by: Lordworms <48054792+Lordworms@users.noreply.github.com> Co-authored-by: Jeffrey Vo <jeffrey.vo.australia@gmail.com> Co-authored-by: Dharan Aditya <dharan.aditya@gmail.com> Co-authored-by: Vadim Piven <vadim@piven.tech> Co-authored-by: kosiew <kosiew@gmail.com> Co-authored-by: Tim Van Wassenhove <github@timvw.be> Co-authored-by: nuno-faria <nunofpfaria@gmail.com> Co-authored-by: Mohamed Abdeen <83442793+MohamedAbdeen21@users.noreply.github.com> Co-authored-by: Sergei Grebnov <sergei.grebnov@gmail.com> Co-authored-by: Wendell Smith <wendell.smith@datadoghq.com> Co-authored-by: niebayes <niebayes@gmail.com> Co-authored-by: Matthijs Brobbel <m1brobbel@gmail.com> Co-authored-by: Mustafa Akur <akurmustafa@gmail.com> Co-authored-by: Jagdish Parihar <jatin6972@gmail.com> Co-authored-by: TheBuilderJR <46176773+TheBuilderJR@users.noreply.github.com> Co-authored-by: Jonathan Chen <chenleejonathan@gmail.com> Co-authored-by: Xiangpeng Hao <haoxiangpeng123@gmail.com> Co-authored-by: 张林伟 <lewiszlw520@gmail.com> Co-authored-by: ding-young <lsyhime@snu.ac.kr> Co-authored-by: Will Golioto <36157286+Curricane@users.noreply.github.com> Co-authored-by: chenmch <chenmch@diit.cn> Co-authored-by: Daniel Mesejo <mesejoleon@gmail.com> Co-authored-by: Mrinal Paliwal <mrinal16164@iiitd.ac.in> Co-authored-by: Gabriel <45515538+gabotechs@users.noreply.github.com> Co-authored-by: Owen Leung <owen.leung2@gmail.com> Co-authored-by: Edmondo Porcu <edmondo.porcu@gmail.com>

2025-01-16

NestedLoopJoin Projection Pushdown (#14120) * nlj proj pushdown Signed-off-by: Jay Zhan <jay.zhan@synnada.ai> * fmt Signed-off-by: Jay Zhan <jay.zhan@synnada.ai> * move swap proj to util Signed-off-by: Jay Zhan <jay.zhan@synnada.ai> * fmt Signed-off-by: Jay Zhan <jay.zhan@synnada.ai> * fix proto Signed-off-by: Jay Zhan <jay.zhan@synnada.ai> * fmt Signed-off-by: Jay Zhan <jay.zhan@synnada.ai> * use none Signed-off-by: Jay Zhan <jay.zhan@synnada.ai> * proto fix Signed-off-by: Jay Zhan <jay.zhan@synnada.ai> * fix slt Signed-off-by: Jay Zhan <jay.zhan@synnada.ai> * Update projection_pushdown.rs * refactor: streamline projection pushdown logic for join operations * minor * fmt Signed-off-by: Jay Zhan <jay.zhan@synnada.ai> --------- Signed-off-by: Jay Zhan <jay.zhan@synnada.ai> Co-authored-by: berkaysynnada <berkay.sahin@synnada.ai>

Propagate table constraints through physical plans to optimize sort operations (#14111) * Add projection to `Constraints` * Add constraints support to `EquivalenceProperties` * Pass constraints to physical plan * Add slt test for primary key sort optimization * Pass constraints to MemoryExec * Update properties.rs * Simplify MemoryExec instantiation * Rename EquivalenceProperties method name for clarity * Refactor projection handling in FileScanConfig * Bug fix * Display constraints on data sources * Bug fix and test improvements * Use different schemas for tests * Lint and visibility fix * Fixes after merge * Review part 1 * Update memory.rs * update dep * update proto * add aggregate distinct * minor * Update order.slt * undo proto * Update properties.rs * Move reserved entry * Update `FileScanConfig` to return a single projected configuration object * Improve constraint based ordering satisfaction logic * Update datafusion/physical-plan/src/aggregates/mod.rs Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com> * Revert "Update `FileScanConfig` to return a single projected configuration object" This reverts commit bbe35d48fb5c4af573fdf0ef81375ea0c72c0327. * Refactor MemoryExec constraints display Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com> * Avoid unnecessary clone * Refactor constraint based ordering satisfaction logic * Cargo fmt * Revert "Avoid unnecessary clone" This reverts commit ab93279287311e4f6b5239fde2e8f98a22141c54. * Avoid unnecessary clone * Update properties.rs * Bug fix * Make `update_elements_with_matching_indices` take iterators for proj_indices * Revert "Make `update_elements_with_matching_indices` take iterators for proj_indices" This reverts commit d136860e2eb5054bd0a57588ea62337aa2712035. --------- Co-authored-by: berkaysynnada <berkay.sahin@synnada.ai> Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com>

2025-01-15

Chore: refactor DataSink traits to avoid duplication (#14121) * add some abstractions to file sinkers and centralize FileSinkConfig based behaviors * satisfy clippy * typo fix * move start_demuxer_task back into demux.rs add file_extension to FileSinkConfig * fix errors * merge get_writer_schema functions add schema() function to DataSink trait make FileSink a subtrait for DataSink * Unify write_all for all FileSink implementers * DRY builder/header fetch * Remove more duplication * enrich documentation for spawn_writer_tasks_and_join * fix cargo doc --------- Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com>

2025-01-13

feat: add support for `LogicalPlan::DML(...)` serde (#14079) * Add support for DML serialization to proto closes: #13616 * add round trip test for DML serde * cover all cases in round trip test * minor: change ordering of enum type

2025-01-01

Supporting writing schema metadata when writing Parquet in parallel (#13866) * refactor: make ParquetSink tests a bit more readable * chore(11770): add new ParquetOptions.skip_arrow_metadata * test(11770): demonstrate that the single threaded ParquetSink is already writing the arrow schema in the kv_meta, and allow disablement * refactor(11770): replace with new method, since the kv_metadata is inherent to TableParquetOptions and therefore we should explicitly make the API apparant that you have to include the arrow schema or not * fix(11770): fix parallel ParquetSink to encode arrow schema into the file metadata, based on the ParquetOptions * refactor(11770): provide deprecation warning for TryFrom * test(11770): update tests with new default to include arrow schema * refactor: including partitioning of arrow schema inserted into kv_metdata * test: update tests for new config prop, as well as the new file partition offsets based upon larger metadata * chore: avoid cloning in tests, and update code docs * refactor: return to the WriterPropertiesBuilder::TryFrom<TableParquetOptions>, and separately add the arrow_schema to the kv_metadata on the TableParquetOptions * refactor: require the arrow_schema key to be present in the kv_metadata, if is required by the configuration * chore: update configs.md * test: update tests to handle the (default) required arrow schema in the kv_metadata * chore: add reference to arrow-rs upstream PR

2024-12-19

[bugfix] ScalarFunctionExpr does not preserve the nullable flag on roundtrip (#13830) * [test] coalesce round trip schema mismatch * [proto] added the nullable flag in PhysicalScalarUdfNode * [bugfix] propagate the nullable flag for serialized scalar UDFS

2024-12-18

Support Null regex override in csv parser options. (#13228) Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>

2024-11-30

support unknown col expr in proto (#13603)

2024-11-15

fix: serialize user-defined window functions to proto (#13421) * Adds roundtrip physical plan test * Adds enum for udwf to `WindowFunction` * initial fix for serializing udwf * Revives deleted test * Adds codec methods for physical plan * Rewrite error message * Minor: rename binding + formatting fixes * Extends `PhysicalExtensionCodec` for udwf * Minor: formatting * Restricts visibility to tests

Remove `BuiltInWindowFunction` (LogicalPlans) (#13393) * Remove BuiltInWindowFunction * fix docs * Fix typo

2024-11-13

Convert `nth_value` builtIn function to User Defined Window Function (#13201) * refactored nth_value * continue * test * proto and rustlint * fix datatype * cont * cont * apply jcsherins early validation * docs * doc * Apply suggestions from code review Co-authored-by: Sherin Jacob <jacob@protoship.io> * passes lint but does not have tests * continue * Update roundtrip_physical_plan.rs * udwf, not udaf * fix bounded but not fixed roundtrip * added * Update datafusion/sqllogictest/test_files/errors.slt Co-authored-by: Sherin Jacob <jacob@protoship.io> --------- Co-authored-by: Sherin Jacob <jacob@protoship.io> Co-authored-by: berkaysynnada <berkay.sahin@synnada.ai> Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>

2024-11-12

support recursive CTEs logical plans in datafusion-proto (#13314) * support LogicaPlan::RecursiveQuery in datafusion-proto * fixed and failing test roundtrip_recursive_query * fix rebase artifact * add node for CteWorkTableScan in datafusion-proto * Use Arc::clone --------- Co-authored-by: jonahgao <jonahgao@msn.com>

2024-11-06

Change `schema_infer_max_rec ` config to use `Option<usize>` rather than `usize` (#13250) * Make schema_infer_max_rec an Option * Add lifetime parameter to CSV and compression BoxStreams

2024-11-05

fix bugs explain with non-correlated query (#13210) * fix bugs explain with non-correlated query * Use explicit enum for physical errors * fix comments / fmt * strip_backtrace to passs ci --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>

2024-10-31

feat: Implement LeftMark join to fix subquery correctness issue (#13134) * Implement LeftMark join In https://github.com/apache/datafusion/pull/12945 the emulation of an mark join has a bug when there is duplicate values in the subquery. This would be fixable by adding a distinct before the join. But this patch instead implements a LeftMark join with the desired semantics and uses that. The LeftMark join will return a row for each in the left input with an additional column "mark" that is true if there was a match in the right input and false otherwise. Note: This patch does not implement the full null semantics for the mark join described in http://btw2017.informatik.uni-stuttgart.de/slidesandpapers/F1-10-37/paper_web.pdf which which will be needed if we and `ANY` subqueries. The version is this patch the mark column will only be true for had a match and false when no match was found, never `null`. * Use mark join in decorrelate subqueries This fixes a correctness issue in the current approach. * Add physical plan sqllogictest * fmt * Fix join type in doc comment * Minor clean ups * Add more documentation to LeftMark join * Remove qualification * fix doc --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>

2024-10-25

Convert `ntile` builtIn function to UDWF (#13040) * converting to ntile udwf * updated the window functions documentation file * wip: update the ntile udwf function * fix the roundtrip_logical_plan.rs * removed builtIn ntile function * fixed field name issue * fixing the return type of ntile udwf * error if UInt64 conversion fails * handling if null is found * handling if value is zero or less than zero * removed unused import * updated prost.rs file * removed dead code * fixed clippy error * added inner doc comment * minor fixes and added roundtrip logical plan test * removed parse_expr in ntile

Introduce `binary_as_string` parquet option, upgrade to arrow/parquet `53.2.0` (#12816) * Update to arrow-rs 53.2.0 * introduce binary_as_string parquet option * Fix test --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>

2024-10-23

feat: Convert CumeDist to UDWF (#13051) * Transferred cumedist * fixes * remove expr tests * small fix * small fix * check * clippy fix * roundtrip fix