These commits are when the Protocol Buffers files have changed: (only the last 100 relevant commits are shown)
| Commit: | dbedb1e | |
|---|---|---|
| Author: | Ben Bellick | |
fix(table-functions): validate concrete schema output types
| Commit: | 68be1dc | |
|---|---|---|
| Author: | YongChul Kwon | |
feat(window): add an explicit RANGE boundary overflow policy Add one shared UNSPECIFIED/ERROR/EXTEND enum and a typed field to both window invocation forms. Keep frame policy independent of aggregate options and define rejection and default behavior. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
| Commit: | 0222b91 | |
|---|---|---|
| Author: | YongChul Kwon | |
refactor(window): split RANGE compatibility from boundary policies Keep non-floating-point RANGE compatibility and in-range membership in this proposal. Allow mixed integer widths and UTC calendar months. Move ROWS arithmetic, exact floating-point membership, and explicit boundary overflow to separate proposals. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
| Commit: | ee6c682 | |
|---|---|---|
| Author: | YongChul Kwon | |
feat: specify UUID literal byte order and test syntax Define network byte order without restricting UUID version or variant bits, and support unquoted UUID literals throughout the function-test grammar. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
| Commit: | 3cddb53 | |
|---|---|---|
| Author: | YongChul Kwon | |
feat(proto): use plan-level detachment for recursive lambdas Merge the updated detachment proposal and move recursive examples to Plan tables. Preserve accepted lambda documentation and distinguish global ordinal ownership from lexical parameter scope. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
| Commit: | 1d6bc32 | |
|---|---|---|
| Author: | YongChul Kwon | |
feat(proto): move global detachment tables to Plan Allow named lambdas and other plan-wide entries to use plan-level tables while preserving PlanRel and ExtendedExpression locality. Validate single-use detached entries across all plan-wide roots without changing lexical scope. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
| Commit: | 852dc57 | |
|---|---|---|
| Author: | Ben Bellick | |
fix(table-functions): use conventional advanced extension field number
| Commit: | d235d2e | |
|---|---|---|
| Author: | Ben Bellick | |
docs(table-functions): require constant value arguments
| Commit: | 40a136a | |
|---|---|---|
| Author: | YongChul Kwon | |
feat(proto): allow recursive named lambdas Declare return types for named-lambda dependency cycles so self-recursion and mutual recursion can be type-checked without expanding definitions. Preserve lexical scope and distinguish recursive calls from cycles in detached encoding. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
| Commit: | dddf77b | |
|---|---|---|
| Author: | YongChul Kwon | |
feat(proto): support detached named-lambda bodies Give each named lambda local expression and relation tables to bound protobuf nesting without changing evaluation or lexical scope. Clarify subquery-local bindings while preserving the prohibition on caller-scope capture and cyclic named-lambda dependencies. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
| Commit: | e7e214f | |
|---|---|---|
| Author: | YongChul Kwon | |
feat(proto): support recursive named lambdas and detached bodies Allow named lambdas to call themselves or each other using declared return types to break typing cycles. Add definition-local detachment tables so expression and subquery bodies can be encoded without deep protobuf nesting. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
| Commit: | 53e3e49 | |
|---|---|---|
| Author: | YongChul Kwon | |
feat(window)!: define RANGE compatibility and boundary semantics Define core RANGE membership and configurable boundary overflow independently of scalar arithmetic functions. Specify mathematical ROWS boundaries and distinguish frame-level options from function-defined dialect options. BREAKING CHANGE: Existing RANGE plans may change validity or frame membership, including finite floating-point cases without overflow. ROWS boundary arithmetic must not wrap, clamp to existing rows, or fail because of internal integer limits. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
| Commit: | 2a16cf8 | |
|---|---|---|
| Author: | Aleksandr Efimov | |
| Committer: | GitHub | |
docs!: output a mask's selected fields in the order it lists them (#1247) A mask now outputs the fields it selects in the order it lists them. `field_references.md` says so instead of keeping it as a discussion point, and the `MaskExpression` comment in `algebra.proto` no longer says a mask only removes elements. This is the reading proposed in #1225: DuckDB writes `SELECT c2, c0 FROM t_mix` as one read masking `[2, 0]`, and substrait-java, substrait-validator, DataFusion and DuckDB all read it in that order. A mask in schema order behaves as before. BREAKING CHANGE: Masks output fields in selection order. For a schema [c0, c1, c2], a mask [2, 0] outputs [c2, c0]. Consumers that use schema order must change; masks already in schema order are unaffected. Closes #1225 <!-- Reviewable:start --> - - - This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/substrait-io/substrait/1247) <!-- Reviewable:end --> --------- Co-authored-by: YongChul Kwon <1606237+yongchul@users.noreply.github.com>
The documentation is generated from this commit.
| Commit: | 2ff6384 | |
|---|---|---|
| Author: | Ben Bellick | |
docs(table-functions): remove evaluation context wording
| Commit: | 3df0858 | |
|---|---|---|
| Author: | Ben Bellick | |
refactor(table-functions): use a struct output schema
| Commit: | 1fc9b53 | |
|---|---|---|
| Author: | Ben Bellick | |
fix(table-functions): define nullability without scalar handling modes
| Commit: | 0192cba | |
|---|---|---|
| Author: | Ben Bellick | |
docs(table-functions): trim redundant explanations and assertions
| Commit: | fda6ccf | |
|---|---|---|
| Author: | Ben Bellick | |
docs(table-functions): describe current behavior without future operators
| Commit: | 1ac962c | |
|---|---|---|
| Author: | Ben Bellick | |
docs(table-functions): remove redundant scoping comment
| Commit: | 2ebfb3f | |
|---|---|---|
| Author: | Ben Bellick | |
refactor(table-functions): remove the table expression wrapper
| Commit: | 62af569 | |
|---|---|---|
| Author: | Ben Bellick | |
feat(table-functions): introduce shared table function calls Separate the callable representation from standalone relational invocation so per-row operators can reuse it. Preserve producer-resolved output schemas and support YAML function options. Use positional struct outputs, validate documentation examples, and generate table function catalog documentation.
| Commit: | b721457 | |
|---|---|---|
| Author: | Ben Bellick | |
feat(table-functions): restore simple table function proposal Recover the proposal from #876 on current main. Define explicit unnest overloads for one through four lists, leaving arbitrary-arity output derivation for future work.
| Commit: | 70e6b61 | |
|---|---|---|
| Author: | YongChul Kwon | |
| Committer: | GitHub | |
feat(proto): support named lambdas (#1221) Allow plans to define typed lambda expressions once, invoke them by anchor from multiple call sites, and pass them to higher-order functions. This supports shared expression subtrees and functions implemented directly in Substrait expressions without requiring an external extension implementation. Named lambdas are declared in `Plan.named_lambdas` using `NamedLambda`. `LambdaInvocation` selects either an inline `lambda` or a `named_lambda` containing a `NamedLambdaReference` through a `oneof`, since both invocation forms remain valid and are mutually exclusive. Named-lambda anchors use a namespace separate from extension functions. `Expression.NamedLambdaReference` denotes a function value without invoking it. Its non-nullable function type is derived from the referenced lambda's parameter and body types, allowing functions such as `transform` and `filter` to accept named lambdas directly without an inline forwarding lambda. Existing inline lambda definitions keep their parameters and body together. Named lambdas are closed over their parameters, and references do not capture the caller's scope. The definition dependency graph must be acyclic, counting both direct invocations and function-valued references, including references nested in higher-order function arguments. This lets consumers execute invocations directly or inline their definitions without recursive expansion. For a non-empty dialect `supported_expressions` list, `LAMBDA_INVOCATION` alone advertises inline calls. Declaring both `LAMBDA_INVOCATION` and `NAMED_LAMBDA_REFERENCE` also advertises direct named calls, without a separate callee-type flag. Reference-only declarations remain valid for passing named lambdas to supported higher-order functions. These rules apply to both shorthand and object entries. An absent or empty list retains its existing meaning of unrestricted expression support, including both invocation forms. Closes #1220 BREAKING CHANGE: Moving `LambdaInvocation.lambda` into a `oneof` preserves the wire encoding of existing inline-lambda invocations but may break generated source APIs requiring code change around `oneof` access. <!-- Reviewable:start --> - - - This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/substrait-io/substrait/1221) <!-- Reviewable:end --> <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Plans can define reusable named lambdas and invoke them by reference or inline, with structured arguments. * **Documentation** * Updated lambda and serialization guidance to explain named lambdas, references, and invocation requirements. * **Examples** * Added an example that reuses one named identity lambda for multiple invocations. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Ben Bellick <36523439+benbellick@users.noreply.github.com> Co-authored-by: Niels Pardon <mail@niels-pardon.de>
| Commit: | 60d51cb | |
|---|---|---|
| Author: | YongChul Kwon | |
| Committer: | GitHub | |
docs(spec): clarify window function concepts and frame semantics (#1246) Organize window-function documentation around the core concepts of partitions, peer groups (rows sharing the same ordering values), and per-row window frames. Use a shared dataset to compare ROWS and RANGE frames with CurrentRow, Preceding, and Following bounds, making partition boundaries and the effect of tied ordering values concrete. Following Victor's guidance in #1105, group invariants by bound mode and make protobuf comments link to the documentation instead of duplicating the detailed rules. Clarify that functions sharing partitioning and ordering can still have different frames. Preserve the existing RANGE compatibility rule: `add(T, D) -> T` and `subtract(T, D) -> T` must be defined. Replace the obsolete `timestamp` example with `precision_timestamp<P>` / `interval_day<P>` and remove the duplicated examples from protobuf comments. No protobuf fields or wire layout change. Changes to RANGE type compatibility, including widened boundary types, additional arithmetic arguments, and selection among extension declarations, are deferred to #1227. This PR does not resolve or close that issue. Fixes #1182. Fixes #1226. <!-- Reviewable:start --> - - - This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/substrait-io/substrait/1246) <!-- Reviewable:end --> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Niels Pardon <mail@niels-pardon.de> Co-authored-by: Victor Barua <victor.barua@datadoghq.com>
| Commit: | aee4477 | |
|---|---|---|
| Author: | YongChul Kwon | |
| Committer: | GitHub | |
Apply batched suggestions from code review Co-authored-by: Victor Barua <victor.barua@datadoghq.com>
| Commit: | 554bbfc | |
|---|---|---|
| Author: | YongChul Kwon | |
| Committer: | GitHub | |
feat(proto): support detached relation subtrees (#1236) Builds on the detached-expression encoding merged in #1218. Extend the `PlanRel`-local bounded-depth encoding to relation trees. Producers can move relation subtrees into `PlanRel.detached_rels` and replace them with `Rel.detached_rel_ordinal` references. Detached relations are encoding artifacts and do not change plan semantics. As with detached expressions, consumers validate only the graph reachable from the containing `PlanRel` entry point and extension payloads they interpret. Reachable ordinals must be in range and container-local, each detached entry may be referenced at most once within that graph, and references must be acyclic. A reachable detached entry cannot itself be an ordinal; its children may be. `ReferenceRel` remains the mechanism for intentional relation sharing. Unreachable detached entries are allowed and need not be resolved or validated. This lets producers prune subtrees without compacting detached tables or rewriting surviving ordinals. Unused entries are not additional execution roots; consumers retaining opaque extension payloads must preserve references inside them when removing or renumbering entries. ### Why not `ReferenceRel`? `ReferenceRel` creates a plan-wide dependency intended for relation sharing. Using it only for encoding chunking makes an extracted `PlanRel` depend on other `Plan.relations` entries and requires dependency-closure tracking and ordinal rewriting when unrelated entries are removed or reordered. Keeping detached relations local makes each `PlanRel` self-contained and independently transportable. The protobuf additions are backward-compatible for existing plans. Plans using detached relation references require consumers that understand the new encoding because older consumers will see an unset relation variant. Closes #1217. <!-- Reviewable:start --> - - - This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/substrait-io/substrait/1236) <!-- Reviewable:end --> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: Niels Pardon <mail@niels-pardon.de>
| Commit: | 8234616 | |
|---|---|---|
| Author: | YongChul Kwon | |
feat(proto): unify lambda invocation callees Invoke non-nullable function-valued expressions, including named lambdas, parameters, and returned functions. Retain the existing inline callee as a deprecated field with dual-write migration rules, and declare unified invocation support in dialects. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
| Commit: | 8a40ce1 | |
|---|---|---|
| Author: | YongChul Kwon | |
fix(proto): preserve detached expressions after merge Restore detached_expression_ordinal at field 18 and assign the new named_lambda_reference field 19 to avoid the collision with main. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
| Commit: | c1b046f | |
|---|---|---|
| Author: | YongChul Kwon | |
| Committer: | GitHub | |
Merge branch 'main' into yongchul/issue1220
| Commit: | 6438487 | |
|---|---|---|
| Author: | YongChul Kwon | |
| Committer: | GitHub | |
Apply batched suggestions from code review Co-authored-by: Niels Pardon <mail@niels-pardon.de>
| Commit: | 9b342e8 | |
|---|---|---|
| Author: | YongChul Kwon | |
feat(proto): support detached relations in extended expressions Allow expression-only subqueries to bound relation nesting using a container-local relation table. Require reachable detached entries to select a non-ordinal variant and complete the examples with schema and output names. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
| Commit: | 71693bc | |
|---|---|---|
| Author: | YongChul Kwon | |
| Committer: | GitHub | |
Apply batched suggestions from code review Co-authored-by: Niels Pardon <mail@niels-pardon.de>
| Commit: | 0987dd8 | |
|---|---|---|
| Author: | YongChul Kwon | |
docs: clarify detached relations and expressions Remove the unused detached relation from the example and the pruning guidance. Use a section heading and links that explicitly cover relations and expressions. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
| Commit: | 5d3c4d3 | |
|---|---|---|
| Author: | YongChul Kwon | |
refactor(proto): use named lambda reference messages in invocations Align invocation examples, documentation, and serialization tests with the shared reference message. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
| Commit: | 37891ab | |
|---|---|---|
| Author: | YongChul Kwon | |
fix(spec): align detached relation validation with expressions
| Commit: | 98bfa9e | |
|---|---|---|
| Author: | YongChul Kwon | |
| Committer: | YongChul Kwon | |
feat(proto): support detached relation subtrees
| Commit: | afe6283 | |
|---|---|---|
| Author: | Niels Pardon | |
| Committer: | GitHub | |
docs(protos): drop stale Expression.Enum and optional enum argument references (#1260)
| Commit: | 1b02337 | |
|---|---|---|
| Author: | YongChul Kwon | |
docs: address window function documentation review feedback Clarify shared ordering semantics and implicit offset presence, explain the removed decimal example, and reference the migration policy without duplicating its requirements. Remove trailing punctuation from window documentation URLs so generated Rust docs retain valid targets. Leave the proposed ROWS/i64-only legacy-equivalence restriction for further discussion. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
| Commit: | 6340159 | |
|---|---|---|
| Author: | YongChul Kwon | |
| Committer: | GitHub | |
Merge branch 'main' into yongchul/window-function-docs
| Commit: | 795e693 | |
|---|---|---|
| Author: | YongChul Kwon | |
| Committer: | GitHub | |
feat(proto): support detached expression subtrees (#1218) Allow producers to reduce protobuf expression nesting by moving subtrees into `PlanRel.detached_expressions` or `ExtendedExpression.detached_expressions` and replacing them with zero-based `Expression.detached_expression_ordinal` references. This supports both relational plans and expression-only protocols. Detachment is encoding-only, not computation sharing. Expressions inside `Any` extension payloads may reference the containing table. Consumers validate the expression trees reachable from the relation or referred expressions and interpreted extension payloads. In that reachable portion, references must be in range and acyclic, each entry may be referenced at most once, and entries cannot themselves be ordinals. Unreachable entries are allowed and need not be resolved or validated. Consumers retaining opaque extension payloads must preserve their reference targets. Expression-kind requirements apply after resolution. Detachment is an encoding capability rather than a dialect expression kind; encoding-capability negotiation is outside this PR. Detachment reduces expression-child nesting, not nesting within literals, types, field reference paths, or masks. Relation trees continue to use `ReferenceRel`. Closes #1217 <!-- Reviewable:start --> - - - This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/substrait-io/substrait/1218) <!-- Reviewable:end --> --------- Co-authored-by: Ben Bellick <36523439+benbellick@users.noreply.github.com> Co-authored-by: Niels Pardon <par@zurich.ibm.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
| Commit: | e5e1fbd | |
|---|---|---|
| Author: | YongChul Kwon | |
| Committer: | GitHub | |
feat(proto): support detachment in extended expressions Add expression-only detachment and dialect capability declarations. Allow unused entries while keeping references local, at most once, and acyclic. Clarify invalid encodings and consolidate the rules and examples.
| Commit: | aeff4a2 | |
|---|---|---|
| Author: | YongChul Kwon | |
| Committer: | GitHub | |
refactor(proto): compact detached expression field
| Commit: | 4fae3f1 | |
|---|---|---|
| Author: | YongChul Kwon | |
| Committer: | GitHub | |
refactor(proto): focus detachment on expressions
| Commit: | 6bf13a5 | |
|---|---|---|
| Author: | YongChul Kwon | |
| Committer: | GitHub | |
refactor(proto): inline detached subtree ordinals
| Commit: | 11ac672 | |
|---|---|---|
| Author: | YongChul Kwon | |
| Committer: | GitHub | |
feat(proto): support detached plan subtrees
| Commit: | 8ef1f7c | |
|---|---|---|
| Author: | YongChul Kwon | |
| Committer: | GitHub | |
Apply suggestion from @benbellick Co-authored-by: Ben Bellick <36523439+benbellick@users.noreply.github.com>
| Commit: | 9351854 | |
|---|---|---|
| Author: | YongChul Kwon | |
| Committer: | GitHub | |
feat(proto): support detachment in extended expressions Add expression-only detachment and dialect capability declarations. Allow unused entries while keeping references local, at most once, and acyclic. Clarify invalid encodings and consolidate the rules and examples.
| Commit: | 097d826 | |
|---|---|---|
| Author: | YongChul Kwon | |
| Committer: | GitHub | |
refactor(proto): compact detached expression field
| Commit: | 2b47591 | |
|---|---|---|
| Author: | YongChul Kwon | |
| Committer: | GitHub | |
refactor(proto): focus detachment on expressions
| Commit: | b40a0ee | |
|---|---|---|
| Author: | YongChul Kwon | |
| Committer: | GitHub | |
refactor(proto): inline detached subtree ordinals
| Commit: | 05a4b1c | |
|---|---|---|
| Author: | YongChul Kwon | |
| Committer: | GitHub | |
feat(proto): support detached plan subtrees
| Commit: | c1f3723 | |
|---|---|---|
| Author: | YongChul Kwon | |
fix(spec): clarify window frames and RANGE offset compatibility Explain partitions, peer groups, and frames with ROWS and RANGE examples, and centralize window invariants in the documentation as suggested in #1105. Allow widened RANGE boundary types while requiring two-argument arithmetic overloads. Document unsupported extra arguments and unresolved arithmetic selection from #1227. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
| Commit: | aa83bdb | |
|---|---|---|
| Author: | YongChul Kwon | |
feat(proto): add named lambda reference expressions Allow named lambdas to be passed as function-valued arguments without inline forwarding lambdas. Include function-valued references in the acyclic definition dependency graph. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
| Commit: | c3a15f4 | |
|---|---|---|
| Author: | YongChul Kwon | |
refactor(proto): model reusable lambdas as named lambdas Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
| Commit: | c52872c | |
|---|---|---|
| Author: | YongChul Kwon | |
| Committer: | GitHub | |
feat(proto): support detached relation subtrees
| Commit: | d38d8ca | |
|---|---|---|
| Author: | YongChul Kwon | |
| Committer: | GitHub | |
refactor(proto): compact detached expression field
| Commit: | 3b90302 | |
|---|---|---|
| Author: | YongChul Kwon | |
| Committer: | GitHub | |
refactor(proto): focus detachment on expressions
| Commit: | 3697eb4 | |
|---|---|---|
| Author: | YongChul Kwon | |
| Committer: | GitHub | |
refactor(proto): inline detached subtree ordinals
| Commit: | c4e9601 | |
|---|---|---|
| Author: | YongChul Kwon | |
| Committer: | GitHub | |
feat(proto): support detached plan subtrees
| Commit: | 0c266da | |
|---|---|---|
| Author: | YongChul Kwon | |
refactor(proto): clarify plan function definition naming Rename PlanFunction to PlanFunctionDefinition and describe the feature consistently as plan-defined functions.
| Commit: | c58cddd | |
|---|---|---|
| Author: | YongChul Kwon | |
feat(proto): name plan-scoped functions Add an optional diagnostic name that has no semantic identity and is not required to be unique.
| Commit: | f5134f7 | |
|---|---|---|
| Author: | YongChul Kwon | |
feat(proto): support plan-scoped functions Allow plans to define typed expression functions once and invoke them by anchor from multiple call sites. Plan-function anchors use a namespace separate from extension functions, and definitions remain inlineable through closed, acyclic references. Closes #1220
| Commit: | 0a97678 | |
|---|---|---|
| Author: | YongChul Kwon | |
refactor(proto): compact detached expression field
| Commit: | 1b5802e | |
|---|---|---|
| Author: | YongChul Kwon | |
refactor(proto): focus detachment on expressions
| Commit: | b7d147b | |
|---|---|---|
| Author: | YongChul Kwon | |
refactor(proto): inline detached subtree ordinals
| Commit: | 5f575b5 | |
|---|---|---|
| Author: | YongChul Kwon | |
feat(proto): support detached plan subtrees
| Commit: | 6aa8932 | |
|---|---|---|
| Author: | Ben Bellick | |
| Committer: | GitHub | |
docs: document protobuf field replacement migrations (#1181) Define a standard migration sequence for replacing protobuf fields. This keeps field documentation focused on field-specific behavior instead of repeating general migration guidance.
| Commit: | 48f6348 | |
|---|---|---|
| Author: | Ben Bellick | |
docs: centralize window bound migration guidance
| Commit: | 6706015 | |
|---|---|---|
| Author: | YongChul Kwon | |
| Committer: | GitHub | |
feat(protos)!: support expression in window aggregate bounds (#1105) # Motivation Like #748 in FetchRel, `offset` can be expressions in window spec. # Proposal * Extend the offset to expression, the same way we did for FetchRel. * deprecate offset from preceding and following message. * Clarify window frame bound documentation especially around window frame, ordering, and current rows. The proto and documentation changes closes #617 and #930 BREAKING CHANGE: deprecates the `offset` field of the `Preceding` and `Following` window bounds in favor of the new `offset_expr` field, which accepts an arbitrary expression instead of an integer literal. Consumers should add `offset_expr` support before producers switch to it. The PR also requires explicit window frame bounds type. Producers should set the bounds type and Consumers should reject `BOUNDS_TYPE_UNSPECIFIED` in such cases. <!-- Reviewable:start --> - - - This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/substrait-io/substrait/1105) <!-- Reviewable:end --> --------- Co-authored-by: Ben Bellick <36523439+benbellick@users.noreply.github.com> Co-authored-by: Niels Pardon <mail@niels-pardon.de> Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
| Commit: | 5970d5c | |
|---|---|---|
| Author: | YongChul Kwon | |
| Committer: | YongChul Kwon | |
fix new lines. mandate setting bounds type
| Commit: | 02977f4 | |
|---|---|---|
| Author: | YongChul Kwon | |
| Committer: | GitHub | |
Apply suggestions from code review Co-authored-by: Ben Bellick <36523439+benbellick@users.noreply.github.com> Co-authored-by: Niels Pardon <mail@niels-pardon.de>
| Commit: | 16ce6ed | |
|---|---|---|
| Author: | YongChul Kwon | |
| Committer: | GitHub | |
Merge branch 'main' into window_agg_exprs
| Commit: | 8fa4079 | |
|---|---|---|
| Author: | YongChul Kwon | |
| Committer: | GitHub | |
Apply suggestions from code review Co-authored-by: Ben Bellick <36523439+benbellick@users.noreply.github.com>
| Commit: | 25940fb | |
|---|---|---|
| Author: | Niels Pardon | |
| Committer: | GitHub | |
feat(protos)!: deprecate OuterReference.steps_out in favor of rel_reference (#1132) BREAKING CHANGE: `Expression.FieldReference.OuterReference.steps_out` is deprecated in favor of `rel_reference`, which names the binding relation via its plan-wide unique `RelCommon.rel_anchor` and therefore resolves unambiguously in DAG-shaped plans with shared relations (`ReferenceRel`), where counting subquery boundaries upward does not. Consumers should add `rel_reference` support before producers switch to it. Note that both fields are members of the same `outer_reference_type` oneof, so a plan cannot carry both forms at once: the migration is consumer-first rather than dual-write. Nothing is removed by this change: the deprecation is metadata only, consumers are not required to understand or validate it, and no existing plan changes meaning. `steps_out` remains fully functional and is scheduled for removal in a later release.
| Commit: | e0440eb | |
|---|---|---|
| Author: | YongChul Kwon | |
Clarify window bound documentation
| Commit: | 4c62b5d | |
|---|---|---|
| Author: | YongChul Kwon | |
more edits. revert pyproject.toml
| Commit: | 52eb70b | |
|---|---|---|
| Author: | YongChul Kwon | |
| Committer: | GitHub | |
Apply suggestions from code review Co-authored-by: Niels Pardon <mail@niels-pardon.de>
| Commit: | 2a15218 | |
|---|---|---|
| Author: | copilot-swe-agent[bot] | |
| Committer: | GitHub | |
Clarify null offset expression comments Co-authored-by: yongchul <1606237+yongchul@users.noreply.github.com>
| Commit: | 6f02d4f | |
|---|---|---|
| Author: | copilot-swe-agent[bot] | |
| Committer: | GitHub | |
Restore UpdateRel common field Co-authored-by: yongchul <1606237+yongchul@users.noreply.github.com>
| Commit: | e1fa40d | |
|---|---|---|
| Author: | copilot-swe-agent[bot] | |
| Committer: | GitHub | |
Clarify window offset bound guidance Co-authored-by: yongchul <1606237+yongchul@users.noreply.github.com>
| Commit: | df74360 | |
|---|---|---|
| Author: | YongChul Kwon | |
Further tighten comment and documentation
| Commit: | 9614fc2 | |
|---|---|---|
| Author: | YongChul Kwon | |
| Committer: | GitHub | |
Apply suggestions from code review Co-authored-by: Niels Pardon <mail@niels-pardon.de>
| Commit: | d10aa14 | |
|---|---|---|
| Author: | YongChul Kwon | |
tweak some wordings
| Commit: | cdf0102 | |
|---|---|---|
| Author: | Niels Pardon | |
| Committer: | GitHub | |
fix: add missing RelCommon field to UpdateRel (#1168)
| Commit: | e205e90 | |
|---|---|---|
| Author: | copilot-swe-agent[bot] | |
| Committer: | GitHub | |
fix: normalize algebra proto line ending Co-authored-by: yongchul <1606237+yongchul@users.noreply.github.com>
| Commit: | 5222bbb | |
|---|---|---|
| Author: | YongChul Kwon | |
| Committer: | GitHub | |
Merge branch 'main' into window_agg_exprs
| Commit: | d0885be | |
|---|---|---|
| Author: | YongChul Kwon | |
| Committer: | GitHub | |
Apply suggestions from code review Co-authored-by: Ben Bellick <36523439+benbellick@users.noreply.github.com> Co-authored-by: Niels Pardon <mail@niels-pardon.de>
| Commit: | 4d5d57a | |
|---|---|---|
| Author: | Niels Pardon | |
| Committer: | GitHub | |
feat: add advanced_extension to ExpandRel (#1167)
| Commit: | a902858 | |
|---|---|---|
| Author: | Niels Pardon | |
| Committer: | GitHub | |
docs: clarify that an unset RelCommon.emit_kind means Direct (#1142)
| Commit: | 0cc4341 | |
|---|---|---|
| Author: | YongChul Kwon | |
Revise documentation and comments around window frame bounds
| Commit: | 450bcc2 | |
|---|---|---|
| Author: | YongChul Kwon | |
| Committer: | GitHub | |
Apply suggestions from code review Co-authored-by: Ben Bellick <36523439+benbellick@users.noreply.github.com>
| Commit: | 5153493 | |
|---|---|---|
| Author: | Mohammad Linjawi | |
| Committer: | GitHub | |
feat: add unbound type for partially bound plans (#1081)
| Commit: | 08b2736 | |
|---|---|---|
| Author: | Niels Pardon | |
| Committer: | GitHub | |
feat(protos)!: remove deprecated left_keys/right_keys from join rels (#1129) BREAKING CHANGE: The deprecated `left_keys` and `right_keys` fields have been removed from `HashJoinRel` and `MergeJoinRel`. Producers must use the `keys` field instead.
| Commit: | b7e8d02 | |
|---|---|---|
| Author: | Niels Pardon | |
| Committer: | GitHub | |
feat(protos)!: remove deprecated offset/count from FetchRel (#1130) BREAKING CHANGE: The deprecated `offset` and `count` fields have been removed from `FetchRel`. Producers must use `offset_expr` and `count_expr` instead.
| Commit: | 7b0bd66 | |
|---|---|---|
| Author: | Niels Pardon | |
| Committer: | GitHub | |
feat(protos)!: remove deprecated VirtualTable.values field (#1131)
| Commit: | 4032b43 | |
|---|---|---|
| Author: | YongChul Kwon | |
remove oneof and deprecated the explicit offset
| Commit: | a4ad9f4 | |
|---|---|---|
| Author: | Niels Pardon | |
| Committer: | GitHub | |
feat(protos)!: remove deprecated IntervalDayToSecond.microseconds field (#1116)
| Commit: | 858397d | |
|---|---|---|
| Author: | Niels Pardon | |
| Committer: | GitHub | |
feat(protos)!: require explicit interval precision and allow picoseconds (#1110) BREAKING CHANGE: An unset `IntervalDay.precision` no longer has a defined meaning. Previously an unset precision was treated as `6` (microseconds); implementations should now reject an unset precision instead of applying a default. Producers must set `precision` explicitly.
| Commit: | 3ae33e7 | |
|---|---|---|
| Author: | YongChul Kwon | |
| Committer: | GitHub | |
feat: introduce LateralJoinRel for a correlated subquery evaluation (#973) # Background SQL or relational query plan may have subqueries. Some subqueries can be easily decorrelated in terms of joins but in other cases it is non-trivial to do such rewriting. Some queries can be de-correlated (i.e., rewritten as a join) but sometimes it is non-trivial to decorrelate complex subqueries. ApplyRel is one of the way to model a generic correlated subquery execution and implemented in multiple systems (e.g., Google Spanner, Oracle, SQL Server family). The semantic of apply is that the subquery is executed per row from the input rather than operating on set of the rows at a time like join. The subquery can reference columns in the input row as outer references (thus each subquery is *correlated* to each input row). Apply operation is inherently expensive thus typically used in a special context such as index lookup join or in a pinch where a system couldn't decorrelate the operation. # Proposal ~~Extend JoinRel with `lateral` flag to represent lateral join.~~ `LATERAL` keyword was introduced in SQL 99, as part of table reference. When the table source is wrapped with LATERAL, the subquery *MAY* reference columns and tables preceding the LATERAL table reference, and allows correlated subquery. ~~By this nature of scoping of tables and references, we propose to extend `JoinRel` with lateral flag to denote lateral join, and let right input as the correlated subquery, which can reference the fields from left input.~~ ~~In this way, we reuse all the join semantics automatically without introducing a dedicated Rel.~~ ~~Same change was made to CrossRel.~~ We decided to introduce a new LateralJoinRel for clarity and to ease handling the new semantic in existing consumers. The change is backward compatible. This PR depends on #1031 . # AI disclaimer This PR was assisted by Claude Opus 4.6. <!-- Reviewable:start --> - - - This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/substrait-io/substrait/973) <!-- Reviewable:end --> --------- Co-authored-by: Victor Barua <victor.barua@datadoghq.com> Co-authored-by: Ben Bellick <36523439+benbellick@users.noreply.github.com>
| Commit: | 0cdad6d | |
|---|---|---|
| Author: | Niels Pardon | |
| Committer: | GitHub | |
feat(protos)!: remove deprecated Type.user_defined_type_reference field (#1095) BREAKING CHANGE: removes deprecated Type.user_defined_type_reference field