Proto commits in substrait-io/substrait

These commits are when the Protocol Buffers files have changed: (only the last 100 relevant commits are shown)

2026-09-15

refactor(proto): clarify plan function definition naming Rename PlanFunction to PlanFunctionDefinition and describe the feature consistently as plan-defined functions.

feat(proto): name plan-scoped functions Add an optional diagnostic name that has no semantic identity and is not required to be unique.

feat(proto): support plan-scoped functions Allow plans to define typed expression functions once and invoke them by anchor from multiple call sites. Plan-function anchors use a namespace separate from extension functions, and definitions remain inlineable through closed, acyclic references. Closes #1220

refactor(proto): compact detached expression field

refactor(proto): focus detachment on expressions

2026-09-12

refactor(proto): inline detached subtree ordinals

2026-09-11

feat(proto): support detached plan subtrees

2026-08-27

docs: document protobuf field replacement migrations (#1181) Define a standard migration sequence for replacing protobuf fields. This keeps field documentation focused on field-specific behavior instead of repeating general migration guidance.

The documentation is generated from this commit.

2026-08-19

docs: centralize window bound migration guidance

The documentation is generated from this commit.

2026-08-18

feat(protos)!: support expression in window aggregate bounds (#1105) # Motivation Like #748 in FetchRel, `offset` can be expressions in window spec. # Proposal * Extend the offset to expression, the same way we did for FetchRel. * deprecate offset from preceding and following message. * Clarify window frame bound documentation especially around window frame, ordering, and current rows. The proto and documentation changes closes #617 and #930 BREAKING CHANGE: deprecates the `offset` field of the `Preceding` and `Following` window bounds in favor of the new `offset_expr` field, which accepts an arbitrary expression instead of an integer literal. Consumers should add `offset_expr` support before producers switch to it. The PR also requires explicit window frame bounds type. Producers should set the bounds type and Consumers should reject `BOUNDS_TYPE_UNSPECIFIED` in such cases. <!-- Reviewable:start --> - - - This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/substrait-io/substrait/1105) <!-- Reviewable:end --> --------- Co-authored-by: Ben Bellick <36523439+benbellick@users.noreply.github.com> Co-authored-by: Niels Pardon <mail@niels-pardon.de> Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>

2026-08-17

fix new lines. mandate setting bounds type

Apply suggestions from code review Co-authored-by: Ben Bellick <36523439+benbellick@users.noreply.github.com> Co-authored-by: Niels Pardon <mail@niels-pardon.de>

Merge branch 'main' into window_agg_exprs

2026-08-13

Apply suggestions from code review Co-authored-by: Ben Bellick <36523439+benbellick@users.noreply.github.com>

feat(protos)!: deprecate OuterReference.steps_out in favor of rel_reference (#1132) BREAKING CHANGE: `Expression.FieldReference.OuterReference.steps_out` is deprecated in favor of `rel_reference`, which names the binding relation via its plan-wide unique `RelCommon.rel_anchor` and therefore resolves unambiguously in DAG-shaped plans with shared relations (`ReferenceRel`), where counting subquery boundaries upward does not. Consumers should add `rel_reference` support before producers switch to it. Note that both fields are members of the same `outer_reference_type` oneof, so a plan cannot carry both forms at once: the migration is consumer-first rather than dual-write. Nothing is removed by this change: the deprecation is metadata only, consumers are not required to understand or validate it, and no existing plan changes meaning. `steps_out` remains fully functional and is scheduled for removal in a later release.

Clarify window bound documentation

more edits. revert pyproject.toml

Apply suggestions from code review Co-authored-by: Niels Pardon <mail@niels-pardon.de>

2026-08-12

Clarify null offset expression comments Co-authored-by: yongchul <1606237+yongchul@users.noreply.github.com>

Restore UpdateRel common field Co-authored-by: yongchul <1606237+yongchul@users.noreply.github.com>

Clarify window offset bound guidance Co-authored-by: yongchul <1606237+yongchul@users.noreply.github.com>

Further tighten comment and documentation

Apply suggestions from code review Co-authored-by: Niels Pardon <mail@niels-pardon.de>

tweak some wordings

fix: add missing RelCommon field to UpdateRel (#1168)

2026-08-11

fix: normalize algebra proto line ending Co-authored-by: yongchul <1606237+yongchul@users.noreply.github.com>

Merge branch 'main' into window_agg_exprs

Apply suggestions from code review Co-authored-by: Ben Bellick <36523439+benbellick@users.noreply.github.com> Co-authored-by: Niels Pardon <mail@niels-pardon.de>

2026-08-07

feat: add advanced_extension to ExpandRel (#1167)

2026-08-05

docs: clarify that an unset RelCommon.emit_kind means Direct (#1142)

2026-07-29

Revise documentation and comments around window frame bounds

Apply suggestions from code review Co-authored-by: Ben Bellick <36523439+benbellick@users.noreply.github.com>

2026-07-21

feat: add unbound type for partially bound plans (#1081)

2026-07-16

feat(protos)!: remove deprecated left_keys/right_keys from join rels (#1129) BREAKING CHANGE: The deprecated `left_keys` and `right_keys` fields have been removed from `HashJoinRel` and `MergeJoinRel`. Producers must use the `keys` field instead.

feat(protos)!: remove deprecated offset/count from FetchRel (#1130) BREAKING CHANGE: The deprecated `offset` and `count` fields have been removed from `FetchRel`. Producers must use `offset_expr` and `count_expr` instead.

feat(protos)!: remove deprecated VirtualTable.values field (#1131)

2026-07-15

remove oneof and deprecated the explicit offset

feat(protos)!: remove deprecated IntervalDayToSecond.microseconds field (#1116)

2026-07-09

feat(protos)!: require explicit interval precision and allow picoseconds (#1110) BREAKING CHANGE: An unset `IntervalDay.precision` no longer has a defined meaning. Previously an unset precision was treated as `6` (microseconds); implementations should now reject an unset precision instead of applying a default. Producers must set `precision` explicitly.

2026-07-02

feat: introduce LateralJoinRel for a correlated subquery evaluation (#973) # Background SQL or relational query plan may have subqueries. Some subqueries can be easily decorrelated in terms of joins but in other cases it is non-trivial to do such rewriting. Some queries can be de-correlated (i.e., rewritten as a join) but sometimes it is non-trivial to decorrelate complex subqueries. ApplyRel is one of the way to model a generic correlated subquery execution and implemented in multiple systems (e.g., Google Spanner, Oracle, SQL Server family). The semantic of apply is that the subquery is executed per row from the input rather than operating on set of the rows at a time like join. The subquery can reference columns in the input row as outer references (thus each subquery is *correlated* to each input row). Apply operation is inherently expensive thus typically used in a special context such as index lookup join or in a pinch where a system couldn't decorrelate the operation. # Proposal ~~Extend JoinRel with `lateral` flag to represent lateral join.~~ `LATERAL` keyword was introduced in SQL 99, as part of table reference. When the table source is wrapped with LATERAL, the subquery *MAY* reference columns and tables preceding the LATERAL table reference, and allows correlated subquery. ~~By this nature of scoping of tables and references, we propose to extend `JoinRel` with lateral flag to denote lateral join, and let right input as the correlated subquery, which can reference the fields from left input.~~ ~~In this way, we reuse all the join semantics automatically without introducing a dedicated Rel.~~ ~~Same change was made to CrossRel.~~ We decided to introduce a new LateralJoinRel for clarity and to ease handling the new semantic in existing consumers. The change is backward compatible. This PR depends on #1031 . # AI disclaimer This PR was assisted by Claude Opus 4.6. <!-- Reviewable:start --> - - - This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/substrait-io/substrait/973) <!-- Reviewable:end --> --------- Co-authored-by: Victor Barua <victor.barua@datadoghq.com> Co-authored-by: Ben Bellick <36523439+benbellick@users.noreply.github.com>

2026-06-25

feat(protos)!: remove deprecated Type.user_defined_type_reference field (#1095) BREAKING CHANGE: removes deprecated Type.user_defined_type_reference field

2026-06-15

feat: support expression in preceding and following window bound

2026-06-10

docs: require matching root relation names (#1101)

2026-06-02

feat(protos): remove deprecated Expression.Enum message (#1086) BREAKING CHANGE: removes deprecated Expression.Enum message

feat(protos): remove deprecated args fields (#1085) BREAKING CHANGE: removed args field from ScalarFunction BREAKING CHANGE: removed args field from AggregateFunction BREAKING CHANGE: removed args field from WindowFunction

2026-05-14

feat(docs): clarify distinction between enumeration arguments and options (#1005)

2026-05-13

docs: fix duplicate words in spec (#1079)

2026-05-12

feat: clarify post_join_filter. add residual_expressions to hash join and merge join (#1044) # Background JoinRel, HashJoinRel, and MergeJoinRel have a field named`post_join_filter`, causing confusion. The confusing point is whether `post_join_filter` is part of the join predicate (i.e., two rows are **matched** when it evaluates to `true`) or not (i.e., as the name says, this is **post** join filter, thus `filter` evaluates after the join operation). Substrait [FAQ](https://substrait.io/faq/#what-is-the-purpose-of-the-post-join-filter-field-on-join-relations) calls out that the `post_join_filter` is part of join predicate but @yongchul was deeply discontent about the explanation and the naming. After lengthy discussion in the Slack channel, #807 was created. After more back-and-forth, the original intent of `post_join_filter` was indeed [**post** join filter, not part of join predicate](https://github.com/substrait-io/substrait/pull/807#issuecomment-4230380222). Also, it appears that Calcite has precisely the same name and same semantic, post join filter. Following code point shows placing the filter on top of the join. https://github.com/apache/calcite/blob/0a39568b167592ded8db1128b5838982ffe264f3/core/src/main/java/org/apache/calcite/rel/rules/LoptOptimizeJoinRule.java#L553-L559 # What this changes do? TL;DR; Make `post_join_filter` as `post_join_filter` again. 1. Documents `post_join_filter` is not join condition but semantically filter on top of join. 2. Introduce `residual_expression` to HashJoinRel and MergeJoinRel so that they can express non-equality join predicate -- I guess this was the confusion who tried to shove non-equality join predicate to `HashJoin` and `MergeJoin` somewhere and distorted the meaning without much thought. 3. Drop `equi` from hash join and merge join documentation as with `residual_expression` they are capable of supporting arbitrary join predicate. 4. Fix the FAQ. # Breaking change? This is debatable. According to previous FAQ, `post_join_filter` was `residual_expression` introduced in this PR. If we take the FAQ as part of the "spec", then this is a breaking change. If we don't consider FAQ as a spec, then this is not a breaking change but clarification and adding a new feature -- HashJoinRel, MergeJoinRel now supports arbitrary join predicate. # AI disclaimer The PR is assisted by Claud Opus 4.6. <!-- Reviewable:start --> - - - This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/substrait-io/substrait/1044) <!-- Reviewable:end -->

2026-05-06

docs: clarify output schema handling for extension relations (#1018)

2026-05-01

feat!: add id-based outer reference resolution for DAG plans (#1031) BREAKING CHANGE: NOT A REAL BREAKING CHANGE. OuterReference.steps_out field is now under oneof -- implicit optional to explicit optional. This is not a real breaking change as Substrait always required non-zero value. Add optional RelCommon.id field and OuterReference.id_reference to support unambiguous outer reference resolution in plans with common subexpressions (ReferenceRel). The existing steps_out offset-based mechanism remains for backward compatibility with tree-shaped plans. The `RelCommon.rel_anchor` can be used for other purpose but for now only required when it is used to resolve outer reference. See #1024 for detailed motivating example and discussion. # AI disclaimer This PR was assisted by Claude Opus 4.6. # NOT A REAL BREAKING CHANGE `OuterReference.steps_out` field is moved into `oneof`. The field requires non-zero values thus there is no wire format change (the value has been *always* present on the wire for `steps_out` when used). <!-- Reviewable:start --> - - - This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/substrait-io/substrait/1031) <!-- Reviewable:end --> --------- Co-authored-by: Ben Bellick <36523439+benbellick@users.noreply.github.com>

2026-04-07

feat!: remove deprecated time, timestamp and timestamp_tz types (#994) BREAKING CHANGE: removes the deprecated `time`, `timestamp` and `timestamp_tz` types from: - proto files - dialect schema - extension yamls - ANTLR grammar - test cases - coverage python code - documentation closes #980 Signed-off-by: Niels Pardon <par@zurich.ibm.com>

feat: add TopNRel physical operator with WITH TIES support (#1009)

2026-04-06

feat!: removed AggregateRel.Grouping.grouping_expressions (#1002) Use AggregateRel.grouping_expressions along with AggregateRel.Grouping.expression_references instead BREAKING CHANGE: removed Aggregate.Grouping.grouping_expressions field

2026-04-02

feat: add literal based encoding for UDT

2026-03-27

feat: introduce ExecutionContextVariable and ExecutionBehaviour #945) ExecutionContextVariable is a new expression introduced to handle common SQL function likes: * current_date: retrieving the current date * current_timestamp: retrieving the timestamp with and without applying a timezone * current_timezone: retrieving the local timezone These functions are special because some systems evaluate them per row, and some evaluate once them per transaction/execution and re-use the value. The evaluation mode for ExecutionContextVariables is expressed via the plan-level ExecutionBehaviour message, which captures the different evaluation modes available.

2026-03-18

docs: adopt canonical URN format urn:substrait:extension:<namespace>:<id> - Update documentation to define canonical URN format - Add backwards compatibility note for extension: prefix - Update all extension YAML files to use canonical form - Update schema to enforce canonical format

Merge remote-tracking branch 'origin/main' into ben.bellick/clarify-urn-structure

2026-03-06

feat: only use urns for referencing simple extensions (#971) BREAKING CHANGE: drops uri fields from protobufs

2026-03-05

feat(proto): deprecate time type and literal (#985) PrecisionTime should be used instead of Time

2026-02-24

docs: fix typo mentioning interval_month instead of interval_year (#962)

2026-02-23

docs: consistently mark 0 as valid, but not preferred, anchor/reference value (#900)

2026-02-20

feat!: remove capabilities.proto (#952) The Substrait capabilities protobuf data structure has been replaced by the Substrait dialect in #816. BREAKING CHANGE: removes capabilities.proto and corresponding documentation Signed-off-by: Niels Pardon <par@zurich.ibm.com>

2026-02-03

feat!: remove unused function.proto, parameterized_types.proto, type_expressions.proto (#940) The clarification question asked in https://github.com/substrait-io/substrait/issues/849 raised awareness to me about the following three proto files we include the Substrait specification which do not seem to be used: - https://github.com/substrait-io/substrait/blob/c37d459e93875c33c7b728c3deb7c9bfe5fc7694/proto/substrait/function.proto - https://github.com/substrait-io/substrait/blob/c37d459e93875c33c7b728c3deb7c9bfe5fc7694/proto/substrait/type_expressions.proto - https://github.com/substrait-io/substrait/blob/c37d459e93875c33c7b728c3deb7c9bfe5fc7694/proto/substrait/parameterized_types.proto BREAKING CHANGE: I would propose to remove these proto definitions if they are not used / no longer part of the spec. --------- Signed-off-by: Niels Pardon <par@zurich.ibm.com>

2026-01-14

fix @vbarua comment

2026-01-08

feat: allow multiple order-by in number-less window fns

2025-12-20

feat: introduce lambda expressions (#889)

2025-11-21

tweak: use more restricive URN regex w/o capital letters `^extension:[a-z0-9_.-]+:[a-z0-9_.-]+$`

2025-11-20

Merge branch 'main' into ben.bellick/clarify-urn-structure

2025-11-18

docs: clarify URN required format with regex The required regex is: ^extension:[^:]+:[^:]+$

2025-11-15

feat: support type alias in user defined literal (#868) BREAKING CHANGE: `type_reference` with value **0** is encoded differently on wire. However, the usage of `type_reference` in generated consumer code does not break in practice. **New consumer should consider `type_reference` is set to zero when no oneof is set.** See [Wire Encoding Change](#Wire-Encoding-Change) for detail. # Motivation #857 introduced type alias concept but not plumbed all the way through. User defined literal inlines substrait.Type thus can't use type alias where it probably needs the most. Because of how the current message is defined, it is controversial how we add the type alias support, thus here is the PR for that to discuss and finalize. # Proposal I see at least three options. 1. Make `type_reference` and `type_alias_reference` as oneof. 2. Create a new user defined message that only supporting type alias. This is the cleanest at the cost leaving two ways of specify user defined literal. 3. Introduce `type_alias_anchor` to existing UD literal message. wave hands in the comment to say that "hey! this is actually oneof of `type_anchor` and when set `type_parameters` are ignored!" My preference is 1 > 2 >> 3. * The first option is more future proof especially when we extend the type alias to encode partial alias -- then the type alias *can* use subset of type parameters. * It is also easier to incorporate by implementers by slightly extending existing UD literal code. * We do not have extra message. # Wire Encoding Change Option 1 with oneof is technically a breaking change when `type_reference` value is **0** in terms of strict wire protocol. However, this is not a real breaking change in practice. Below we list the scenarios around existing `type_reference` with value **0**. ## Old producer -> New consumer New consumer will see neither `type_reference` nor `type_alias_reference` are set. The new consumer should consider this as **type_reference is set to zero**. Assuming the new consumer does not update the code (just recompile with new proto definition), access to `type_reference` will return the default value *0*. ## New producer -> Old consumer New producer will set with explicit existence bit when zero is set for `type_reference`. The old consumer will still see zero.

2025-11-03

docs: improve documentation surrounding urns

docs: clarify valid URNs

2025-10-27

feat: introduction of simple table functions introduces a notion of YAML-defined functions which are represented as a new `TableFunctionRel`. Closes #823

2025-10-25

feat: introduction of simple table functions introduces a notion of YAML-defined functions which are represented as a new `TableFunctionRel`. Closes #823

2025-10-23

docs: clarify meaning of LocalFiles (#871) Closes #869 This change clarifies the documentation surrounding `LocalFiles`. It will be clearer that `LocalFiles` can also include references to blob storage etc. as specified by the URI's protocol. It also moves the duplication of the `ReadRel` proto in the documentation to the end of the section to be consistent with other `*Rel`s documetation.

2025-10-10

docs: clarify function signatures for variadic functions (#863)

2025-10-02

feat: per plan type aliases (#857)

2025-09-12

refactor: drop dialect from Plan message

fix: update dialect field number

docs: clarify function signatures for variadic functions

feat: add AdvancedExtension to SavedComputation and LoadedComputation (#858)

feat: canonicalize extension URNs (#859) Introduces the notion of URNs to uniquely identify extensions. As part of the migration towards URNs and away from the current URIs, all references to URIs have a corresponding URN reference introduced in a neighboring field. The URI-fields have all been marked as deprecated. All usages of URIs will be dropped in future work after a migration period.

2025-06-04

remove version from the dialect reference in the Plan proto

feat: create JSON schema to define Substrait dialects (YAML) This extends the dialect files previously introduced in Substrait Go.

2025-05-28

feat: specify build input of hash join operator (#810) # Background Substrait declares build is right and probe is left today. Unfortunately there are other systems (SQL Server family, Oracle, Snowflake, Google Cloud Spanner, etc) chose opposite side and it is a major headache for those systems to deal with the mirror images. # Proposal Adding a `build_left` field to signify which side is the build of the hash join. Default (false) implies the current behavior. Left and right does not change. Spark SQL/Spanner employs this approach and be flexible. The change is backward/binary compatible.

2025-04-17

build: update go_package of .proto files to substrait-protobuf (#804) BREAKING CHANGE: go_package now points to substrait-protobuf

2025-04-16

feat: update proto go_package for substrait-protobufs

2025-03-19

feat: consolidate JoinType into a top-level enum

2025-03-11

docs: add comment to NamedSchema about struct nullability (#789) Closes https://github.com/substrait-io/substrait/issues/787.

feat: implement PrecisionTime (#788)

2025-02-14

feat: add dynamic parameter expression (#780) feat: This PR adds a dynamic parameter expression to Substrait. A dynamic parameter expression represents a placeholder within an expression whose value is determined at runtime. This is particularly useful for parameterized queries where certain values are not known until execution. Additionally, using dynamic parameters can enable other use cases, such as sharing execution plans without embedding sensitive information.

2025-02-13

feat: support picoseconds in precisionTimestamp and precistionTimestampTZ (#777)

2025-01-29

feat: add advanced extension field to DdlRel, WriteRel, and UpdateRel (#766)

2025-01-08

feat: introduce Iceberg table type using metadata file (#758) Adds Iceberg table type and first sub-variety, reading manifest files directly.

2024-12-10

feat: add expression support for count and offset in the fetch operator (#748) * Update FetchRel in Substrait to support expressions for both offset and count. This is done in a way that should be effectively compatible with old systems. Old plans should be able to continue to be consumed treating the old values as int64. BREAKING CHANGE: The encoding of FetchRel has changed in a strictly backwards incompatible way. The change involves transitioning offset and count from a standalone int64 field to a oneof structure, where the original int64 field is marked as deprecated, and a new field of Expression type is introduced. Using a oneof may cause ambiguity between unset and set-to-zero states in older messages. However, the fields are defined such that their logical meaning remains indistinguishable, ensuring consistency across encodings.

2024-11-12

feat: update operator to update a table (#734)

2024-11-05

feat: add CreateMode for CTAS in WriteRel (#715)

2024-10-29

fix: virtualTable expression should represent a row of expression (#727) BREAKING CHANGE: changes the message type for Expressions field in VirtualTable

2024-10-08

feat: define sideband optimization hints (#705)

2024-10-07

feat: enhance VirtualTable to have expression as value (#711) * This enables expressing Virtual table having values as expression * Example query: `select * from (values (1+2, 'Hello'||'World'))` * Mark existing literal field "values" as deprecated