Proto commits in substrait-io/substrait

These commits are when the Protocol Buffers files have changed: (only the last 100 relevant commits are shown)

2026-07-09

feat(protos)!: require explicit interval precision and allow picoseconds (#1110) BREAKING CHANGE: An unset `IntervalDay.precision` no longer has a defined meaning. Previously an unset precision was treated as `6` (microseconds); implementations should now reject an unset precision instead of applying a default. Producers must set `precision` explicitly.

The documentation is generated from this commit.

2026-07-02

feat: introduce LateralJoinRel for a correlated subquery evaluation (#973) # Background SQL or relational query plan may have subqueries. Some subqueries can be easily decorrelated in terms of joins but in other cases it is non-trivial to do such rewriting. Some queries can be de-correlated (i.e., rewritten as a join) but sometimes it is non-trivial to decorrelate complex subqueries. ApplyRel is one of the way to model a generic correlated subquery execution and implemented in multiple systems (e.g., Google Spanner, Oracle, SQL Server family). The semantic of apply is that the subquery is executed per row from the input rather than operating on set of the rows at a time like join. The subquery can reference columns in the input row as outer references (thus each subquery is *correlated* to each input row). Apply operation is inherently expensive thus typically used in a special context such as index lookup join or in a pinch where a system couldn't decorrelate the operation. # Proposal ~~Extend JoinRel with `lateral` flag to represent lateral join.~~ `LATERAL` keyword was introduced in SQL 99, as part of table reference. When the table source is wrapped with LATERAL, the subquery *MAY* reference columns and tables preceding the LATERAL table reference, and allows correlated subquery. ~~By this nature of scoping of tables and references, we propose to extend `JoinRel` with lateral flag to denote lateral join, and let right input as the correlated subquery, which can reference the fields from left input.~~ ~~In this way, we reuse all the join semantics automatically without introducing a dedicated Rel.~~ ~~Same change was made to CrossRel.~~ We decided to introduce a new LateralJoinRel for clarity and to ease handling the new semantic in existing consumers. The change is backward compatible. This PR depends on #1031 . # AI disclaimer This PR was assisted by Claude Opus 4.6. <!-- Reviewable:start --> - - - This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/substrait-io/substrait/973) <!-- Reviewable:end --> --------- Co-authored-by: Victor Barua <victor.barua@datadoghq.com> Co-authored-by: Ben Bellick <36523439+benbellick@users.noreply.github.com>

2026-06-25

feat(protos)!: remove deprecated Type.user_defined_type_reference field (#1095) BREAKING CHANGE: removes deprecated Type.user_defined_type_reference field

2026-06-15

feat: support expression in preceding and following window bound

2026-06-10

docs: require matching root relation names (#1101)

2026-06-02

feat(protos): remove deprecated Expression.Enum message (#1086) BREAKING CHANGE: removes deprecated Expression.Enum message

feat(protos): remove deprecated args fields (#1085) BREAKING CHANGE: removed args field from ScalarFunction BREAKING CHANGE: removed args field from AggregateFunction BREAKING CHANGE: removed args field from WindowFunction

2026-05-14

feat(docs): clarify distinction between enumeration arguments and options (#1005)

2026-05-13

docs: fix duplicate words in spec (#1079)

2026-05-12

feat: clarify post_join_filter. add residual_expressions to hash join and merge join (#1044) # Background JoinRel, HashJoinRel, and MergeJoinRel have a field named`post_join_filter`, causing confusion. The confusing point is whether `post_join_filter` is part of the join predicate (i.e., two rows are **matched** when it evaluates to `true`) or not (i.e., as the name says, this is **post** join filter, thus `filter` evaluates after the join operation). Substrait [FAQ](https://substrait.io/faq/#what-is-the-purpose-of-the-post-join-filter-field-on-join-relations) calls out that the `post_join_filter` is part of join predicate but @yongchul was deeply discontent about the explanation and the naming. After lengthy discussion in the Slack channel, #807 was created. After more back-and-forth, the original intent of `post_join_filter` was indeed [**post** join filter, not part of join predicate](https://github.com/substrait-io/substrait/pull/807#issuecomment-4230380222). Also, it appears that Calcite has precisely the same name and same semantic, post join filter. Following code point shows placing the filter on top of the join. https://github.com/apache/calcite/blob/0a39568b167592ded8db1128b5838982ffe264f3/core/src/main/java/org/apache/calcite/rel/rules/LoptOptimizeJoinRule.java#L553-L559 # What this changes do? TL;DR; Make `post_join_filter` as `post_join_filter` again. 1. Documents `post_join_filter` is not join condition but semantically filter on top of join. 2. Introduce `residual_expression` to HashJoinRel and MergeJoinRel so that they can express non-equality join predicate -- I guess this was the confusion who tried to shove non-equality join predicate to `HashJoin` and `MergeJoin` somewhere and distorted the meaning without much thought. 3. Drop `equi` from hash join and merge join documentation as with `residual_expression` they are capable of supporting arbitrary join predicate. 4. Fix the FAQ. # Breaking change? This is debatable. According to previous FAQ, `post_join_filter` was `residual_expression` introduced in this PR. If we take the FAQ as part of the "spec", then this is a breaking change. If we don't consider FAQ as a spec, then this is not a breaking change but clarification and adding a new feature -- HashJoinRel, MergeJoinRel now supports arbitrary join predicate. # AI disclaimer The PR is assisted by Claud Opus 4.6. <!-- Reviewable:start --> - - - This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/substrait-io/substrait/1044) <!-- Reviewable:end -->

2026-05-06

docs: clarify output schema handling for extension relations (#1018)

2026-05-01

feat!: add id-based outer reference resolution for DAG plans (#1031) BREAKING CHANGE: NOT A REAL BREAKING CHANGE. OuterReference.steps_out field is now under oneof -- implicit optional to explicit optional. This is not a real breaking change as Substrait always required non-zero value. Add optional RelCommon.id field and OuterReference.id_reference to support unambiguous outer reference resolution in plans with common subexpressions (ReferenceRel). The existing steps_out offset-based mechanism remains for backward compatibility with tree-shaped plans. The `RelCommon.rel_anchor` can be used for other purpose but for now only required when it is used to resolve outer reference. See #1024 for detailed motivating example and discussion. # AI disclaimer This PR was assisted by Claude Opus 4.6. # NOT A REAL BREAKING CHANGE `OuterReference.steps_out` field is moved into `oneof`. The field requires non-zero values thus there is no wire format change (the value has been *always* present on the wire for `steps_out` when used). <!-- Reviewable:start --> - - - This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/substrait-io/substrait/1031) <!-- Reviewable:end --> --------- Co-authored-by: Ben Bellick <36523439+benbellick@users.noreply.github.com>

2026-04-07

feat!: remove deprecated time, timestamp and timestamp_tz types (#994) BREAKING CHANGE: removes the deprecated `time`, `timestamp` and `timestamp_tz` types from: - proto files - dialect schema - extension yamls - ANTLR grammar - test cases - coverage python code - documentation closes #980 Signed-off-by: Niels Pardon <par@zurich.ibm.com>

feat: add TopNRel physical operator with WITH TIES support (#1009)

2026-04-06

feat!: removed AggregateRel.Grouping.grouping_expressions (#1002) Use AggregateRel.grouping_expressions along with AggregateRel.Grouping.expression_references instead BREAKING CHANGE: removed Aggregate.Grouping.grouping_expressions field

2026-04-02

feat: add literal based encoding for UDT

2026-03-27

feat: introduce ExecutionContextVariable and ExecutionBehaviour #945) ExecutionContextVariable is a new expression introduced to handle common SQL function likes: * current_date: retrieving the current date * current_timestamp: retrieving the timestamp with and without applying a timezone * current_timezone: retrieving the local timezone These functions are special because some systems evaluate them per row, and some evaluate once them per transaction/execution and re-use the value. The evaluation mode for ExecutionContextVariables is expressed via the plan-level ExecutionBehaviour message, which captures the different evaluation modes available.

2026-03-18

docs: adopt canonical URN format urn:substrait:extension:<namespace>:<id> - Update documentation to define canonical URN format - Add backwards compatibility note for extension: prefix - Update all extension YAML files to use canonical form - Update schema to enforce canonical format

Merge remote-tracking branch 'origin/main' into ben.bellick/clarify-urn-structure

2026-03-06

feat: only use urns for referencing simple extensions (#971) BREAKING CHANGE: drops uri fields from protobufs

2026-03-05

feat(proto): deprecate time type and literal (#985) PrecisionTime should be used instead of Time

2026-02-24

docs: fix typo mentioning interval_month instead of interval_year (#962)

2026-02-23

docs: consistently mark 0 as valid, but not preferred, anchor/reference value (#900)

2026-02-20

feat!: remove capabilities.proto (#952) The Substrait capabilities protobuf data structure has been replaced by the Substrait dialect in #816. BREAKING CHANGE: removes capabilities.proto and corresponding documentation Signed-off-by: Niels Pardon <par@zurich.ibm.com>

2026-02-03

feat!: remove unused function.proto, parameterized_types.proto, type_expressions.proto (#940) The clarification question asked in https://github.com/substrait-io/substrait/issues/849 raised awareness to me about the following three proto files we include the Substrait specification which do not seem to be used: - https://github.com/substrait-io/substrait/blob/c37d459e93875c33c7b728c3deb7c9bfe5fc7694/proto/substrait/function.proto - https://github.com/substrait-io/substrait/blob/c37d459e93875c33c7b728c3deb7c9bfe5fc7694/proto/substrait/type_expressions.proto - https://github.com/substrait-io/substrait/blob/c37d459e93875c33c7b728c3deb7c9bfe5fc7694/proto/substrait/parameterized_types.proto BREAKING CHANGE: I would propose to remove these proto definitions if they are not used / no longer part of the spec. --------- Signed-off-by: Niels Pardon <par@zurich.ibm.com>

2026-01-14

fix @vbarua comment

2026-01-08

feat: allow multiple order-by in number-less window fns

2025-12-20

feat: introduce lambda expressions (#889)

2025-11-21

tweak: use more restricive URN regex w/o capital letters `^extension:[a-z0-9_.-]+:[a-z0-9_.-]+$`

2025-11-20

Merge branch 'main' into ben.bellick/clarify-urn-structure

2025-11-18

docs: clarify URN required format with regex The required regex is: ^extension:[^:]+:[^:]+$

2025-11-15

feat: support type alias in user defined literal (#868) BREAKING CHANGE: `type_reference` with value **0** is encoded differently on wire. However, the usage of `type_reference` in generated consumer code does not break in practice. **New consumer should consider `type_reference` is set to zero when no oneof is set.** See [Wire Encoding Change](#Wire-Encoding-Change) for detail. # Motivation #857 introduced type alias concept but not plumbed all the way through. User defined literal inlines substrait.Type thus can't use type alias where it probably needs the most. Because of how the current message is defined, it is controversial how we add the type alias support, thus here is the PR for that to discuss and finalize. # Proposal I see at least three options. 1. Make `type_reference` and `type_alias_reference` as oneof. 2. Create a new user defined message that only supporting type alias. This is the cleanest at the cost leaving two ways of specify user defined literal. 3. Introduce `type_alias_anchor` to existing UD literal message. wave hands in the comment to say that "hey! this is actually oneof of `type_anchor` and when set `type_parameters` are ignored!" My preference is 1 > 2 >> 3. * The first option is more future proof especially when we extend the type alias to encode partial alias -- then the type alias *can* use subset of type parameters. * It is also easier to incorporate by implementers by slightly extending existing UD literal code. * We do not have extra message. # Wire Encoding Change Option 1 with oneof is technically a breaking change when `type_reference` value is **0** in terms of strict wire protocol. However, this is not a real breaking change in practice. Below we list the scenarios around existing `type_reference` with value **0**. ## Old producer -> New consumer New consumer will see neither `type_reference` nor `type_alias_reference` are set. The new consumer should consider this as **type_reference is set to zero**. Assuming the new consumer does not update the code (just recompile with new proto definition), access to `type_reference` will return the default value *0*. ## New producer -> Old consumer New producer will set with explicit existence bit when zero is set for `type_reference`. The old consumer will still see zero.

2025-11-03

docs: improve documentation surrounding urns

docs: clarify valid URNs

2025-10-27

feat: introduction of simple table functions introduces a notion of YAML-defined functions which are represented as a new `TableFunctionRel`. Closes #823

2025-10-25

feat: introduction of simple table functions introduces a notion of YAML-defined functions which are represented as a new `TableFunctionRel`. Closes #823

2025-10-23

docs: clarify meaning of LocalFiles (#871) Closes #869 This change clarifies the documentation surrounding `LocalFiles`. It will be clearer that `LocalFiles` can also include references to blob storage etc. as specified by the URI's protocol. It also moves the duplication of the `ReadRel` proto in the documentation to the end of the section to be consistent with other `*Rel`s documetation.

2025-10-10

docs: clarify function signatures for variadic functions (#863)

2025-10-02

feat: per plan type aliases (#857)

2025-09-12

refactor: drop dialect from Plan message

fix: update dialect field number

docs: clarify function signatures for variadic functions

feat: add AdvancedExtension to SavedComputation and LoadedComputation (#858)

feat: canonicalize extension URNs (#859) Introduces the notion of URNs to uniquely identify extensions. As part of the migration towards URNs and away from the current URIs, all references to URIs have a corresponding URN reference introduced in a neighboring field. The URI-fields have all been marked as deprecated. All usages of URIs will be dropped in future work after a migration period.

2025-06-04

remove version from the dialect reference in the Plan proto

feat: create JSON schema to define Substrait dialects (YAML) This extends the dialect files previously introduced in Substrait Go.

2025-05-28

feat: specify build input of hash join operator (#810) # Background Substrait declares build is right and probe is left today. Unfortunately there are other systems (SQL Server family, Oracle, Snowflake, Google Cloud Spanner, etc) chose opposite side and it is a major headache for those systems to deal with the mirror images. # Proposal Adding a `build_left` field to signify which side is the build of the hash join. Default (false) implies the current behavior. Left and right does not change. Spark SQL/Spanner employs this approach and be flexible. The change is backward/binary compatible.

2025-04-17

build: update go_package of .proto files to substrait-protobuf (#804) BREAKING CHANGE: go_package now points to substrait-protobuf

2025-04-16

feat: update proto go_package for substrait-protobufs

2025-03-19

feat: consolidate JoinType into a top-level enum

2025-03-11

docs: add comment to NamedSchema about struct nullability (#789) Closes https://github.com/substrait-io/substrait/issues/787.

feat: implement PrecisionTime (#788)

2025-02-14

feat: add dynamic parameter expression (#780) feat: This PR adds a dynamic parameter expression to Substrait. A dynamic parameter expression represents a placeholder within an expression whose value is determined at runtime. This is particularly useful for parameterized queries where certain values are not known until execution. Additionally, using dynamic parameters can enable other use cases, such as sharing execution plans without embedding sensitive information.

2025-02-13

feat: support picoseconds in precisionTimestamp and precistionTimestampTZ (#777)

2025-01-29

feat: add advanced extension field to DdlRel, WriteRel, and UpdateRel (#766)

2025-01-08

feat: introduce Iceberg table type using metadata file (#758) Adds Iceberg table type and first sub-variety, reading manifest files directly.

2024-12-10

feat: add expression support for count and offset in the fetch operator (#748) * Update FetchRel in Substrait to support expressions for both offset and count. This is done in a way that should be effectively compatible with old systems. Old plans should be able to continue to be consumed treating the old values as int64. BREAKING CHANGE: The encoding of FetchRel has changed in a strictly backwards incompatible way. The change involves transitioning offset and count from a standalone int64 field to a oneof structure, where the original int64 field is marked as deprecated, and a new field of Expression type is introduced. Using a oneof may cause ambiguity between unset and set-to-zero states in older messages. However, the fields are defined such that their logical meaning remains indistinguishable, ensuring consistency across encodings.

2024-11-12

feat: update operator to update a table (#734)

2024-11-05

feat: add CreateMode for CTAS in WriteRel (#715)

2024-10-29

fix: virtualTable expression should represent a row of expression (#727) BREAKING CHANGE: changes the message type for Expressions field in VirtualTable

2024-10-08

feat: define sideband optimization hints (#705)

2024-10-07

feat: enhance VirtualTable to have expression as value (#711) * This enables expressing Virtual table having values as expression * Example query: `select * from (values (1+2, 'Hello'||'World'))` * Mark existing literal field "values" as deprecated

2024-09-27

feat: clarify behaviour of SetRel operations (#708) Clarified language of existing SetRel operations to distinguish ALL and DISTINCT behaviour. Added language around ALL versions of operations to better specify how many versions of a record should appear in the relation output. feat: added SET_OP_MINUS_PRIMARY_ALL feat: added SET_OP_INTERSECTION_MULTISET_ALL These operations corresponds to SQL: * EXCEPT ALL * INTERSECT ALL

feat: change grouping expressions in AggregateRel to references (#706) BREAKING CHANGE: This PR changes the definition of grouping sets in `AggregateRel` to consist of references into a list of grouping expressions instead of consisting of expressions directly. With the previous definition, consumers had to deduplicate the expressions in the grouping sets in order to execute the query or even derive the output schema (which is problematic, as explained below). With this change, the responsibility of deduplicating expressions is now on the producer. Concretely, consumers are now expected to be simpler: The list of grouping expressions immediately provides the information needed to derive the output schema and the list of grouping sets explicitly and unambiguously provides the equality of grouping expressions. Producers now have to specify the grouping sets explicitly. If their internal representation of grouping sets consists of full grouping expressions (rather than references), then they must deduplicate these expressions according to their internal notion of expression equality in order to produce grouping sets consisting of references to these deduplicated expressions. If the previous format is desired, it can be obtained from the new format by (1) deduplicating the grouping expressions (according to the previously applicable definition of expression equality), (2) re-establishing the duplicates using the emit clause, and (3) "dereferencing" the references in the grouping sets, i.e., by replacing each reference in the grouping sets with the expression it refers to. The previous version was problematic because it required the *consumers* to deduplicate the expressions from the grouping sets. This, in turn, requires to parse and understand 100% of these expression even in cases where that understanding is otherwise optional, which is in opposition to the general philosophy of allowing for simple-minded consumers. The new version avoids that problem and, thus, allows consumers to be simpler. The issues are discussed in even more detail in #700. --------- Signed-off-by: Ingo Müller <ingomueller@google.com> Co-authored-by: Weston Pace <weston.pace@gmail.com>

2024-09-11

feat: add optional metadata containing field names to RelCommon (#696) This PR introduces a new metadata field in the `RelCommon` hint definition which contains the output field names for any relation. This enables full resolution roundtrip for `Expand` in the `spark` module. Signed-off-by: Andrew Coleman <andrew_coleman@uk.ibm.com>

2024-08-21

feat: define mark join (#682) Defines the Mark Join used by a number of engines in query rewrites. --------- Co-authored-by: Weston Pace <weston.pace@gmail.com> Co-authored-by: Jacques Nadeau <jacquesnadeau@gmail.com>

2024-08-10

feat: add CSV (text) file support (#646) First version of CSV options (inspired by various engines/libraries)

feat: add precision to IntervalDay and new IntervalCompound type (#665) * Update IntervalDay to support multiple subsecond precisions (0-9). This is done in a way that should be effectively compatible with old systems. Old plans should be able to be continue to be consumed treating the old values as microsecond precision. * Add new IntervalCompound type which is a combination of IntervalMonth and IntervalDay BREAKING CHANGE: The encoding of IntervalDay literals has changed in a strictly backwards incompatible way. However, the logical meaning across encoding is maintained using a oneof. Moving a field into a oneof makes unset/set to zero unclear with older messages but the fields are defined such that the logical meaning of the two is indistinct. If neither microseconds nor precision is set, the value can be considered a precision 6 value. If you aren't using IntervalDay type, you will not need to make any changes. BREAKING CHANGE: TypeExpression and Parameterized type protobufs (used to serialize output derivation) are updated to match the now compound nature of IntervalDay. If you use protobuf to serialize output derivation that refer to IntervalDay type, you will need to rework that logic. fixes #664 --------- Co-authored-by: Jacques Nadeau <jacques@apache.org>

2024-08-08

feat: normalize the join types (#662) There are two groups of join types in the definition with differing enums. This PR leaves JoinRel's SEMI and ANTI as the canonical names for LEFT_SEMI and LEFT_ANTI. Aliases are not allowed due to JSON (and text) serialization behavior. This PR also adds RIGHT_SEMI and RIGHT_ANTI to JoinRel's JoinType. RIGHT_SINGLE is added to all types. The PR correspondingly renames SINGLE to LEFT_SINGLE, ANTI TO LEFT_ANTI, and SEMI to LEFT_SEMI. Finally this PR adds LEFT_SINGLE to all of the other join types. BREAKING CHANGE: JoinRel's type enum now has LEFT_SINGLE instead of SINGLE. Similarly there is now LEFT_ANTI and LEFT_SEMI. Other values are available in all join type enums. This affects JSON and text formats only (binary plans -- the interoperable part of Substrait -- will still be compatible before and after this change). --------- Co-authored-by: Jacques Nadeau <jacquesnadeau@gmail.com>

2024-07-28

fix: use int64 instead of uint64 for PrecisionTimestamp(Tz) literal value (#668) This allows timestamps to refer to time before epoch, and aligns with other systems (Spark, DataFusion/Arrow, DuckDB, Postgres, Parquet at least) BREAKING CHANGE: PrecisionTimestamp(Tz) literal's value is now int64 instead of uint64

2024-07-11

fix: include precision information in PrecisionTimestamp and PrecisionTimestampTZ literals (#659) BREAKING CHANGE: changes the message type for Literal PrecisionTimestamp and PrecisionTimestampTZ The PrecisionTimestamp and PrecisionTimestampTZ literals were introduced in #594, and there was some discussion about their contents in https://github.com/substrait-io/substrait/pull/594#discussion_r1471844566. In their current form, they only contain a i64 value, which I believe is meant to be a precision-dependent number of fractional seconds since epoch. However, the literals don't contain the precision itself, so it's impossible to interpret a PrecisionTimestamp or PrecisionTimestampTZ literal without other information. This is in contrast to e.g. varchar, whose literal does specify the length, or decimal, whose literal specifies scale and precision. @westonpace curious for your thoughts since you were part of that original discussion - am I missing something or is this a bug? This PR changes the Literal types for PrecisionTimestamp and PrecisionTimestampTZ to contain a PrecisionTimestamp message instead of a i64. That message then contains the i64 value as well as the i32 type. This is a breaking change, but given in their current form these literals aren't usable (unless I'm missing something), would that be okay? Currently I used the same message for both PrecisionTimestamp and PrecisionTimestampTZ, but I can also make a copy for PTTZ if that'd be better for forwards-compatibility.

2024-07-03

feat: allow naming/aliasing relations (#649) Same goal as in #648 - to support what Spark's and DataFusion's SubqueryAlias relation does. As Substrait mostly only referes to columns by their index, there is no inherent need for table name/qualifiers within Substrait. However, some consumers, e.g. DataFusion, require column names to be either unique or qualified for joins, which is troublesome w/o the possibility to qualify relations. Also, for debugging failed plans and for roundtrip testing of X -> Substrait -> X conversions, it would be convenient to have proper, human-readable names to refer to. Closes #571

2024-06-28

feat: make optimization a repeated field (#653) BREAKING CHANGE: consumers must now check for multiple optimization messages within an AdvancedExtension

2024-06-26

feat: make optimization a repeated field

2024-05-08

docs: add note about output

2024-05-07

feat: use ranges for all quantifiers

chore: format

feat: tweaks and function definitions

2024-05-04

feat: advance_extension in MatchRecognizeRel

feat: trino and snowflake support running/final measure options

2024-04-30

feat: sketch of MATCH_RECOGNIZE support

2024-04-18

docs: clarify nullability information source for empty literals (#602)

2024-04-17

docs: clarify language around user-defined types (#604)

2024-04-16

feat: allow FetchRel to specify a return of ALL results (#622) (#627)

fix: use Literal.Struct not Type.Struct

2024-04-15

feat: allow FetchRel to specify a return of ALL results

2024-04-03

doc: clarify nullability information source for empty literals

2024-03-28

refactor: threading the compatability needle

ci: making buf format happy

refactor: avoid breaking JSON compatability

docs: clarify language around user-define types

2024-02-22

feat: include precision parameter in timestamp types (#594) This PR introduces new `PrecisionTimestamp` and `PrecisionTimestampTZ` types that accept an optional precision parameter to specify fractional second precision. Closes https://github.com/substrait-io/substrait/issues/592 --------- Co-authored-by: Weston Pace <weston.pace@gmail.com>

2024-01-23

docs: add more explanation of comparison function (in hash/merge join) (#586)

fix: add missing RelCommon field to WriteRel and DdlRel (#591) Adding the RelCommon field to WriteRel and DdlRel message types so that they can be handled with the same common functionality as other operators. Addition of a new field is not a breaking change. Context from [Slack discussion](https://substrait.slack.com/archives/C02D7CTQXHD/p1705839323529739)

2024-01-17

feat: add custom equality behavior to the hash/merge join (#585) This PR (hopefully) concludes various discussions around flags such as `null_equals_null` (Datafusion) and `null_aware` (Velox). The goal of these flags is to slightly tweak the definition of "equality" in an equijoin relation. This PR introduces a new EquiJoinKey message that can be used by physical join relations to define how keys should be compared. These custom equality functions are needed in a variety of scenarios: ## Optimizing set operations Set operations (e.g. set difference) can sometimes be satisfied by an equi-join. When this happens the user typically wants the equality comparison to be "is not distinct from" ## Flattening correlated subqueries Some kinds of correlated subqueries can be removed during optimization and replaced with an anti-join. Depending on the original query ("not in" vs "where not exists") there may be slightly different behaviors with respect to null an we may want to use "might equals" as the comparison. ## String collations Collations define the ordering and equality of a column. Different columns can have different collations. The equi-join must use the comparison function defined by the collation.

2023-12-11

feat: add missing rels to rel message (#582) The `ReferenceRel`, `WriteRel`, and `DdlRel` were defined in `algebra.proto` but not part of `message Rel` which meant they were unusable. This PR adds those back. It is inspired by #288 but more targeted in scope. One change from that original PR which I also kept was replacing the word `tuple` with `record` in the documentation for consistency. This is not to imply that `ReferenceRel`, `WriteRel`, or `DdlRel` are "complete" or "stable" in any way. I feel these relations are still quite ill defined. However, my hope is that by making them usable we can inspire further change to them. BREAKING CHANGE: The enum `WriteRel::OutputMode` had an option change from `OUTPUT_MODE_MODIFIED_TUPLES` to `OUTPUT_MODE_MODIFIED_RECORDS` BREAKING CHANGE: The message `AggregateFunction.ReferenceRel` has moved to `ReferenceRel`.

2023-11-22

docs: expand the ExpandRel documentation (#546)

2023-10-20

feat: add NestedLoopJoinRel definition (#561) This PR adds the NestedLoopJoinRel proto definition. Closes https://github.com/substrait-io/substrait/issues/456 --------- Co-authored-by: Weston Pace <weston.pace@gmail.com>

2023-09-08

feat: allow agg functions to be used in windows (#540)

2023-08-21

feat: add windowrel support in proto (#399) Co-authored-by: Weston Pace <weston.pace@gmail.com> Co-authored-by: Victor Barua <victor.barua@datadoghq.com>