Proto commits in substrait-io/substrait

These commits are when the Protocol Buffers files have changed: (only the last 100 relevant commits are shown)

2025-04-17

build: update go_package of .proto files to substrait-protobuf (#804) BREAKING CHANGE: go_package now points to substrait-protobuf

The documentation is generated from this commit.

2025-04-16

feat: update proto go_package for substrait-protobufs

The documentation is generated from this commit.

2025-03-19

feat: consolidate JoinType into a top-level enum

2025-03-11

docs: add comment to NamedSchema about struct nullability (#789) Closes https://github.com/substrait-io/substrait/issues/787.

feat: implement PrecisionTime (#788)

2025-02-14

feat: add dynamic parameter expression (#780) feat: This PR adds a dynamic parameter expression to Substrait. A dynamic parameter expression represents a placeholder within an expression whose value is determined at runtime. This is particularly useful for parameterized queries where certain values are not known until execution. Additionally, using dynamic parameters can enable other use cases, such as sharing execution plans without embedding sensitive information.

2025-02-13

feat: support picoseconds in precisionTimestamp and precistionTimestampTZ (#777)

2025-01-29

feat: add advanced extension field to DdlRel, WriteRel, and UpdateRel (#766)

2025-01-08

feat: introduce Iceberg table type using metadata file (#758) Adds Iceberg table type and first sub-variety, reading manifest files directly.

2024-12-10

feat: add expression support for count and offset in the fetch operator (#748) * Update FetchRel in Substrait to support expressions for both offset and count. This is done in a way that should be effectively compatible with old systems. Old plans should be able to continue to be consumed treating the old values as int64. BREAKING CHANGE: The encoding of FetchRel has changed in a strictly backwards incompatible way. The change involves transitioning offset and count from a standalone int64 field to a oneof structure, where the original int64 field is marked as deprecated, and a new field of Expression type is introduced. Using a oneof may cause ambiguity between unset and set-to-zero states in older messages. However, the fields are defined such that their logical meaning remains indistinguishable, ensuring consistency across encodings.

2024-11-12

feat: update operator to update a table (#734)

2024-11-05

feat: add CreateMode for CTAS in WriteRel (#715)

2024-10-29

fix: virtualTable expression should represent a row of expression (#727) BREAKING CHANGE: changes the message type for Expressions field in VirtualTable

2024-10-08

feat: define sideband optimization hints (#705)

2024-10-07

feat: enhance VirtualTable to have expression as value (#711) * This enables expressing Virtual table having values as expression * Example query: `select * from (values (1+2, 'Hello'||'World'))` * Mark existing literal field "values" as deprecated

2024-09-27

feat: clarify behaviour of SetRel operations (#708) Clarified language of existing SetRel operations to distinguish ALL and DISTINCT behaviour. Added language around ALL versions of operations to better specify how many versions of a record should appear in the relation output. feat: added SET_OP_MINUS_PRIMARY_ALL feat: added SET_OP_INTERSECTION_MULTISET_ALL These operations corresponds to SQL: * EXCEPT ALL * INTERSECT ALL

feat: change grouping expressions in AggregateRel to references (#706) BREAKING CHANGE: This PR changes the definition of grouping sets in `AggregateRel` to consist of references into a list of grouping expressions instead of consisting of expressions directly. With the previous definition, consumers had to deduplicate the expressions in the grouping sets in order to execute the query or even derive the output schema (which is problematic, as explained below). With this change, the responsibility of deduplicating expressions is now on the producer. Concretely, consumers are now expected to be simpler: The list of grouping expressions immediately provides the information needed to derive the output schema and the list of grouping sets explicitly and unambiguously provides the equality of grouping expressions. Producers now have to specify the grouping sets explicitly. If their internal representation of grouping sets consists of full grouping expressions (rather than references), then they must deduplicate these expressions according to their internal notion of expression equality in order to produce grouping sets consisting of references to these deduplicated expressions. If the previous format is desired, it can be obtained from the new format by (1) deduplicating the grouping expressions (according to the previously applicable definition of expression equality), (2) re-establishing the duplicates using the emit clause, and (3) "dereferencing" the references in the grouping sets, i.e., by replacing each reference in the grouping sets with the expression it refers to. The previous version was problematic because it required the *consumers* to deduplicate the expressions from the grouping sets. This, in turn, requires to parse and understand 100% of these expression even in cases where that understanding is otherwise optional, which is in opposition to the general philosophy of allowing for simple-minded consumers. The new version avoids that problem and, thus, allows consumers to be simpler. The issues are discussed in even more detail in #700. --------- Signed-off-by: Ingo Müller <ingomueller@google.com> Co-authored-by: Weston Pace <weston.pace@gmail.com>

2024-09-11

feat: add optional metadata containing field names to RelCommon (#696) This PR introduces a new metadata field in the `RelCommon` hint definition which contains the output field names for any relation. This enables full resolution roundtrip for `Expand` in the `spark` module. Signed-off-by: Andrew Coleman <andrew_coleman@uk.ibm.com>

2024-08-21

feat: define mark join (#682) Defines the Mark Join used by a number of engines in query rewrites. --------- Co-authored-by: Weston Pace <weston.pace@gmail.com> Co-authored-by: Jacques Nadeau <jacquesnadeau@gmail.com>

2024-08-10

feat: add CSV (text) file support (#646) First version of CSV options (inspired by various engines/libraries)

feat: add precision to IntervalDay and new IntervalCompound type (#665) * Update IntervalDay to support multiple subsecond precisions (0-9). This is done in a way that should be effectively compatible with old systems. Old plans should be able to be continue to be consumed treating the old values as microsecond precision. * Add new IntervalCompound type which is a combination of IntervalMonth and IntervalDay BREAKING CHANGE: The encoding of IntervalDay literals has changed in a strictly backwards incompatible way. However, the logical meaning across encoding is maintained using a oneof. Moving a field into a oneof makes unset/set to zero unclear with older messages but the fields are defined such that the logical meaning of the two is indistinct. If neither microseconds nor precision is set, the value can be considered a precision 6 value. If you aren't using IntervalDay type, you will not need to make any changes. BREAKING CHANGE: TypeExpression and Parameterized type protobufs (used to serialize output derivation) are updated to match the now compound nature of IntervalDay. If you use protobuf to serialize output derivation that refer to IntervalDay type, you will need to rework that logic. fixes #664 --------- Co-authored-by: Jacques Nadeau <jacques@apache.org>

2024-08-08

feat: normalize the join types (#662) There are two groups of join types in the definition with differing enums. This PR leaves JoinRel's SEMI and ANTI as the canonical names for LEFT_SEMI and LEFT_ANTI. Aliases are not allowed due to JSON (and text) serialization behavior. This PR also adds RIGHT_SEMI and RIGHT_ANTI to JoinRel's JoinType. RIGHT_SINGLE is added to all types. The PR correspondingly renames SINGLE to LEFT_SINGLE, ANTI TO LEFT_ANTI, and SEMI to LEFT_SEMI. Finally this PR adds LEFT_SINGLE to all of the other join types. BREAKING CHANGE: JoinRel's type enum now has LEFT_SINGLE instead of SINGLE. Similarly there is now LEFT_ANTI and LEFT_SEMI. Other values are available in all join type enums. This affects JSON and text formats only (binary plans -- the interoperable part of Substrait -- will still be compatible before and after this change). --------- Co-authored-by: Jacques Nadeau <jacquesnadeau@gmail.com>

2024-07-28

fix: use int64 instead of uint64 for PrecisionTimestamp(Tz) literal value (#668) This allows timestamps to refer to time before epoch, and aligns with other systems (Spark, DataFusion/Arrow, DuckDB, Postgres, Parquet at least) BREAKING CHANGE: PrecisionTimestamp(Tz) literal's value is now int64 instead of uint64

2024-07-11

fix: include precision information in PrecisionTimestamp and PrecisionTimestampTZ literals (#659) BREAKING CHANGE: changes the message type for Literal PrecisionTimestamp and PrecisionTimestampTZ The PrecisionTimestamp and PrecisionTimestampTZ literals were introduced in #594, and there was some discussion about their contents in https://github.com/substrait-io/substrait/pull/594#discussion_r1471844566. In their current form, they only contain a i64 value, which I believe is meant to be a precision-dependent number of fractional seconds since epoch. However, the literals don't contain the precision itself, so it's impossible to interpret a PrecisionTimestamp or PrecisionTimestampTZ literal without other information. This is in contrast to e.g. varchar, whose literal does specify the length, or decimal, whose literal specifies scale and precision. @westonpace curious for your thoughts since you were part of that original discussion - am I missing something or is this a bug? This PR changes the Literal types for PrecisionTimestamp and PrecisionTimestampTZ to contain a PrecisionTimestamp message instead of a i64. That message then contains the i64 value as well as the i32 type. This is a breaking change, but given in their current form these literals aren't usable (unless I'm missing something), would that be okay? Currently I used the same message for both PrecisionTimestamp and PrecisionTimestampTZ, but I can also make a copy for PTTZ if that'd be better for forwards-compatibility.

2024-07-03

feat: allow naming/aliasing relations (#649) Same goal as in #648 - to support what Spark's and DataFusion's SubqueryAlias relation does. As Substrait mostly only referes to columns by their index, there is no inherent need for table name/qualifiers within Substrait. However, some consumers, e.g. DataFusion, require column names to be either unique or qualified for joins, which is troublesome w/o the possibility to qualify relations. Also, for debugging failed plans and for roundtrip testing of X -> Substrait -> X conversions, it would be convenient to have proper, human-readable names to refer to. Closes #571

2024-06-28

feat: make optimization a repeated field (#653) BREAKING CHANGE: consumers must now check for multiple optimization messages within an AdvancedExtension

2024-06-26

feat: make optimization a repeated field

2024-05-08

docs: add note about output

2024-05-07

feat: use ranges for all quantifiers

chore: format

feat: tweaks and function definitions

2024-05-04

feat: advance_extension in MatchRecognizeRel

feat: trino and snowflake support running/final measure options

2024-04-30

feat: sketch of MATCH_RECOGNIZE support

2024-04-18

docs: clarify nullability information source for empty literals (#602)

2024-04-17

docs: clarify language around user-defined types (#604)

2024-04-16

feat: allow FetchRel to specify a return of ALL results (#622) (#627)

fix: use Literal.Struct not Type.Struct

2024-04-15

feat: allow FetchRel to specify a return of ALL results

2024-04-03

doc: clarify nullability information source for empty literals

2024-03-28

refactor: threading the compatability needle

ci: making buf format happy

refactor: avoid breaking JSON compatability

docs: clarify language around user-define types

2024-02-22

feat: include precision parameter in timestamp types (#594) This PR introduces new `PrecisionTimestamp` and `PrecisionTimestampTZ` types that accept an optional precision parameter to specify fractional second precision. Closes https://github.com/substrait-io/substrait/issues/592 --------- Co-authored-by: Weston Pace <weston.pace@gmail.com>

2024-01-23

docs: add more explanation of comparison function (in hash/merge join) (#586)

fix: add missing RelCommon field to WriteRel and DdlRel (#591) Adding the RelCommon field to WriteRel and DdlRel message types so that they can be handled with the same common functionality as other operators. Addition of a new field is not a breaking change. Context from [Slack discussion](https://substrait.slack.com/archives/C02D7CTQXHD/p1705839323529739)

2024-01-17

feat: add custom equality behavior to the hash/merge join (#585) This PR (hopefully) concludes various discussions around flags such as `null_equals_null` (Datafusion) and `null_aware` (Velox). The goal of these flags is to slightly tweak the definition of "equality" in an equijoin relation. This PR introduces a new EquiJoinKey message that can be used by physical join relations to define how keys should be compared. These custom equality functions are needed in a variety of scenarios: ## Optimizing set operations Set operations (e.g. set difference) can sometimes be satisfied by an equi-join. When this happens the user typically wants the equality comparison to be "is not distinct from" ## Flattening correlated subqueries Some kinds of correlated subqueries can be removed during optimization and replaced with an anti-join. Depending on the original query ("not in" vs "where not exists") there may be slightly different behaviors with respect to null an we may want to use "might equals" as the comparison. ## String collations Collations define the ordering and equality of a column. Different columns can have different collations. The equi-join must use the comparison function defined by the collation.

2023-12-11

feat: add missing rels to rel message (#582) The `ReferenceRel`, `WriteRel`, and `DdlRel` were defined in `algebra.proto` but not part of `message Rel` which meant they were unusable. This PR adds those back. It is inspired by #288 but more targeted in scope. One change from that original PR which I also kept was replacing the word `tuple` with `record` in the documentation for consistency. This is not to imply that `ReferenceRel`, `WriteRel`, or `DdlRel` are "complete" or "stable" in any way. I feel these relations are still quite ill defined. However, my hope is that by making them usable we can inspire further change to them. BREAKING CHANGE: The enum `WriteRel::OutputMode` had an option change from `OUTPUT_MODE_MODIFIED_TUPLES` to `OUTPUT_MODE_MODIFIED_RECORDS` BREAKING CHANGE: The message `AggregateFunction.ReferenceRel` has moved to `ReferenceRel`.

2023-11-22

docs: expand the ExpandRel documentation (#546)

2023-10-20

feat: add NestedLoopJoinRel definition (#561) This PR adds the NestedLoopJoinRel proto definition. Closes https://github.com/substrait-io/substrait/issues/456 --------- Co-authored-by: Weston Pace <weston.pace@gmail.com>

2023-09-08

feat: allow agg functions to be used in windows (#540)

2023-08-21

feat: add windowrel support in proto (#399) Co-authored-by: Weston Pace <weston.pace@gmail.com> Co-authored-by: Victor Barua <victor.barua@datadoghq.com>

2023-08-11

feat: allow agg functions to be used in windows

2023-08-10

feat: add expand rel (#368) Adds an expand relation. This relation can be used to create near-duplicate copies of each input row based on templates describing how to create the copies. This is used within spark to implement certain operations like aggregate rollup and pivot longer. --------- Co-authored-by: Weston Pace <weston.pace@gmail.com>

2023-08-09

feat!: require compound functions names in extension references (#537) BREAKING CHANGE: plans referencing functions using simple names (e.g. not vs not:bool) will no longer be valid. --------- Co-authored-by: Weston Pace <weston.pace@gmail.com>

2023-08-07

feat!: require compound functions names in extension references

2023-07-24

docs: clarify requirements for groupings and measures in AggregateRel (#525)

2023-07-21

docs: document IfThen proto (#530)

docs: document IfThen proto

feat: add ExchangeRel as a type in Rel (#518) The `ExchangeRel` is already in the spec, but it is not a type of `Rel` yet. Co-authored-by: tanjunsheng <tanjunsheng@bytedance.com>

2023-04-07

fix: typo in the comment/docstring (#492)

2023-01-18

feat: add extended expression for expression only evaluation (#405)

2023-01-16

feat(spec): add physical plans for hashJoin and mergeJoin (#336)

2022-11-25

feat: add nested type constructor expressions (#351)

2022-11-14

feat: optional args are now specified separately from required args (#342) BREAKING CHANGE: optional arguments are no longer allowed to be specified as a part of FunctionArgument messages. Instead they are now specified separately as part of the function invocation. BREAKING CHANGE: optional arguments are now specified separately from required arguments in the YAML specification. Co-authored-by: Benjamin Kietzman <bengilgit@gmail.com> Co-authored-by: Benjamin Kietzman <bengilgit@gmail.com>

feat: add best effort filter to read rel and clarify that the pre-masked schema should be used (#271)

2022-11-01

fix: rename version fields which conflict with sysmacros (#362)

2022-10-05

feat: attach Substrait version number to plans (#347)

2022-09-29

docs: clarify description of set relation (#344)

2022-09-02

feat: add dwrf file format (#304)

2022-08-11

Implementation for ReferenceRel (#284) * feat: introducing a ReferenceRel operator to express DAG/Spool. (#283) Co-authored-by: Jacques Nadeau <jacques@apache.org>

feat: add and_not boolean function (#276) Co-authored-by: Phillip Cloud <417981+cpcloud@users.noreply.github.com>

2022-08-10

feat: add support for DDL and INSERT/DELETE/UPDATE operations (#252) * feat: add support for DDL and INSERT/DELETE/UPDATE operations (#252) Co-authored-by: Jacques Nadeau <jacquesnadeau@gmail.com>

2022-07-26

fix: specify how function arguments are to be bound (#231) BREAKING CHANGE: function argument bindings were open to interpretation before, and were often produced incorrectly; therefore, this change semantically shifts some responsibilities from the consumers to the producers.

2022-07-04

feat: introduce compound (parameterizable) extension types and variations (#196) * feat: introduce compound (parameterizable) extension types and variations

2022-06-25

docs: inlining comments into protobuf and linking protobuf in website… (#232) * docs: inlining comments into protobuf and linking protobuf in website (#215) Co-authored-by: Carlo Curino <curino@apache.org>

2022-06-09

feat: support nullable and non-default variation user-defined types (#217) * feat: support nullable and non-default variation user-defined type expressions

2022-06-05

feat: add Arrow and Orc file formats (#169) BREAKING CHANGE: The `substrait/ReadRel/LocalFiles/format` field is deprecated. This will cause a hard break in compatibility. Newer consumers will not be able to read older files. Older consumers will not be able to read newer files. One should now express format concepts using the file_format oneof field. Co-authored-by: Jacques Nadeau <jacques@apache.org>

2022-05-20

Add support for alternative argument type support in the protobuf representation (type and option arguments) (#161) * feat: support type function arguments in protobuf Type arguments can now be passed to functions. This modifies the structure for specifying function arguments in general, deprecating the old structure. Co-authored-by: Jacques Nadeau <jacquesnadeau@gmail.com>

2022-05-16

feat: literals for extension types (#197) Add protobuf fields and messages to support specifying literals for user-defined types, and for specifying type variations in literals.

feat: support fractional seconds for interval_day literals (#199)

2022-05-13

feat: add flag FailureBehavior in Cast expression (#186)

2022-05-12

feat: add invocation property to AggregateFunction message for specifying distinct vs all (#191) * feat: add new invocation property to AggregateFunction message * docs: add AggregateFunction invocation docs Co-authored-by: James Taylor <james@qack.io>

2022-04-27

build(go): set Go package names and enable buf generate for Go (#168) Co-authored-by: Phillip Cloud <417981+cpcloud@users.noreply.github.com>

2022-04-07

fix: add missing switch expression (#160)

2022-04-05

style: format protos using buf format (#157)

2022-03-27

Added some comments to the .proto file explaining the different path types in a LocalFiles message (#133) * Added some comments to the .proto file explaining the different path types in a LocalFiles message

An updated Exchange rel. (#142) Update exchange rel documentation and proto definition.

2022-03-05

Add license header to .proto files (#140) * Add license header to .proto files * Add github action to check licence on .proto files

2022-02-07

Added a section explaining how slicing reads are handled (#109) * Added a section explaining how slicing reads are handled

2022-02-06

feat: add subquery representation (#134) * feat: add correlated subquery representation * refactor: rename operation enums * refactor: move IN predicate up * docs: add comments * chore: field ordering * docs: move comment up to message * style: format * docs: add subquery docs * style: format markdown

2022-02-05

refactor: combine relations and expressions into algebra.proto (#136) * refactor: move expression into relations * refactor: rename relations to algebra * chore: fix relations include

ci: proto-format-check CI job not working (#135)

2022-02-01

Add CrossProduct Relation and Single-join type (#132) * Add CrossProduct Relation and single-join type * Add description of missing join types and cross product op * Update single join description

2021-12-14

Update date type literal to be in days (#106)

2021-12-09

Enforce editorconfig settings in CI (#112) * Enforce editorconfig settings in CI * Add runs-on to editorconfig check

2021-12-07

Name _reference fields consistently (#107) (instead of using _pointer sometimes)

2021-12-06

Add partition info to ReadRel (#102) This closes #101

Propose alternative for empty map/list literals to reduce comment reliance. (#105)