package substrait

Get desktop application:
View/edit binary Protocol Buffers messages

An aggregate function.

Used in: AggregateRel.Measure, ExpressionReference

• uint32 function_reference = 1

  Points to a function_anchor defined in this plan, which must refer to an aggregate function in the associated YAML file. Required; 0 is considered to be a valid anchor/reference.

• repeated FunctionArgument arguments = 7

  The arguments to be bound to the function. This must have exactly the number of arguments specified in the function definition, and the argument types must also match exactly: - Value arguments must be bound using FunctionArgument.value, and the expression in that must yield a value of a type that a function overload is defined for. - Type arguments must be bound using FunctionArgument.type, and a function overload must be defined for that type. - Enum arguments must be bound using FunctionArgument.enum followed by Enum.specified, with a string that case-insensitively matches one of the allowed options. - Optional enum arguments must be bound using FunctionArgument.enum followed by either Enum.specified or Enum.unspecified. If specified, the string must case-insensitively match one of the allowed options.

• repeated FunctionOption options = 8

  Options to specify behavior for corner cases, or leave behavior unspecified if the consumer does not need specific behavior in these cases.

• optional Type output_type = 5

  Must be set to the return type of the function, exactly as derived using the declaration in the extension.

• AggregationPhase phase = 4

  Describes which part of the aggregation to perform within the context of distributed algorithms. Required. Must be set to INITIAL_TO_RESULT for aggregate functions that are not decomposable.

• repeated SortField sorts = 3

  If specified, the aggregated records are ordered according to this list before they are aggregated. The first sort field has the highest priority; only if a sort field determines two records to be equivalent is the next field queried. This field is optional.

• AggregateFunction.AggregationInvocation invocation = 6

  Specifies whether equivalent records are merged before being aggregated. Optional, defaults to AGGREGATION_INVOCATION_ALL.

• repeated Expression args = 2

  deprecated; use arguments instead

Method in which equivalent records are merged before being aggregated.

Used in: AggregateFunction, Expression.WindowFunction

• AGGREGATION_INVOCATION_UNSPECIFIED = 0

  This default value implies AGGREGATION_INVOCATION_ALL.

• AGGREGATION_INVOCATION_ALL = 1

  Use all values in the aggregation calculation.

• AGGREGATION_INVOCATION_DISTINCT = 2

  Use only distinct values in the aggregation calculation.

This rel is used to create references, in case we refer to a RelRoot field names will be ignored

• int32 subtree_ordinal = 1

The relational operator representing a GROUP BY Aggregate

Used in: Rel

• optional RelCommon common = 1

• optional Rel input = 2

  Input of the aggregation

• repeated AggregateRel.Grouping groupings = 3

  A list of expression grouping that the aggregation measured should be calculated for.

• repeated AggregateRel.Measure measures = 4

  A list of one or more aggregate expressions along with an optional filter.

• optional extensions.AdvancedExtension advanced_extension = 10

Used in: AggregateRel

• repeated Expression grouping_expressions = 1

Used in: AggregateRel

• optional AggregateFunction measure = 1

• optional Expression filter = 2

  An optional boolean expression that acts to filter which records are included in the measure. True means include this record for calculation within the measure. Helps to support SUM(<c>) FILTER(WHERE...) syntax without masking opportunities for optimization

Describes which part of an aggregation or window function to perform within the context of distributed algorithms.

Used in: AggregateFunction, Expression.WindowFunction

• AGGREGATION_PHASE_UNSPECIFIED = 0

  Implies `INTERMEDIATE_TO_RESULT`.

• AGGREGATION_PHASE_INITIAL_TO_INTERMEDIATE = 1

  Specifies that the function should be run only up to the point of generating an intermediate value, to be further aggregated later using INTERMEDIATE_TO_INTERMEDIATE or INTERMEDIATE_TO_RESULT.

• AGGREGATION_PHASE_INTERMEDIATE_TO_INTERMEDIATE = 2

  Specifies that the inputs of the aggregate or window function are the intermediate values of the function, and that the output should also be an intermediate value, to be further aggregated later using INTERMEDIATE_TO_INTERMEDIATE or INTERMEDIATE_TO_RESULT.

• AGGREGATION_PHASE_INITIAL_TO_RESULT = 3

  A complete invocation: the function should aggregate the given set of inputs to yield a single return value. This style must be used for aggregate or window functions that are not decomposable.

• AGGREGATION_PHASE_INTERMEDIATE_TO_RESULT = 4

  Specifies that the inputs of the aggregate or window function are the intermediate values of the function, generated previously using INITIAL_TO_INTERMEDIATE and possibly INTERMEDIATE_TO_INTERMEDIATE calls. This call should combine the intermediate values to yield the final return value.

Defines a set of Capabilities that a system (producer or consumer) supports.

• repeated string substrait_versions = 1

  List of Substrait versions this system supports

• repeated string advanced_extension_type_urls = 2

  list of com.google.Any message types this system supports for advanced extensions.

• repeated Capabilities.SimpleExtension simple_extensions = 3

  list of simple extensions this system supports.

Used in: Capabilities

• string uri = 1

• repeated string function_keys = 2

• repeated string type_keys = 3

• repeated string type_variation_keys = 4

Cartesian product relational operator of two tables (left and right)

Used in: Rel

• optional RelCommon common = 1

• optional Rel left = 2

• optional Rel right = 3

• optional extensions.AdvancedExtension advanced_extension = 10

• oneof write_type

  Definition of which type of object we are operating on

  • NamedObjectWrite named_object = 1

  • ExtensionObject extension_object = 2

• optional NamedStruct table_schema = 3

  The columns that will be modified (representing after-image of a schema change)

• optional Expression.Literal.Struct table_defaults = 4

  The default values for the columns (representing after-image of a schema change) E.g., in case of an ALTER TABLE that changes some of the column default values, we expect the table_defaults Struct to report a full list of default values reflecting the result of applying the ALTER TABLE operator successfully

• DdlRel.DdlObject object = 5

  Which type of object we operate on

• DdlRel.DdlOp op = 6

  The type of operation to perform

• optional Rel view_definition = 7

  The body of the CREATE VIEW

Used in: DdlRel

• DDL_OBJECT_UNSPECIFIED = 0

• DDL_OBJECT_TABLE = 1

  A Table object in the system

• DDL_OBJECT_VIEW = 2

  A View object in the system

Used in: DdlRel

• DDL_OP_UNSPECIFIED = 0

• DDL_OP_CREATE = 1

  A create operation (for any object)

• DDL_OP_CREATE_OR_REPLACE = 2

  A create operation if the object does not exist, or replaces it (equivalent to a DROP + CREATE) if the object already exists

• DDL_OP_ALTER = 3

  An operation that modifies the schema (e.g., column names, types, default values) for the target object

• DDL_OP_DROP = 4

  An operation that removes an object from the system

• DDL_OP_DROP_IF_EXIST = 5

  An operation that removes an object from the system (without throwing an exception if the object did not exist)

Used in: DerivationExpression.BinaryOp, DerivationExpression.ExpressionDecimal, DerivationExpression.ExpressionFixedBinary, DerivationExpression.ExpressionFixedChar, DerivationExpression.ExpressionList, DerivationExpression.ExpressionMap, DerivationExpression.ExpressionStruct, DerivationExpression.ExpressionVarChar, DerivationExpression.IfElse, DerivationExpression.ReturnProgram, DerivationExpression.ReturnProgram.Assignment, DerivationExpression.UnaryOp, FunctionSignature.Aggregate, FunctionSignature.Scalar, FunctionSignature.Window

• oneof kind

  • Type.Boolean bool = 1

  • Type.I8 i8 = 2

  • Type.I16 i16 = 3

  • Type.I32 i32 = 5

  • Type.I64 i64 = 7

  • Type.FP32 fp32 = 10

  • Type.FP64 fp64 = 11

  • Type.String string = 12

  • Type.Binary binary = 13

  • Type.Timestamp timestamp = 14

  • Type.Date date = 16

  • Type.Time time = 17

  • Type.IntervalYear interval_year = 19

  • Type.IntervalDay interval_day = 20

  • Type.TimestampTZ timestamp_tz = 29

  • Type.UUID uuid = 32

  • DerivationExpression.ExpressionFixedChar fixed_char = 21

  • DerivationExpression.ExpressionVarChar varchar = 22

  • DerivationExpression.ExpressionFixedBinary fixed_binary = 23

  • DerivationExpression.ExpressionDecimal decimal = 24

  • DerivationExpression.ExpressionStruct struct = 25

  • DerivationExpression.ExpressionList list = 27

  • DerivationExpression.ExpressionMap map = 28

  • DerivationExpression.ExpressionUserDefined user_defined = 30

  • uint32 user_defined_pointer = 31

    Deprecated in favor of user_defined, which allows nullability and variations to be specified. If user_defined_pointer is encountered, treat it as being non-nullable and having the default variation.

  • string type_parameter_name = 33

  • string integer_parameter_name = 34

  • int32 integer_literal = 35

  • DerivationExpression.UnaryOp unary_op = 36

  • DerivationExpression.BinaryOp binary_op = 37

  • DerivationExpression.IfElse if_else = 38

  • DerivationExpression.ReturnProgram return_program = 39

Used in: DerivationExpression

• BinaryOp.BinaryOpType op_type = 1

• optional DerivationExpression arg1 = 2

• optional DerivationExpression arg2 = 3

Used in: BinaryOp

• BINARY_OP_TYPE_UNSPECIFIED = 0

• BINARY_OP_TYPE_PLUS = 1

• BINARY_OP_TYPE_MINUS = 2

• BINARY_OP_TYPE_MULTIPLY = 3

• BINARY_OP_TYPE_DIVIDE = 4

• BINARY_OP_TYPE_MIN = 5

• BINARY_OP_TYPE_MAX = 6

• BINARY_OP_TYPE_GREATER_THAN = 7

• BINARY_OP_TYPE_LESS_THAN = 8

• BINARY_OP_TYPE_AND = 9

• BINARY_OP_TYPE_OR = 10

• BINARY_OP_TYPE_EQUALS = 11

• BINARY_OP_TYPE_COVERS = 12

Used in: DerivationExpression

• optional DerivationExpression scale = 1

• optional DerivationExpression precision = 2

• uint32 variation_pointer = 3

• Type.Nullability nullability = 4

Used in: DerivationExpression

• optional DerivationExpression length = 1

• uint32 variation_pointer = 2

• Type.Nullability nullability = 3

Used in: DerivationExpression

• optional DerivationExpression length = 1

• uint32 variation_pointer = 2

• Type.Nullability nullability = 3

Used in: DerivationExpression

• optional DerivationExpression type = 1

• uint32 variation_pointer = 2

• Type.Nullability nullability = 3

Used in: DerivationExpression

• optional DerivationExpression key = 1

• optional DerivationExpression value = 2

• uint32 variation_pointer = 3

• Type.Nullability nullability = 4

• repeated string names = 1

• optional ExpressionStruct struct = 2

Used in: DerivationExpression, ExpressionNamedStruct

• repeated DerivationExpression types = 1

• uint32 variation_pointer = 2

• Type.Nullability nullability = 3

Used in: DerivationExpression

• uint32 type_pointer = 1

• uint32 variation_pointer = 2

• Type.Nullability nullability = 3

Used in: DerivationExpression

• optional DerivationExpression length = 1

• uint32 variation_pointer = 2

• Type.Nullability nullability = 3

Used in: DerivationExpression

• optional DerivationExpression if_condition = 1

• optional DerivationExpression if_return = 2

• optional DerivationExpression else_return = 3

Used in: DerivationExpression

• repeated ReturnProgram.Assignment assignments = 1

• optional DerivationExpression final_expression = 2

Used in: ReturnProgram

• string name = 1

• optional DerivationExpression expression = 2

Used in: DerivationExpression

• UnaryOp.UnaryOpType op_type = 1

• optional DerivationExpression arg = 2

Used in: UnaryOp

• UNARY_OP_TYPE_UNSPECIFIED = 0

• UNARY_OP_TYPE_BOOLEAN_NOT = 1

A redistribution operation

• optional RelCommon common = 1

• optional Rel input = 2

• int32 partition_count = 3

• repeated ExchangeRel.ExchangeTarget targets = 4

• oneof exchange_kind

  the type of exchange used

  • ExchangeRel.ScatterFields scatter_by_fields = 5

  • ExchangeRel.SingleBucketExpression single_target = 6

  • ExchangeRel.MultiBucketExpression multi_target = 7

  • ExchangeRel.RoundRobin round_robin = 8

  • ExchangeRel.Broadcast broadcast = 9

• optional extensions.AdvancedExtension advanced_extension = 10

Send all data to every target.

Used in: ExchangeRel

(message has no fields)

The message to describe partition targets of an exchange

Used in: ExchangeRel

• repeated int32 partition_id = 1

  Describes the partition id(s) to send. If this is empty, all data is sent to this target.

• oneof target_type

  • string uri = 2

  • google.protobuf.Any extended = 3

Returns zero or more bucket numbers per record

Used in: ExchangeRel

• optional Expression expression = 1

• bool constrained_to_count = 2

Route approximately

Used in: ExchangeRel

• bool exact = 1

  whether the round robin behavior is required to exact (per record) or approximate. Defaults to approximate.

Used in: ExchangeRel

• repeated Expression.FieldReference fields = 1

Returns a single bucket number per record.

Used in: ExchangeRel

• optional Expression expression = 1

Used in: AggregateFunction, AggregateRel.Grouping, AggregateRel.Measure, ExchangeRel.MultiBucketExpression, ExchangeRel.SingleBucketExpression, Expression.Cast, Expression.EmbeddedFunction, Expression.FieldReference, Expression.IfThen, Expression.IfThen.IfClause, Expression.MultiOrList, Expression.MultiOrList.Record, Expression.Nested.List, Expression.Nested.Map.KeyValue, Expression.Nested.Struct, Expression.ScalarFunction, Expression.SingularOrList, Expression.Subquery.InPredicate, Expression.Subquery.SetComparison, Expression.SwitchExpression, Expression.SwitchExpression.IfValue, Expression.WindowFunction, ExpressionReference, FilterRel, FunctionArgument, HashJoinRel, JoinRel, MergeJoinRel, ProjectRel, ReadRel, SortField

• oneof rex_type

  • Expression.Literal literal = 1

  • Expression.FieldReference selection = 2

  • Expression.ScalarFunction scalar_function = 3

  • Expression.WindowFunction window_function = 5

  • Expression.IfThen if_then = 6

  • Expression.SwitchExpression switch_expression = 7

  • Expression.SingularOrList singular_or_list = 8

  • Expression.MultiOrList multi_or_list = 9

  • Expression.Cast cast = 11

  • Expression.Subquery subquery = 12

  • Expression.Nested nested = 13

  • Expression.Enum enum = 10

    deprecated: enum literals are only sensible in the context of function arguments, for which FunctionArgument should now be used

Used in: Expression

• optional Type type = 1

• optional Expression input = 2

• Cast.FailureBehavior failure_behavior = 3

Used in: Cast

• FAILURE_BEHAVIOR_UNSPECIFIED = 0

• FAILURE_BEHAVIOR_RETURN_NULL = 1

• FAILURE_BEHAVIOR_THROW_EXCEPTION = 2

• repeated Expression arguments = 1

• optional Type output_type = 2

• oneof kind

  • EmbeddedFunction.PythonPickleFunction python_pickle_function = 3

  • EmbeddedFunction.WebAssemblyFunction web_assembly_function = 4

Used in: EmbeddedFunction

• bytes function = 1

• repeated string prerequisite = 2

Used in: EmbeddedFunction

• bytes script = 1

• repeated string prerequisite = 2

Used in: Expression

• oneof enum_kind

  • string specified = 1

  • Enum.Empty unspecified = 2

Used in: Enum

(message has no fields)

A reference to an inner part of a complex object. Can reference reference a single element or a masked version of elements

Used in: ExchangeRel.ScatterFields, Expression, HashJoinRel, MergeJoinRel

• oneof reference_type

  Whether this is composed of a single element reference or a masked element subtree

  • ReferenceSegment direct_reference = 1

  • MaskExpression masked_reference = 2

• oneof root_type

  Whether this reference has an origin of a root struct or is based on the ouput of an expression. When this is a RootReference and direct_reference above is used, the direct_reference must be of a type StructField.

  • Expression expression = 3

  • FieldReference.RootReference root_reference = 4

  • FieldReference.OuterReference outer_reference = 5

A root reference for the outer relation's subquery

Used in: FieldReference

• uint32 steps_out = 1

  number of subquery boundaries to traverse up for this field's reference This value must be >= 1

Singleton that expresses this FieldReference is rooted off the root incoming record type

Used in: FieldReference

(message has no fields)

Used in: Expression

• repeated IfThen.IfClause ifs = 1

• optional Expression else = 2

Used in: IfThen

• optional Expression if = 1

• optional Expression then = 2

Used in: Expression, Literal.List, Literal.Map.KeyValue, Literal.Struct, ReferenceSegment.MapKey, SwitchExpression.IfValue

• oneof literal_type

  • bool boolean = 1

  • int32 i8 = 2

  • int32 i16 = 3

  • int32 i32 = 5

  • int64 i64 = 7

  • float fp32 = 10

  • double fp64 = 11

  • string string = 12

  • bytes binary = 13

  • int64 timestamp = 14

    Timestamp in units of microseconds since the UNIX epoch.

  • int32 date = 16

    Date in units of days since the UNIX epoch.

  • int64 time = 17

    Time in units of microseconds past midnight

  • Literal.IntervalYearToMonth interval_year_to_month = 19

  • Literal.IntervalDayToSecond interval_day_to_second = 20

  • string fixed_char = 21

  • Literal.VarChar var_char = 22

  • bytes fixed_binary = 23

  • Literal.Decimal decimal = 24

  • Literal.Struct struct = 25

  • Literal.Map map = 26

  • int64 timestamp_tz = 27

    Timestamp in units of microseconds since the UNIX epoch.

  • bytes uuid = 28

  • Type null = 29

    a typed null literal

  • Literal.List list = 30

  • Type.List empty_list = 31

  • Type.Map empty_map = 32

  • Literal.UserDefined user_defined = 33

• bool nullable = 50

  whether the literal type should be treated as a nullable type. Applies to all members of union other than the Typed null (which should directly declare nullability).

• uint32 type_variation_reference = 51

  optionally points to a type_variation_anchor defined in this plan. Applies to all members of union other than the Typed null (which should directly declare the type variation).

Used in: Literal

• bytes value = 1

  little-endian twos-complement integer representation of complete value (ignoring precision) Always 16 bytes in length

• int32 precision = 2

  The maximum number of digits allowed in the value. the maximum precision is 38.

• int32 scale = 3

  declared scale of decimal literal

Used in: Literal

• int32 days = 1

• int32 seconds = 2

• int32 microseconds = 3

Used in: Literal

• int32 years = 1

• int32 months = 2

Used in: Literal

• repeated Literal values = 1

  A homogeneously typed list of literals

Used in: Literal

• repeated Map.KeyValue key_values = 1

Used in: Map

• optional Literal key = 1

• optional Literal value = 2

Used in: DdlRel, Literal, ReadRel.VirtualTable

• repeated Literal fields = 1

  A possibly heterogeneously typed list of literals

Used in: Literal

• uint32 type_reference = 1

  points to a type_anchor defined in this plan

• repeated Type.Parameter type_parameters = 3

  The parameters to be bound to the type class, if the type class is parameterizable.

• optional google.protobuf.Any value = 2

  the value of the literal, serialized using some type-specific protobuf message

Used in: Literal

• string value = 1

• uint32 length = 2

A reference that takes an existing subtype and selectively removes fields from it. For example, one might initially have an inner struct with 100 fields but a a particular operation only needs to interact with only 2 of those 100 fields. In this situation, one would use a mask expression to eliminate the 98 fields that are not relevant to the rest of the operation pipeline. Note that this does not fundamentally alter the structure of data beyond the elimination of unecessary elements.

Used in: FieldReference, ReadRel

• optional MaskExpression.StructSelect select = 1

• bool maintain_singular_struct = 2

Used in: Select

• repeated ListSelect.ListSelectItem selection = 1

• optional Select child = 2

Used in: ListSelect

• oneof type

  • ListSelectItem.ListElement item = 1

  • ListSelectItem.ListSlice slice = 2

Used in: ListSelectItem

• int32 field = 1

Used in: ListSelectItem

• int32 start = 1

• int32 end = 2

Used in: Select

• oneof select

  • MapSelect.MapKey key = 1

  • MapSelect.MapKeyExpression expression = 2

• optional Select child = 3

Used in: MapSelect

• string map_key = 1

Used in: MapSelect

• string map_key_expression = 1

Used in: ListSelect, MapSelect, StructItem

• oneof type

  • StructSelect struct = 1

  • ListSelect list = 2

  • MapSelect map = 3

Used in: StructSelect

• int32 field = 1

• optional Select child = 2

Used in: MaskExpression, Select

• repeated StructItem struct_items = 1

Used in: Expression

• repeated Expression value = 1

• repeated MultiOrList.Record options = 2

Used in: MultiOrList

• repeated Expression fields = 1

Expression to dynamically construct nested types.

Used in: Expression

• bool nullable = 1

  Whether the returned nested type is nullable.

• uint32 type_variation_reference = 2

  Optionally points to a type_variation_anchor defined in this plan for the returned nested type.

• oneof nested_type

  • Nested.Struct struct = 3

  • Nested.List list = 4

  • Nested.Map map = 5

Used in: Nested

• repeated Expression values = 1

  A homogeneously-typed list of one or more expressions that form the list entries. To specify an empty list, use Literal.empty_list (otherwise type information would be missing).

Used in: Nested

• repeated Map.KeyValue key_values = 1

  One or more key-value pairs. To specify an empty map, use Literal.empty_map (otherwise type information would be missing).

Used in: Map

• optional Expression key = 1

  Mandatory key/value expressions.

• optional Expression value = 2

Used in: Nested

• repeated Expression fields = 1

  Zero or more possibly heterogeneously-typed list of expressions that form the struct fields.

A way to reference the inner property of a complex record. Can reference either a map key by literal, a struct field by the ordinal position of the desired field or a particular element in an array. Supports expressions that would roughly translate to something similar to: a.b[2].c['my_map_key'].x where a,b,c and x are struct field references (ordinalized in the internal representation here), [2] is a list offset and ['my_map_key'] is a reference into a map field.

Used in: FieldReference, ReferenceSegment.ListElement, ReferenceSegment.MapKey, ReferenceSegment.StructField

• oneof reference_type

  • ReferenceSegment.MapKey map_key = 1

  • ReferenceSegment.StructField struct_field = 2

  • ReferenceSegment.ListElement list_element = 3

Used in: ReferenceSegment

• int32 offset = 1

  zero-indexed ordinal position of element in list

• optional ReferenceSegment child = 2

  Optional child segment

Used in: ReferenceSegment

• optional Literal map_key = 1

  literal based reference to specific possible value in map.

• optional ReferenceSegment child = 2

  Optional child segment

Used in: ReferenceSegment

• int32 field = 1

  zero-indexed ordinal position of field in struct

• optional ReferenceSegment child = 2

  Optional child segment

A scalar function call.

Used in: Expression

• uint32 function_reference = 1

  Points to a function_anchor defined in this plan, which must refer to a scalar function in the associated YAML file. Required; avoid using anchor/reference zero.

• repeated FunctionArgument arguments = 4

  The arguments to be bound to the function. This must have exactly the number of arguments specified in the function definition, and the argument types must also match exactly: - Value arguments must be bound using FunctionArgument.value, and the expression in that must yield a value of a type that a function overload is defined for. - Type arguments must be bound using FunctionArgument.type. - Enum arguments must be bound using FunctionArgument.enum followed by Enum.specified, with a string that case-insensitively matches one of the allowed options.

• repeated FunctionOption options = 5

  Options to specify behavior for corner cases, or leave behavior unspecified if the consumer does not need specific behavior in these cases.

• optional Type output_type = 3

  Must be set to the return type of the function, exactly as derived using the declaration in the extension.

• repeated Expression args = 2

  Deprecated; use arguments instead.

Used in: Expression

• optional Expression value = 1

• repeated Expression options = 2

Subquery relation expression

Used in: Expression

• oneof subquery_type

  • Subquery.Scalar scalar = 1

    Scalar subquery

  • Subquery.InPredicate in_predicate = 2

    x IN y predicate

  • Subquery.SetPredicate set_predicate = 3

    EXISTS/UNIQUE predicate

  • Subquery.SetComparison set_comparison = 4

    ANY/ALL predicate

Predicate checking that the left expression is contained in the right subquery Examples: x IN (SELECT * FROM t) (x, y) IN (SELECT a, b FROM t)

Used in: Subquery

• repeated Expression needles = 1

• optional Rel haystack = 2

A subquery with one row and one column. This is often an aggregate though not required to be.

Used in: Subquery

• optional Rel input = 1

A subquery comparison using ANY or ALL. Examples: SELECT * FROM t1 WHERE x < ANY(SELECT y from t2)

Used in: Subquery

• SetComparison.ReductionOp reduction_op = 1

  ANY or ALL

• SetComparison.ComparisonOp comparison_op = 2

  A comparison operator

• optional Expression left = 3

  left side of the expression

• optional Rel right = 4

  right side of the expression

Used in: SetComparison

• COMPARISON_OP_UNSPECIFIED = 0

• COMPARISON_OP_EQ = 1

• COMPARISON_OP_NE = 2

• COMPARISON_OP_LT = 3

• COMPARISON_OP_GT = 4

• COMPARISON_OP_LE = 5

• COMPARISON_OP_GE = 6

Used in: SetComparison

• REDUCTION_OP_UNSPECIFIED = 0

• REDUCTION_OP_ANY = 1

• REDUCTION_OP_ALL = 2

A predicate over a set of rows in the form of a subquery EXISTS and UNIQUE are common SQL forms of this operation.

Used in: Subquery

• SetPredicate.PredicateOp predicate_op = 1

  TODO: should allow expressions

• optional Rel tuples = 2

Used in: SetPredicate

• PREDICATE_OP_UNSPECIFIED = 0

• PREDICATE_OP_EXISTS = 1

• PREDICATE_OP_UNIQUE = 2

Used in: Expression

• optional Expression match = 3

• repeated SwitchExpression.IfValue ifs = 1

• optional Expression else = 2

Used in: SwitchExpression

• optional Literal if = 1

• optional Expression then = 2

A window function call.

Used in: Expression

• uint32 function_reference = 1

  Points to a function_anchor defined in this plan, which must refer to a window function in the associated YAML file. Required; 0 is considered to be a valid anchor/reference.

• repeated FunctionArgument arguments = 9

  The arguments to be bound to the function. This must have exactly the number of arguments specified in the function definition, and the argument types must also match exactly: - Value arguments must be bound using FunctionArgument.value, and the expression in that must yield a value of a type that a function overload is defined for. - Type arguments must be bound using FunctionArgument.type, and a function overload must be defined for that type. - Enum arguments must be bound using FunctionArgument.enum followed by Enum.specified, with a string that case-insensitively matches one of the allowed options.

• repeated FunctionOption options = 11

  Options to specify behavior for corner cases, or leave behavior unspecified if the consumer does not need specific behavior in these cases.

• optional Type output_type = 7

  Must be set to the return type of the function, exactly as derived using the declaration in the extension.

• AggregationPhase phase = 6

  Describes which part of the window function to perform within the context of distributed algorithms. Required. Must be set to INITIAL_TO_RESULT for window functions that are not decomposable.

• repeated SortField sorts = 3

  If specified, the records that are part of the window defined by upper_bound and lower_bound are ordered according to this list before they are aggregated. The first sort field has the highest priority; only if a sort field determines two records to be equivalent is the next field queried. This field is optional, and is only allowed if the window function is defined to support sorting.

• AggregateFunction.AggregationInvocation invocation = 10

  Specifies whether equivalent records are merged before being aggregated. Optional, defaults to AGGREGATION_INVOCATION_ALL.

• repeated Expression partitions = 2

  When one or more partition expressions are specified, two records are considered to be in the same partition if and only if these expressions yield an equal tuple of values for both. When computing the window function, only the subset of records within the bounds that are also in the same partition as the current record are aggregated.

• optional WindowFunction.Bound lower_bound = 5

  Defines the record relative to the current record from which the window extends. The bound is inclusive. If the lower bound indexes a record greater than the upper bound, TODO (null range/no records passed? wrapping around as if lower/upper were swapped? error? null?). Optional; defaults to the start of the partition.

• optional WindowFunction.Bound upper_bound = 4

  Defines the record relative to the current record up to which the window extends. The bound is inclusive. If the upper bound indexes a record less than the lower bound, TODO (null range/no records passed? wrapping around as if lower/upper were swapped? error? null?). Optional; defaults to the end of the partition.

• repeated Expression args = 8

  Deprecated; use arguments instead.

Defines one of the two boundaries for the window of a window function.

Used in: WindowFunction

• oneof kind

  • Bound.Preceding preceding = 1

    The bound extends some number of records behind the current record.

  • Bound.Following following = 2

    The bound extends some number of records ahead of the current record.

  • Bound.CurrentRow current_row = 3

    The bound extends to the current record.

  • Bound.Unbounded unbounded = 4

    The bound extends to the start of the partition or the end of the partition, depending on whether this represents the upper or lower bound.

Defines that the bound extends to or from the current record.

Used in: Bound

(message has no fields)

Defines that the bound extends this far ahead of the current record.

Used in: Bound

• int64 offset = 1

  A strictly positive integer specifying the number of records that the window extends ahead of the current record. Required. Use CurrentRow for offset zero and Preceding for negative offsets.

Defines that the bound extends this far back from the current record.

Used in: Bound

• int64 offset = 1

  A strictly positive integer specifying the number of records that the window extends back from the current record. Required. Use CurrentRow for offset zero and Following for negative offsets.

Defines an "unbounded bound": for lower bounds this means the start of the partition, and for upper bounds this means the end of the partition.

Used in: Bound

(message has no fields)

Used in: ExtendedExpression

• oneof expr_type

  • Expression expression = 1

  • AggregateFunction measure = 2

• repeated string output_names = 3

  Field names in depth-first order

Describe a set of operations to complete. For compactness sake, identifiers are normalized at the plan level.

• optional Version version = 7

  Substrait version of the expression. Optional up to 0.17.0, required for later versions.

• repeated extensions.SimpleExtensionURI extension_uris = 1

  a list of yaml specifications this expression may depend on

• repeated extensions.SimpleExtensionDeclaration extensions = 2

  a list of extensions this expression may depend on

• repeated ExpressionReference referred_expr = 3

  one or more expression trees with same order in plan rel

• optional NamedStruct base_schema = 4

• optional extensions.AdvancedExtension advanced_extensions = 5

  additional extensions associated with this expression.

• repeated string expected_type_urls = 6

  A list of com.google.Any entities that this plan may use. Can be used to warn if some embedded message types are unknown. Note that this list may include message types that are ignorable (optimizations) or that are unused. In many cases, a consumer may be able to work with a plan even if one or more message types defined here are unknown.

Stub to support extension with a zero inputs

Used in: Rel

• optional RelCommon common = 1

• optional google.protobuf.Any detail = 2

Stub to support extension with multiple inputs

Used in: Rel

• optional RelCommon common = 1

• repeated Rel inputs = 2

• optional google.protobuf.Any detail = 3

A stub type that can be used to extend/introduce new table types outside the specification.

Used in: DdlRel, WriteRel

• optional google.protobuf.Any detail = 1

Stub to support extension with a single input

Used in: Rel

• optional RelCommon common = 1

• optional Rel input = 2

• optional google.protobuf.Any detail = 3

The relational operator representing LIMIT/OFFSET or TOP type semantics.

Used in: Rel

• optional RelCommon common = 1

• optional Rel input = 2

• int64 offset = 3

  the offset expressed in number of records

• int64 count = 4

  the amount of records to return

• optional extensions.AdvancedExtension advanced_extension = 10

The relational operator capturing simple FILTERs (as in the WHERE clause of SQL)

Used in: Rel

• optional RelCommon common = 1

• optional Rel input = 2

• optional Expression condition = 3

• optional extensions.AdvancedExtension advanced_extension = 10

The argument of a function

Used in: AggregateFunction, Expression.ScalarFunction, Expression.WindowFunction

• oneof arg_type

  • string enum = 1

  • Type type = 2

  • Expression value = 3

An optional function argument. Typically used for specifying behavior in invalid or corner cases.

Used in: AggregateFunction, Expression.ScalarFunction, Expression.WindowFunction

• string name = 1

  Name of the option to set. If the consumer does not recognize the option, it must reject the plan. The name is matched case-insensitively with option names defined for the function.

• repeated string preference = 2

  List of behavior options allowed by the producer. At least one must be specified; to leave an option unspecified, simply don't add an entry to `options`. The consumer must use the first option from the list that it supports. If the consumer supports none of the specified options, it must reject the plan. The name is matched case-insensitively and must match one of the option values defined for the option.

List of function signatures available.

(message has no fields)

• repeated Argument arguments = 2

• string name = 3

• optional Description description = 4

• bool deterministic = 7

• bool session_dependent = 8

• optional DerivationExpression output_type = 9

• oneof final_variable_behavior

  • FinalArgVariadic variadic = 10

  • FinalArgNormal normal = 11

• bool ordered = 14

• uint64 max_set = 12

• optional Type intermediate_type = 13

• repeated Implementation implementations = 15

Used in: Aggregate, Scalar, Window

• string name = 1

• oneof argument_kind

  • Argument.ValueArgument value = 2

  • Argument.TypeArgument type = 3

  • Argument.EnumArgument enum = 4

Used in: Argument

• repeated string options = 1

• bool optional = 2

Used in: Argument

• optional ParameterizedType type = 1

Used in: Argument

• optional ParameterizedType type = 1

• bool constant = 2

Used in: Aggregate, Scalar, Window

• string language = 1

• string body = 2

Used in: Aggregate, Scalar, Window

(message has no fields)

Used in: Aggregate, Scalar, Window

• int64 min_args = 1

  the minimum number of arguments allowed for the list of final arguments (inclusive).

• int64 max_args = 2

  the maximum number of arguments allowed for the list of final arguments (exclusive)

• FinalArgVariadic.ParameterConsistency consistency = 3

  the type of parameterized type consistency

Used in: FinalArgVariadic

• PARAMETER_CONSISTENCY_UNSPECIFIED = 0

• PARAMETER_CONSISTENCY_CONSISTENT = 1

  All argument must be the same concrete type.

• PARAMETER_CONSISTENCY_INCONSISTENT = 2

  Each argument can be any possible concrete type afforded by the bounds of any parameter defined in the arguments specification.

Used in: Aggregate, Scalar, Window

• Implementation.Type type = 1

• string uri = 2

Used in: Implementation

• TYPE_UNSPECIFIED = 0

• TYPE_WEB_ASSEMBLY = 1

• TYPE_TRINO_JAR = 2

• repeated Argument arguments = 2

• repeated string name = 3

• optional Description description = 4

• bool deterministic = 7

• bool session_dependent = 8

• optional DerivationExpression output_type = 9

• oneof final_variable_behavior

  • FinalArgVariadic variadic = 10

  • FinalArgNormal normal = 11

• repeated Implementation implementations = 12

• repeated Argument arguments = 2

• repeated string name = 3

• optional Description description = 4

• bool deterministic = 7

• bool session_dependent = 8

• optional DerivationExpression intermediate_type = 9

• optional DerivationExpression output_type = 10

• oneof final_variable_behavior

  • FinalArgVariadic variadic = 16

  • FinalArgNormal normal = 17

• bool ordered = 11

• uint64 max_set = 12

• Window.WindowType window_type = 14

• repeated Implementation implementations = 15

Used in: Window

• WINDOW_TYPE_UNSPECIFIED = 0

• WINDOW_TYPE_STREAMING = 1

• WINDOW_TYPE_PARTITION = 2

The hash equijoin join operator will build a hash table out of the right input based on a set of join keys. It will then probe that hash table for incoming inputs, finding matches.

Used in: Rel

• optional RelCommon common = 1

• optional Rel left = 2

• optional Rel right = 3

• repeated Expression.FieldReference left_keys = 4

• repeated Expression.FieldReference right_keys = 5

• optional Expression post_join_filter = 6

• HashJoinRel.JoinType type = 7

• optional extensions.AdvancedExtension advanced_extension = 10

Used in: HashJoinRel

• JOIN_TYPE_UNSPECIFIED = 0

• JOIN_TYPE_INNER = 1

• JOIN_TYPE_OUTER = 2

• JOIN_TYPE_LEFT = 3

• JOIN_TYPE_RIGHT = 4

• JOIN_TYPE_LEFT_SEMI = 5

• JOIN_TYPE_RIGHT_SEMI = 6

• JOIN_TYPE_LEFT_ANTI = 7

• JOIN_TYPE_RIGHT_ANTI = 8

The binary JOIN relational operator left-join-right, including various join types, a join condition and post_join_filter expression

Used in: Rel

• optional RelCommon common = 1

• optional Rel left = 2

• optional Rel right = 3

• optional Expression expression = 4

• optional Expression post_join_filter = 5

• JoinRel.JoinType type = 6

• optional extensions.AdvancedExtension advanced_extension = 10

Used in: JoinRel

• JOIN_TYPE_UNSPECIFIED = 0

• JOIN_TYPE_INNER = 1

• JOIN_TYPE_OUTER = 2

• JOIN_TYPE_LEFT = 3

• JOIN_TYPE_RIGHT = 4

• JOIN_TYPE_SEMI = 5

• JOIN_TYPE_ANTI = 6

• JOIN_TYPE_SINGLE = 7

  This join is useful for nested sub-queries where we need exactly one tuple in output (or throw exception) See Section 3.2 of https://15721.courses.cs.cmu.edu/spring2018/papers/16-optimizer2/hyperjoins-btw2017.pdf

The merge equijoin does a join by taking advantage of two sets that are sorted on the join keys. This allows the join operation to be done in a streaming fashion.

Used in: Rel

• optional RelCommon common = 1

• optional Rel left = 2

• optional Rel right = 3

• repeated Expression.FieldReference left_keys = 4

• repeated Expression.FieldReference right_keys = 5

• optional Expression post_join_filter = 6

• MergeJoinRel.JoinType type = 7

• optional extensions.AdvancedExtension advanced_extension = 10

Used in: MergeJoinRel

• JOIN_TYPE_UNSPECIFIED = 0

• JOIN_TYPE_INNER = 1

• JOIN_TYPE_OUTER = 2

• JOIN_TYPE_LEFT = 3

• JOIN_TYPE_RIGHT = 4

• JOIN_TYPE_LEFT_SEMI = 5

• JOIN_TYPE_RIGHT_SEMI = 6

• JOIN_TYPE_LEFT_ANTI = 7

• JOIN_TYPE_RIGHT_ANTI = 8

A base object for writing (e.g., a table or a view).

Used in: DdlRel, WriteRel

• repeated string names = 1

  The list of string is used to represent namespacing (e.g., mydb.mytable). This assumes shared catalog between systems exchanging a message.

• optional extensions.AdvancedExtension advanced_extension = 10

A message for modeling name/type pairs. Useful for representing relation schemas. Notes: * The names field is in depth-first order. For example a schema such as: a: int64 b: struct<c: float32, d: string> would have a `names` field that looks like: ["a", "b", "c", "d"] * Only struct fields are contained in this field's elements, * Map keys should be traversed first, then values when producing/consuming

Used in: DdlRel, ExtendedExpression, ReadRel, WriteRel

• repeated string names = 1

  list of names in dfs order

• optional Type.Struct struct = 2

Used in: FunctionSignature.Argument.TypeArgument, FunctionSignature.Argument.ValueArgument, ParameterizedType.ParameterizedList, ParameterizedType.ParameterizedMap, ParameterizedType.ParameterizedStruct, ParameterizedType.TypeParameter

• oneof kind

  • Type.Boolean bool = 1

  • Type.I8 i8 = 2

  • Type.I16 i16 = 3

  • Type.I32 i32 = 5

  • Type.I64 i64 = 7

  • Type.FP32 fp32 = 10

  • Type.FP64 fp64 = 11

  • Type.String string = 12

  • Type.Binary binary = 13

  • Type.Timestamp timestamp = 14

  • Type.Date date = 16

  • Type.Time time = 17

  • Type.IntervalYear interval_year = 19

  • Type.IntervalDay interval_day = 20

  • Type.TimestampTZ timestamp_tz = 29

  • Type.UUID uuid = 32

  • ParameterizedType.ParameterizedFixedChar fixed_char = 21

  • ParameterizedType.ParameterizedVarChar varchar = 22

  • ParameterizedType.ParameterizedFixedBinary fixed_binary = 23

  • ParameterizedType.ParameterizedDecimal decimal = 24

  • ParameterizedType.ParameterizedStruct struct = 25

  • ParameterizedType.ParameterizedList list = 27

  • ParameterizedType.ParameterizedMap map = 28

  • ParameterizedType.ParameterizedUserDefined user_defined = 30

  • uint32 user_defined_pointer = 31

    Deprecated in favor of user_defined, which allows nullability and variations to be specified. If user_defined_pointer is encountered, treat it as being non-nullable and having the default variation.

  • ParameterizedType.TypeParameter type_parameter = 33

Used in: ParameterizedDecimal, ParameterizedFixedBinary, ParameterizedFixedChar, ParameterizedVarChar

• oneof integer_type

  • int32 literal = 1

  • IntegerParameter parameter = 2

Used in: IntegerOption

• string name = 1

• optional NullableInteger range_start_inclusive = 2

• optional NullableInteger range_end_exclusive = 3

Used in: IntegerParameter

• int64 value = 1

Used in: ParameterizedType

• optional IntegerOption scale = 1

• optional IntegerOption precision = 2

• uint32 variation_pointer = 3

• Type.Nullability nullability = 4

Used in: ParameterizedType

• optional IntegerOption length = 1

• uint32 variation_pointer = 2

• Type.Nullability nullability = 3

Used in: ParameterizedType

• optional IntegerOption length = 1

• uint32 variation_pointer = 2

• Type.Nullability nullability = 3

Used in: ParameterizedType

• optional ParameterizedType type = 1

• uint32 variation_pointer = 2

• Type.Nullability nullability = 3

Used in: ParameterizedType

• optional ParameterizedType key = 1

• optional ParameterizedType value = 2

• uint32 variation_pointer = 3

• Type.Nullability nullability = 4

• repeated string names = 1

  list of names in dfs order

• optional ParameterizedStruct struct = 2

Used in: ParameterizedType, ParameterizedNamedStruct

• repeated ParameterizedType types = 1

• uint32 variation_pointer = 2

• Type.Nullability nullability = 3

Used in: ParameterizedType

• uint32 type_pointer = 1

• uint32 variation_pointer = 2

• Type.Nullability nullability = 3

Used in: ParameterizedType

• optional IntegerOption length = 1

• uint32 variation_pointer = 2

• Type.Nullability nullability = 3

Used in: ParameterizedType

• string name = 1

• repeated ParameterizedType bounds = 2

Describe a set of operations to complete. For compactness sake, identifiers are normalized at the plan level.

• optional Version version = 6

  Substrait version of the plan. Optional up to 0.17.0, required for later versions.

• repeated extensions.SimpleExtensionURI extension_uris = 1

  a list of yaml specifications this plan may depend on

• repeated extensions.SimpleExtensionDeclaration extensions = 2

  a list of extensions this plan may depend on

• repeated PlanRel relations = 3

  one or more relation trees that are associated with this plan.

• optional extensions.AdvancedExtension advanced_extensions = 4

  additional extensions associated with this plan.

• repeated string expected_type_urls = 5

  A list of com.google.Any entities that this plan may use. Can be used to warn if some embedded message types are unknown. Note that this list may include message types that are ignorable (optimizations) or that are unused. In many cases, a consumer may be able to work with a plan even if one or more message types defined here are unknown.

Either a relation or root relation

Used in: Plan

• oneof rel_type

  • Rel rel = 1

    Any relation (used for references and CTEs)

  • RelRoot root = 2

    The root of a relation tree

This message type can be used to deserialize only the version of a Substrait Plan message. This prevents deserialization errors when there were breaking changes between the Substrait version of the tool that produced the plan and the Substrait version used to deserialize it, such that a consumer can emit a more helpful error message in this case.

• optional Version version = 6

This operator allows to represent calculated expressions of fields (e.g., a+b). Direct/Emit are used to represent classical relational projections

Used in: Rel

• optional RelCommon common = 1

• optional Rel input = 2

• repeated Expression expressions = 3

• optional extensions.AdvancedExtension advanced_extension = 10

The scan operator of base data (physical or virtual), including filtering and projection.

Used in: Rel

• optional RelCommon common = 1

• optional NamedStruct base_schema = 2

• optional Expression filter = 3

• optional Expression best_effort_filter = 11

• optional Expression.MaskExpression projection = 4

• optional extensions.AdvancedExtension advanced_extension = 10

• oneof read_type

  Definition of which type of scan operation is to be performed

  • ReadRel.VirtualTable virtual_table = 5

  • ReadRel.LocalFiles local_files = 6

  • ReadRel.NamedTable named_table = 7

  • ReadRel.ExtensionTable extension_table = 8

A stub type that can be used to extend/introduce new table types outside the specification.

Used in: ReadRel

• optional google.protobuf.Any detail = 1

Represents a list of files in input of a scan operation

Used in: ReadRel

• repeated LocalFiles.FileOrFiles items = 1

• optional extensions.AdvancedExtension advanced_extension = 10

Many files consist of indivisible chunks (e.g. parquet row groups or CSV rows). If a slice partially selects an indivisible chunk then the consumer should employ some rule to decide which slice to include the chunk in (e.g. include it in the slice that contains the midpoint of the chunk)

Used in: LocalFiles

• oneof path_type

  • string uri_path = 1

    A URI that can refer to either a single folder or a single file

  • string uri_path_glob = 2

    A URI where the path portion is a glob expression that can identify zero or more paths. Consumers should support the POSIX syntax. The recursive globstar (**) may not be supported.

  • string uri_file = 3

    A URI that refers to a single file

  • string uri_folder = 4

    A URI that refers to a single folder

• uint64 partition_index = 6

  The index of the partition this item belongs to

• uint64 start = 7

  The start position in byte to read from this item

• uint64 length = 8

  The length in byte to read from this item

• oneof file_format

  The format of the files.

  • FileOrFiles.ParquetReadOptions parquet = 9

  • FileOrFiles.ArrowReadOptions arrow = 10

  • FileOrFiles.OrcReadOptions orc = 11

  • google.protobuf.Any extension = 12

  • FileOrFiles.DwrfReadOptions dwrf = 13

Used in: FileOrFiles

(message has no fields)

Used in: FileOrFiles

(message has no fields)

Used in: FileOrFiles

(message has no fields)

Used in: FileOrFiles

(message has no fields)

A base table. The list of string is used to represent namespacing (e.g., mydb.mytable). This assumes shared catalog between systems exchanging a message.

Used in: ReadRel

• repeated string names = 1

• optional extensions.AdvancedExtension advanced_extension = 10

A table composed of literals.

Used in: ReadRel

• repeated Expression.Literal.Struct values = 1

A relation (used internally in a plan)

Used in: AggregateRel, CrossRel, DdlRel, ExchangeRel, Expression.Subquery.InPredicate, Expression.Subquery.Scalar, Expression.Subquery.SetComparison, Expression.Subquery.SetPredicate, ExtensionMultiRel, ExtensionSingleRel, FetchRel, FilterRel, HashJoinRel, JoinRel, MergeJoinRel, PlanRel, ProjectRel, RelRoot, SetRel, SortRel, WriteRel

• oneof rel_type

  • ReadRel read = 1

  • FilterRel filter = 2

  • FetchRel fetch = 3

  • AggregateRel aggregate = 4

  • SortRel sort = 5

  • JoinRel join = 6

  • ProjectRel project = 7

  • SetRel set = 8

  • ExtensionSingleRel extension_single = 9

  • ExtensionMultiRel extension_multi = 10

  • ExtensionLeafRel extension_leaf = 11

  • CrossRel cross = 12

  • HashJoinRel hash_join = 13

    Physical relations

  • MergeJoinRel merge_join = 14

Common fields for all relational operators

Used in: AggregateRel, CrossRel, ExchangeRel, ExtensionLeafRel, ExtensionMultiRel, ExtensionSingleRel, FetchRel, FilterRel, HashJoinRel, JoinRel, MergeJoinRel, ProjectRel, ReadRel, SetRel, SortRel

• oneof emit_kind

  • RelCommon.Direct direct = 1

    The underlying relation is output as is (no reordering or projection of columns)

  • RelCommon.Emit emit = 2

    Allows to control for order and inclusion of fields

• optional RelCommon.Hint hint = 3

• optional extensions.AdvancedExtension advanced_extension = 4

Direct indicates no change on presence and ordering of fields in the output

Used in: RelCommon

(message has no fields)

Remap which fields are output and in which order

Used in: RelCommon

• repeated int32 output_mapping = 1

Changes to the operation that can influence efficiency/performance but should not impact correctness.

Used in: RelCommon

• optional Hint.Stats stats = 1

• optional Hint.RuntimeConstraint constraint = 2

• optional extensions.AdvancedExtension advanced_extension = 10

TODO: nodes, cpu threads/%, memory, iops, etc.

Used in: Hint

• optional extensions.AdvancedExtension advanced_extension = 10

The statistics related to a hint (physical properties of records)

Used in: Hint

• double row_count = 1

• double record_size = 2

• optional extensions.AdvancedExtension advanced_extension = 10

A relation with output field names. This is for use at the root of a `Rel` tree.

Used in: PlanRel

• optional Rel input = 1

  A relation

• repeated string names = 2

  Field names in depth-first order

The relational set operators (intersection/union/etc..)

Used in: Rel

• optional RelCommon common = 1

• repeated Rel inputs = 2

  The first input is the primary input, the remaining are secondary inputs. There must be at least two inputs.

• SetRel.SetOp op = 3

• optional extensions.AdvancedExtension advanced_extension = 10

Used in: SetRel

• SET_OP_UNSPECIFIED = 0

• SET_OP_MINUS_PRIMARY = 1

• SET_OP_MINUS_MULTISET = 2

• SET_OP_INTERSECTION_PRIMARY = 3

• SET_OP_INTERSECTION_MULTISET = 4

• SET_OP_UNION_DISTINCT = 5

• SET_OP_UNION_ALL = 6

The description of a field to sort on (including the direction of sorting and null semantics)

Used in: AggregateFunction, Expression.WindowFunction, SortRel

• optional Expression expr = 1

• oneof sort_kind

  • SortField.SortDirection direction = 2

  • uint32 comparison_function_reference = 3

Used in: SortField

• SORT_DIRECTION_UNSPECIFIED = 0

• SORT_DIRECTION_ASC_NULLS_FIRST = 1

• SORT_DIRECTION_ASC_NULLS_LAST = 2

• SORT_DIRECTION_DESC_NULLS_FIRST = 3

• SORT_DIRECTION_DESC_NULLS_LAST = 4

• SORT_DIRECTION_CLUSTERED = 5

The ORDERY BY (or sorting) relational operator. Beside describing a base relation, it includes a list of fields to sort on

Used in: Rel

• optional RelCommon common = 1

• optional Rel input = 2

• repeated SortField sorts = 3

• optional extensions.AdvancedExtension advanced_extension = 10

Used in: AggregateFunction, Expression.Cast, Expression.EmbeddedFunction, Expression.Literal, Expression.ScalarFunction, Expression.WindowFunction, FunctionArgument, FunctionSignature.Aggregate, Type.List, Type.Map, Type.Parameter, Type.Struct

• oneof kind

  • Type.Boolean bool = 1

  • Type.I8 i8 = 2

  • Type.I16 i16 = 3

  • Type.I32 i32 = 5

  • Type.I64 i64 = 7

  • Type.FP32 fp32 = 10

  • Type.FP64 fp64 = 11

  • Type.String string = 12

  • Type.Binary binary = 13

  • Type.Timestamp timestamp = 14

  • Type.Date date = 16

  • Type.Time time = 17

  • Type.IntervalYear interval_year = 19

  • Type.IntervalDay interval_day = 20

  • Type.TimestampTZ timestamp_tz = 29

  • Type.UUID uuid = 32

  • Type.FixedChar fixed_char = 21

  • Type.VarChar varchar = 22

  • Type.FixedBinary fixed_binary = 23

  • Type.Decimal decimal = 24

  • Type.Struct struct = 25

  • Type.List list = 27

  • Type.Map map = 28

  • Type.UserDefined user_defined = 30

  • uint32 user_defined_type_reference = 31

    Deprecated in favor of user_defined, which allows nullability and variations to be specified. If user_defined_type_reference is encountered, treat it as being non-nullable and having the default variation.

Used in: DerivationExpression, ParameterizedType, Type

• uint32 type_variation_reference = 1

• Nullability nullability = 2

Used in: DerivationExpression, ParameterizedType, Type

• uint32 type_variation_reference = 1

• Nullability nullability = 2

Used in: DerivationExpression, ParameterizedType, Type

• uint32 type_variation_reference = 1

• Nullability nullability = 2

Used in: Type

• int32 scale = 1

• int32 precision = 2

• uint32 type_variation_reference = 3

• Nullability nullability = 4

Used in: DerivationExpression, ParameterizedType, Type

• uint32 type_variation_reference = 1

• Nullability nullability = 2

Used in: DerivationExpression, ParameterizedType, Type

• uint32 type_variation_reference = 1

• Nullability nullability = 2

Used in: Type

• int32 length = 1

• uint32 type_variation_reference = 2

• Nullability nullability = 3

Start compound types.

Used in: Type

• int32 length = 1

• uint32 type_variation_reference = 2

• Nullability nullability = 3

Used in: DerivationExpression, ParameterizedType, Type

• uint32 type_variation_reference = 1

• Nullability nullability = 2

Used in: DerivationExpression, ParameterizedType, Type

• uint32 type_variation_reference = 1

• Nullability nullability = 2

Used in: DerivationExpression, ParameterizedType, Type

• uint32 type_variation_reference = 1

• Nullability nullability = 2

Used in: DerivationExpression, ParameterizedType, Type

• uint32 type_variation_reference = 1

• Nullability nullability = 2

Used in: DerivationExpression, ParameterizedType, Type

• uint32 type_variation_reference = 1

• Nullability nullability = 2

Used in: DerivationExpression, ParameterizedType, Type

• uint32 type_variation_reference = 1

• Nullability nullability = 2

Used in: Expression.Literal, Type

• optional Type type = 1

• uint32 type_variation_reference = 2

• Nullability nullability = 3

Used in: Expression.Literal, Type

• optional Type key = 1

• optional Type value = 2

• uint32 type_variation_reference = 3

• Nullability nullability = 4

Used in: DerivationExpression.ExpressionDecimal, DerivationExpression.ExpressionFixedBinary, DerivationExpression.ExpressionFixedChar, DerivationExpression.ExpressionList, DerivationExpression.ExpressionMap, DerivationExpression.ExpressionStruct, DerivationExpression.ExpressionUserDefined, DerivationExpression.ExpressionVarChar, ParameterizedType.ParameterizedDecimal, ParameterizedType.ParameterizedFixedBinary, ParameterizedType.ParameterizedFixedChar, ParameterizedType.ParameterizedList, ParameterizedType.ParameterizedMap, ParameterizedType.ParameterizedStruct, ParameterizedType.ParameterizedUserDefined, ParameterizedType.ParameterizedVarChar, Binary, Boolean, Date, Decimal, FP32, FP64, FixedBinary, FixedChar, I16, I32, I64, I8, IntervalDay, IntervalYear, List, Map, String, Struct, Time, Timestamp, TimestampTZ, UUID, UserDefined, VarChar

• NULLABILITY_UNSPECIFIED = 0

• NULLABILITY_NULLABLE = 1

• NULLABILITY_REQUIRED = 2

Used in: Expression.Literal.UserDefined, UserDefined

• oneof parameter

  • google.protobuf.Empty null = 1

    Explicitly null/unspecified parameter, to select the default value (if any).

  • Type data_type = 2

    Data type parameters, like the i32 in LIST<i32>.

  • bool boolean = 3

    Value parameters, like the 10 in VARCHAR<10>.

  • int64 integer = 4

  • string enum = 5

  • string string = 6

Used in: DerivationExpression, ParameterizedType, Type

• uint32 type_variation_reference = 1

• Nullability nullability = 2

Used in: NamedStruct, Type

• repeated Type types = 1

• uint32 type_variation_reference = 2

• Nullability nullability = 3

Used in: DerivationExpression, ParameterizedType, Type

• uint32 type_variation_reference = 1

• Nullability nullability = 2

Used in: DerivationExpression, ParameterizedType, Type

• uint32 type_variation_reference = 1

• Nullability nullability = 2

Used in: DerivationExpression, ParameterizedType, Type

• uint32 type_variation_reference = 1

• Nullability nullability = 2

Used in: DerivationExpression, ParameterizedType, Type

• uint32 type_variation_reference = 1

• Nullability nullability = 2

Used in: Type

• uint32 type_reference = 1

• uint32 type_variation_reference = 2

• Nullability nullability = 3

• repeated Parameter type_parameters = 4

Used in: Type

• int32 length = 1

• uint32 type_variation_reference = 2

• Nullability nullability = 3

Used in: ExtendedExpression, Plan, PlanVersion

• uint32 major_number = 1

  Substrait version number.

• uint32 minor_number = 2

• uint32 patch_number = 3

• string git_hash = 4

  If a particular version of Substrait is used that does not correspond to a version number exactly (for example when using an unofficial fork or using a version that is not yet released or is between versions), set this to the full git hash of the utilized commit of https://github.com/substrait-io/substrait (or fork thereof), represented using a lowercase hex ASCII string 40 characters in length. The version number above should be set to the most recent version tag in the history of that commit.

• string producer = 5

  Identifying information for the producer that created this plan. Under ideal circumstances, consumers should not need this information. However, it is foreseen that consumers may need to work around bugs in particular producers in practice, and therefore may need to know which producer created the plan.

The operator that modifies the content of a database (operates on 1 table at a time, but tuple-selection/source can be based on joining of multiple tables).

• oneof write_type

  Definition of which TABLE we are operating on

  • NamedObjectWrite named_table = 1

  • ExtensionObject extension_table = 2

• optional NamedStruct table_schema = 3

  The schema of the table (must align with Rel input (e.g., number of leaf fields must match))

• WriteRel.WriteOp op = 4

  The type of operation to perform

• optional Rel input = 5

  The relation that determines the tuples to add/remove/modify the schema must match with table_schema. Default values must be explicitly stated in a ProjectRel at the top of the input. The match must also occur in case of DELETE to ensure multi-engine plans are unequivocal.

• WriteRel.OutputMode output = 6

  Output mode determines what is the output of executing this rel

Used in: WriteRel

• OUTPUT_MODE_UNSPECIFIED = 0

• OUTPUT_MODE_NO_OUTPUT = 1

  return no tuples at all

• OUTPUT_MODE_MODIFIED_TUPLES = 2

  this mode makes the operator return all the tuple INSERTED/DELETED/UPDATED by the operator. The operator returns the AFTER-image of any change. This can be further manipulated by operators upstreams (e.g., retunring the typical "count of modified tuples"). For scenarios in which the BEFORE image is required, the user must implement a spool (via references to subplans in the body of the Rel input) and return those with anounter PlanRel.relations.

Used in: WriteRel

• WRITE_OP_UNSPECIFIED = 0

• WRITE_OP_INSERT = 1

  The insert of new tuples in a table

• WRITE_OP_DELETE = 2

  The removal of tuples from a table

• WRITE_OP_UPDATE = 3

  The modification of existing tuples within a table

• WRITE_OP_CTAS = 4

  The Creation of a new table, and the insert of new tuples in the table