package spark.spark_expression

Get desktop application:
View/edit binary Protocol Buffers messages

Used in: Expr

• optional Expr child = 1

• EvalMode eval_mode = 2

Used in: spark_operator.HashAggregate, spark_operator.WindowExpr

• oneof expr_struct

  • Count count = 2

  • Sum sum = 3

  • Min min = 4

  • Max max = 5

  • Avg avg = 6

  • First first = 7

  • Last last = 8

  • BitAndAgg bitAndAgg = 9

  • BitOrAgg bitOrAgg = 10

  • BitXorAgg bitXorAgg = 11

  • Covariance covariance = 12

  • Variance variance = 13

  • Stddev stddev = 14

  • Correlation correlation = 15

  • BloomFilterAgg bloomFilterAgg = 16

Array functions

Used in: Expr

• optional Expr src_array_expr = 1

• optional Expr pos_expr = 2

• optional Expr item_expr = 3

• bool legacy_negative_index = 4

• optional Expr array_expr = 1

• optional Expr delimiter_expr = 2

• optional Expr null_replacement_expr = 3

Used in: AggExpr

• optional Expr child = 1

• optional DataType datatype = 2

• optional DataType sum_datatype = 3

• bool fail_on_error = 4

  currently unused (useful for deciding Ansi vs Legacy mode)

Used in: Expr

• optional Expr left = 1

• optional Expr right = 2

Used in: AggExpr

• optional Expr child = 1

• optional DataType datatype = 2

Used in: AggExpr

• optional Expr child = 1

• optional DataType datatype = 2

Used in: AggExpr

• optional Expr child = 1

• optional DataType datatype = 2

Used in: AggExpr

• optional Expr child = 1

• optional Expr numItems = 2

• optional Expr numBits = 3

• optional DataType datatype = 4

Used in: Expr

• optional Expr bloom_filter = 1

• optional Expr value = 2

Bound to a particular vector array in input batch.

Used in: Expr

• int32 index = 1

• optional DataType datatype = 2

Used in: Expr

• optional Expr expr = 1

  The expr field is added to be consistent with CaseExpr definition in DataFusion. This field is not really used. When constructing a CaseExpr, this expr field is always set to None. The reason that we always set this expr field to None is because Spark parser converts the expr to a EqualTo conditions. After the conversion, we don't see this expr any more so it's always None.

• repeated Expr when = 2

• repeated Expr then = 3

• optional Expr else_expr = 4

Used in: Expr

• optional Expr child = 1

• optional DataType datatype = 2

• string timezone = 3

• EvalMode eval_mode = 4

• bool allow_incompat = 5

Used in: Expr

• optional Expr child = 1

• optional DataType datatype = 2

• bool fail_on_error = 3

Used in: AggExpr

• optional Expr child1 = 1

• optional Expr child2 = 2

• bool null_on_divide_by_zero = 3

• optional DataType datatype = 4

Used in: AggExpr

• repeated Expr children = 1

Used in: AggExpr

• optional Expr child1 = 1

• optional Expr child2 = 2

• bool null_on_divide_by_zero = 3

• optional DataType datatype = 4

• StatisticsType stats_type = 5

Used in: Expr

• repeated Expr values = 1

• repeated string names = 2

Used in: Avg, BitAndAgg, BitOrAgg, BitXorAgg, BloomFilterAgg, BoundReference, Cast, CheckOverflow, Correlation, Covariance, DataType.ListInfo, DataType.MapInfo, DataType.StructInfo, First, Last, Literal, MathExpr, Max, Min, NormalizeNaNAndZero, ScalarFunc, Stddev, Subquery, Sum, UnboundReference, Variance, spark_operator.NativeScan, spark_operator.Scan, spark_operator.SparkStructField

• DataType.DataTypeId type_id = 1

• optional DataType.DataTypeInfo type_info = 2

Used in: DataType

• BOOL = 0

• INT8 = 1

• INT16 = 2

• INT32 = 3

• INT64 = 4

• FLOAT = 5

• DOUBLE = 6

• STRING = 7

• BYTES = 8

• TIMESTAMP = 9

• DECIMAL = 10

• TIMESTAMP_NTZ = 11

• DATE = 12

• NULL = 13

• LIST = 14

• MAP = 15

• STRUCT = 16

Used in: DataType

• oneof datatype_struct

  • DecimalInfo decimal = 2

  • ListInfo list = 3

  • MapInfo map = 4

  • StructInfo struct = 5

Used in: DataTypeInfo

• int32 precision = 1

• int32 scale = 2

Used in: DataTypeInfo

• optional DataType element_type = 1

• bool contains_null = 2

Used in: DataTypeInfo

• optional DataType key_type = 1

• optional DataType value_type = 2

• bool value_contains_null = 3

Used in: DataTypeInfo

• repeated string field_names = 1

• repeated DataType field_datatypes = 2

• repeated bool field_nullable = 3

Used in: Abs, Cast

• LEGACY = 0

• TRY = 1

• ANSI = 2

The basic message representing a Spark expression.

Used in: Abs, ArrayInsert, ArrayJoin, Avg, BinaryExpr, BitAndAgg, BitOrAgg, BitXorAgg, BloomFilterAgg, BloomFilterMightContain, CaseWhen, Cast, CheckOverflow, Correlation, Count, Covariance, CreateNamedStruct, First, GetArrayStructFields, GetStructField, Hour, IfExpr, In, Last, ListExtract, MathExpr, Max, Min, Minute, NormalizeNaNAndZero, ScalarFunc, Second, SortOrder, Stddev, Substring, Sum, ToJson, TruncDate, TruncTimestamp, UnaryExpr, UnaryMinus, Variance, spark_operator.Expand, spark_operator.Filter, spark_operator.HashAggregate, spark_operator.HashJoin, spark_operator.NativeScan, spark_operator.Projection, spark_operator.Sort, spark_operator.SortMergeJoin, spark_operator.SparkPartitionedFile, spark_operator.Window, spark_operator.WindowExpr, spark_operator.WindowSpecDefinition, spark_partitioning.HashRepartition

• oneof expr_struct

  • Literal literal = 2

  • BoundReference bound = 3

  • MathExpr add = 4

  • MathExpr subtract = 5

  • MathExpr multiply = 6

  • MathExpr divide = 7

  • Cast cast = 8

  • BinaryExpr eq = 9

  • BinaryExpr neq = 10

  • BinaryExpr gt = 11

  • BinaryExpr gt_eq = 12

  • BinaryExpr lt = 13

  • BinaryExpr lt_eq = 14

  • UnaryExpr is_null = 15

  • UnaryExpr is_not_null = 16

  • BinaryExpr and = 17

  • BinaryExpr or = 18

  • SortOrder sort_order = 19

  • Substring substring = 20

  • UnaryExpr string_space = 21

  • Hour hour = 22

  • Minute minute = 23

  • Second second = 24

  • CheckOverflow check_overflow = 25

  • BinaryExpr like = 26

  • BinaryExpr startsWith = 27

  • BinaryExpr endsWith = 28

  • BinaryExpr contains = 29

  • BinaryExpr rlike = 30

  • ScalarFunc scalarFunc = 31

  • BinaryExpr eqNullSafe = 32

  • BinaryExpr neqNullSafe = 33

  • BinaryExpr bitwiseAnd = 34

  • BinaryExpr bitwiseOr = 35

  • BinaryExpr bitwiseXor = 36

  • MathExpr remainder = 37

  • CaseWhen caseWhen = 38

  • In in = 39

  • UnaryExpr not = 40

  • UnaryMinus unary_minus = 41

  • BinaryExpr bitwiseShiftRight = 42

  • BinaryExpr bitwiseShiftLeft = 43

  • IfExpr if = 44

  • NormalizeNaNAndZero normalize_nan_and_zero = 45

  • TruncDate truncDate = 46

  • TruncTimestamp truncTimestamp = 47

  • UnaryExpr bitwiseNot = 48

  • Abs abs = 49

  • Subquery subquery = 50

  • UnboundReference unbound = 51

  • BloomFilterMightContain bloom_filter_might_contain = 52

  • CreateNamedStruct create_named_struct = 53

  • GetStructField get_struct_field = 54

  • ToJson to_json = 55

  • ListExtract list_extract = 56

  • GetArrayStructFields get_array_struct_fields = 57

  • ArrayInsert array_insert = 58

  • MathExpr integral_divide = 59

Used in: AggExpr

• optional Expr child = 1

• optional DataType datatype = 2

• bool ignore_nulls = 3

Used in: Expr

• optional Expr child = 1

• int32 ordinal = 2

Used in: Expr

• optional Expr child = 1

• int32 ordinal = 2

Used in: Expr

• optional Expr child = 1

• string timezone = 2

Used in: Expr

• optional Expr if_expr = 1

• optional Expr true_expr = 2

• optional Expr false_expr = 3

Used in: Expr

• optional Expr in_value = 1

• repeated Expr lists = 2

• bool negated = 3

Used in: AggExpr

• optional Expr child = 1

• optional DataType datatype = 2

• bool ignore_nulls = 3

Used in: Expr

• optional Expr child = 1

• optional Expr ordinal = 2

• optional Expr default_value = 3

• bool one_based = 4

• bool fail_on_error = 5

Used in: Expr

• oneof value

  • bool bool_val = 1

  • int32 byte_val = 2

    Protobuf doesn't provide int8 and int16, we put them into int32 and convert to int8 and int16 when deserializing.

  • int32 short_val = 3

  • int32 int_val = 4

  • int64 long_val = 5

  • float float_val = 6

  • double double_val = 7

  • string string_val = 8

  • bytes bytes_val = 9

  • bytes decimal_val = 10

• optional DataType datatype = 11

• bool is_null = 12

Used in: Expr

• optional Expr left = 1

• optional Expr right = 2

• bool fail_on_error = 3

• optional DataType return_type = 4

Used in: AggExpr

• optional Expr child = 1

• optional DataType datatype = 2

Used in: AggExpr

• optional Expr child = 1

• optional DataType datatype = 2

Used in: Expr

• optional Expr child = 1

• string timezone = 2

Used in: Expr

• optional Expr child = 1

• optional DataType datatype = 2

Used in: SortOrder

• NullsFirst = 0

• NullsLast = 1

Used in: Expr

• string func = 1

• repeated Expr args = 2

• optional DataType return_type = 3

Used in: Expr

• optional Expr child = 1

• string timezone = 2

Used in: SortOrder

• Ascending = 0

• Descending = 1

Used in: Expr

• optional Expr child = 1

• SortDirection direction = 2

• NullOrdering null_ordering = 3

Used in: Covariance, Stddev, Variance

• SAMPLE = 0

• POPULATION = 1

Used in: AggExpr

• optional Expr child = 1

• bool null_on_divide_by_zero = 2

• optional DataType datatype = 3

• StatisticsType stats_type = 4

Used in: Expr

• int64 id = 1

• optional DataType datatype = 2

Used in: Expr

• optional Expr child = 1

• int32 start = 2

• int32 len = 3

Used in: AggExpr

• optional Expr child = 1

• optional DataType datatype = 2

• bool fail_on_error = 3

Used in: Expr

• optional Expr child = 1

• string timezone = 2

• string date_format = 3

• string timestamp_format = 4

• string timestamp_ntz_format = 5

• bool ignore_null_fields = 6

Used in: Expr

• optional Expr child = 1

• optional Expr format = 2

Used in: Expr

• optional Expr format = 1

• optional Expr child = 2

• string timezone = 3

Used in: Expr

• optional Expr child = 1

Used in: Expr

• optional Expr child = 1

• bool fail_on_error = 2

Used in: Expr

• string name = 1

• optional DataType datatype = 2

Used in: AggExpr

• optional Expr child = 1

• bool null_on_divide_by_zero = 2

• optional DataType datatype = 3

• StatisticsType stats_type = 4