package spark.spark_expression

Get desktop application:
View/edit binary Protocol Buffers messages

Used in: spark_operator.HashAggregate, spark_operator.WindowExpr

• oneof expr_struct

  • Count count = 2

  • Sum sum = 3

  • Min min = 4

  • Max max = 5

  • Avg avg = 6

  • First first = 7

  • Last last = 8

  • BitAndAgg bitAndAgg = 9

  • BitOrAgg bitOrAgg = 10

  • BitXorAgg bitXorAgg = 11

  • Covariance covariance = 12

  • Variance variance = 13

  • Stddev stddev = 14

  • Correlation correlation = 15

  • BloomFilterAgg bloomFilterAgg = 16

  • CollectSet collectSet = 17

  • Percentile percentile = 18

  • ApproxPercentile approxPercentile = 19

  • HllPlusPlus hllpp = 20

  • CollectList collectList = 21

• optional Expr filter = 89

  Optional filter expression for SQL FILTER (WHERE ...) clause. Only set in Partial aggregation mode; absent in Final/PartialMerge.

• optional QueryContext query_context = 90

  Optional QueryContext for error reporting (contains SQL text and position)

• optional uint64 expr_id = 91

  Unique expression ID for context lookup during error creation

Used in: AggExpr

• optional Expr child = 1

  Child value expression, already cast to Float64 by the serde.

• repeated double percentiles = 2

  The percentiles and accuracy are carried as resolved scalars rather than child Exprs (unlike Percentile/BloomFilterAgg) because they are needed at UDAF construction time to drive return_type and accumulator shape. One or more percentiles in [0.0, 1.0].

• int64 accuracy = 3

  Spark's accuracy argument; relative_error = 1.0 / accuracy.

• bool return_array = 4

  True when the percentile argument was an array (output is a list).

• optional DataType input_type = 5

  Spark's input/output type, used to cast results back from Float64.

Array functions

Used in: Expr

• optional Expr src_array_expr = 1

• optional Expr pos_expr = 2

• optional Expr item_expr = 3

• bool legacy_negative_index = 4

• optional Expr array_expr = 1

• optional Expr delimiter_expr = 2

• optional Expr null_replacement_expr = 3

Spark's ArraysZip takes children: Seq[Expression] and names: Seq[Expression] https://github.com/apache/spark/blob/branch-4.1/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala#L296

Used in: Expr

• repeated Expr values = 1

• repeated string names = 2

Used in: AggExpr

• optional Expr child = 1

• optional DataType datatype = 2

• optional DataType sum_datatype = 3

• EvalMode eval_mode = 4

Used in: Expr

• optional Expr left = 1

• optional Expr right = 2

Used in: ToPrettyString

• UTF8 = 0

• BASIC = 1

• BASE64 = 2

• HEX = 3

• HEX_DISCRETE = 4

Used in: AggExpr

• optional Expr child = 1

• optional DataType datatype = 2

Used in: AggExpr

• optional Expr child = 1

• optional DataType datatype = 2

Used in: AggExpr

• optional Expr child = 1

• optional DataType datatype = 2

Used in: AggExpr

• optional Expr child = 1

• optional Expr numItems = 2

• optional Expr numBits = 3

• optional DataType datatype = 4

• BloomFilterVersion version = 5

  Output serialization version. Spark 4.0 and earlier always wrote V1; Spark 4.1+ defaults to V2 (different bit-scattering algorithm and a `seed` field in the binary format). The JVM serde sets this to the matching version so Comet's aggregate output is byte-equivalent with Spark's.

Used in: Expr

• optional Expr bloom_filter = 1

• optional Expr value = 2

Used in: BloomFilterAgg

• BLOOM_FILTER_VERSION_UNSPECIFIED = 0

• BLOOM_FILTER_VERSION_V1 = 1

• BLOOM_FILTER_VERSION_V2 = 2

Bound to a particular vector array in input batch.

Used in: Expr

• int32 index = 1

• optional DataType datatype = 2

Used in: Expr

• optional Expr expr = 1

  The expr field is added to be consistent with CaseExpr definition in DataFusion. This field is not really used. When constructing a CaseExpr, this expr field is always set to None. The reason that we always set this expr field to None is because Spark parser converts the expr to a EqualTo conditions. After the conversion, we don't see this expr any more so it's always None.

• repeated Expr when = 2

• repeated Expr then = 3

• optional Expr else_expr = 4

Used in: Expr

• optional Expr child = 1

• optional DataType datatype = 2

• string timezone = 3

• EvalMode eval_mode = 4

• bool allow_incompat = 5

• bool is_spark4_plus = 6

  True when running against Spark 4.0+. Controls version-specific cast behaviour such as the handling of leading whitespace before T-prefixed time-only strings.

Used in: Expr

• optional Expr child = 1

• optional DataType datatype = 2

• bool fail_on_error = 3

Used in: AggExpr

• optional Expr child = 1

• optional DataType datatype = 2

Used in: AggExpr

• optional Expr child = 1

• optional DataType datatype = 2

Used in: AggExpr

• optional Expr child1 = 1

• optional Expr child2 = 2

• bool null_on_divide_by_zero = 3

• optional DataType datatype = 4

Used in: AggExpr

• repeated Expr children = 1

Used in: AggExpr

• optional Expr child1 = 1

• optional Expr child2 = 2

• bool null_on_divide_by_zero = 3

• optional DataType datatype = 4

• StatisticsType stats_type = 5

Used in: Expr

• repeated Expr values = 1

• repeated string names = 2

Used in: ToCsv

• string delimiter = 1

• string quote = 2

• string escape = 3

• string null_value = 4

• bool quote_all = 5

• bool ignore_leading_white_space = 6

• bool ignore_trailing_white_space = 7

• string timezone = 8

Used in: ApproxPercentile, Avg, BitAndAgg, BitOrAgg, BitXorAgg, BloomFilterAgg, BoundReference, Cast, CheckOverflow, CollectList, CollectSet, Correlation, Covariance, DataType.ListInfo, DataType.MapInfo, DataType.StructInfo, First, FromJson, JvmScalarUdf, Last, Literal, MathExpr, Max, Min, NormalizeNaNAndZero, Percentile, PreciseTimestampConversion, ScalarFunc, Stddev, Subquery, Sum, UnboundReference, Variance, spark_operator.NativeScanCommon, spark_operator.Scan, spark_operator.ShuffleScan, spark_operator.SparkStructField, spark_operator.WindowExpr

• DataType.DataTypeId type_id = 1

• optional DataType.DataTypeInfo type_info = 2

Used in: DataType

• BOOL = 0

• INT8 = 1

• INT16 = 2

• INT32 = 3

• INT64 = 4

• FLOAT = 5

• DOUBLE = 6

• STRING = 7

• BYTES = 8

• TIMESTAMP = 9

• DECIMAL = 10

• TIMESTAMP_NTZ = 11

• DATE = 12

• NULL = 13

• LIST = 14

• MAP = 15

• STRUCT = 16

• TIME = 17

• YEAR_MONTH_INTERVAL = 18

• DAY_TIME_INTERVAL = 19

• CALENDAR_INTERVAL = 20

Used in: DataType

• oneof datatype_struct

  • DecimalInfo decimal = 2

  • ListInfo list = 3

  • MapInfo map = 4

  • StructInfo struct = 5

Used in: DataTypeInfo

• int32 precision = 1

• int32 scale = 2

Used in: StructInfo

• map<string, string> metadata = 1

Used in: DataTypeInfo

• optional DataType element_type = 1

• bool contains_null = 2

• optional int32 element_field_id = 3

  Parquet field ID for the synthetic list element, when provided by Catalyst metadata.

Used in: DataTypeInfo

• optional DataType key_type = 1

• optional DataType value_type = 2

• bool value_contains_null = 3

• optional int32 key_field_id = 4

  Parquet field IDs for the synthetic map fields, when provided by Catalyst metadata.

• optional int32 value_field_id = 5

Used in: DataTypeInfo

• repeated string field_names = 1

• repeated DataType field_datatypes = 2

• repeated bool field_nullable = 3

• repeated FieldMetadata field_metadata = 4

  Parallel to field_names. Each entry holds Spark field metadata for the corresponding nested field. Currently used to carry parquet.field.id through to the native side. Empty when no fields carry metadata.

Used in: Expr

(message has no fields)

Used in: Avg, Cast, MathExpr, Sum

• LEGACY = 0

• TRY = 1

• ANSI = 2

The basic message representing a Spark expression.

Used in: AggExpr, ApproxPercentile, ArrayInsert, ArrayJoin, ArraysZip, Avg, BinaryExpr, BitAndAgg, BitOrAgg, BitXorAgg, BloomFilterAgg, BloomFilterMightContain, CaseWhen, Cast, CheckOverflow, CollectList, CollectSet, Correlation, Count, Covariance, CreateNamedStruct, First, FromJson, GetArrayStructFields, GetStructField, HllPlusPlus, Hour, HoursTransform, IfExpr, In, JvmScalarUdf, Last, ListExtract, MathExpr, Max, Min, Minute, NormalizeNaNAndZero, Percentile, PreciseTimestampConversion, ScalarFunc, Second, Shuffle, SortOrder, Stddev, Sum, ToCsv, ToJson, ToPrettyString, TruncTimestamp, UnaryExpr, UnaryMinus, UnixTimestamp, Variance, spark_operator.BroadcastNestedLoopJoin, spark_operator.Expand, spark_operator.Explode, spark_operator.Filter, spark_operator.HashAggregate, spark_operator.HashJoin, spark_operator.NativeScanCommon, spark_operator.Projection, spark_operator.Sort, spark_operator.SortMergeJoin, spark_operator.SparkPartitionedFile, spark_operator.Window, spark_operator.WindowExpr, spark_operator.WindowSpecDefinition, spark_partitioning.BoundaryRow, spark_partitioning.HashPartition, spark_partitioning.RangePartition

• oneof expr_struct

  • Literal literal = 2

  • BoundReference bound = 3

  • MathExpr add = 4

  • MathExpr subtract = 5

  • MathExpr multiply = 6

  • MathExpr divide = 7

  • Cast cast = 8

  • BinaryExpr eq = 9

  • BinaryExpr neq = 10

  • BinaryExpr gt = 11

  • BinaryExpr gt_eq = 12

  • BinaryExpr lt = 13

  • BinaryExpr lt_eq = 14

  • UnaryExpr is_null = 15

  • UnaryExpr is_not_null = 16

  • BinaryExpr and = 17

  • BinaryExpr or = 18

  • SortOrder sort_order = 19

  • Hour hour = 22

  • Minute minute = 23

  • Second second = 24

  • CheckOverflow check_overflow = 25

  • BinaryExpr like = 26

  • BinaryExpr rlike = 30

  • ScalarFunc scalarFunc = 31

  • BinaryExpr eqNullSafe = 32

  • BinaryExpr neqNullSafe = 33

  • BinaryExpr bitwiseAnd = 34

  • BinaryExpr bitwiseOr = 35

  • BinaryExpr bitwiseXor = 36

  • MathExpr remainder = 37

  • CaseWhen caseWhen = 38

  • In in = 39

  • UnaryExpr not = 40

  • UnaryMinus unary_minus = 41

  • BinaryExpr bitwiseShiftRight = 42

  • BinaryExpr bitwiseShiftLeft = 43

  • IfExpr if = 44

  • NormalizeNaNAndZero normalize_nan_and_zero = 45

  • TruncTimestamp truncTimestamp = 47

  • Subquery subquery = 50

  • UnboundReference unbound = 51

  • BloomFilterMightContain bloom_filter_might_contain = 52

  • CreateNamedStruct create_named_struct = 53

  • GetStructField get_struct_field = 54

  • ToJson to_json = 55

  • ListExtract list_extract = 56

  • GetArrayStructFields get_array_struct_fields = 57

  • ArrayInsert array_insert = 58

  • MathExpr integral_divide = 59

  • ToPrettyString to_pretty_string = 60

  • Rand rand = 61

  • Rand randn = 62

  • EmptyExpr spark_partition_id = 63

  • EmptyExpr monotonically_increasing_id = 64

  • UnixTimestamp unix_timestamp = 65

  • FromJson from_json = 66

  • ToCsv to_csv = 67

  • HoursTransform hours_transform = 68

  • ArraysZip arrays_zip = 69

  • JvmScalarUdf jvm_scalar_udf = 70

  • PreciseTimestampConversion precise_timestamp_conversion = 71

  • Shuffle shuffle = 72

  • RandStr rand_str = 73

  • Uuid uuid = 74

• optional QueryContext query_context = 90

  Optional QueryContext for error reporting (contains SQL text and position)

• optional uint64 expr_id = 91

  Unique expression ID for context lookup during error creation

Used in: AggExpr

• optional Expr child = 1

• optional DataType datatype = 2

• bool ignore_nulls = 3

Used in: Expr

• optional Expr child = 1

• optional DataType schema = 2

• string timezone = 3

Used in: Expr

• optional Expr child = 1

• int32 ordinal = 2

Used in: Expr

• optional Expr child = 1

• int32 ordinal = 2

approx_count_distinct (Spark's HyperLogLogPlusPlus)

Used in: AggExpr

• optional Expr child = 1

• int32 precision = 2

  Number of addressing bits, computed from relativeSD to match Spark.

Used in: Expr

• optional Expr child = 1

• string timezone = 2

Used in: Expr

• optional Expr child = 1

Used in: Expr

• optional Expr if_expr = 1

• optional Expr true_expr = 2

• optional Expr false_expr = 3

Used in: Expr

• optional Expr in_value = 1

• repeated Expr lists = 2

• bool negated = 3

Scalar UDF dispatched to the JVM via JNI. Native side exports input arrays through Arrow C Data Interface, calls CometUdfBridge.evaluate, and imports the result.

Used in: Expr

• string class_name = 1

  Fully-qualified Java/Scala class name implementing org.apache.comet.udf.CometUDF (must have a public no-arg constructor).

• repeated Expr args = 2

  Argument expressions, evaluated by the native side before invocation.

• optional DataType return_type = 3

  Expected return type. Used to import the result FFI_ArrowArray.

• bool return_nullable = 4

  Whether the result column may contain nulls.

Used in: AggExpr

• optional Expr child = 1

• optional DataType datatype = 2

• bool ignore_nulls = 3

Used in: Expr

• optional Expr child = 1

• optional Expr ordinal = 2

• optional Expr default_value = 3

• bool one_based = 4

• bool fail_on_error = 5

Used in: Literal

• repeated bool boolean_values = 1

  Only one of these fields should be populated based on the array type

• repeated int32 byte_values = 2

• repeated int32 short_values = 3

• repeated int32 int_values = 4

• repeated int64 long_values = 5

• repeated float float_values = 6

• repeated double double_values = 7

• repeated string string_values = 8

• repeated bytes bytes_values = 9

• repeated bytes decimal_values = 10

• repeated ListLiteral list_values = 11

• repeated bool null_mask = 12

Used in: Expr, spark_operator.Following, spark_operator.Preceding

• oneof value

  • bool bool_val = 1

  • int32 byte_val = 2

    Protobuf doesn't provide int8 and int16, we put them into int32 and convert to int8 and int16 when deserializing.

  • int32 short_val = 3

  • int32 int_val = 4

  • int64 long_val = 5

  • float float_val = 6

  • double double_val = 7

  • string string_val = 8

  • bytes bytes_val = 9

  • bytes decimal_val = 10

  • ListLiteral list_val = 11

• optional DataType datatype = 12

• bool is_null = 13

Used in: Expr

• optional Expr left = 1

• optional Expr right = 2

• optional DataType return_type = 4

• EvalMode eval_mode = 5

• bool check_divide_overflow = 6

  Only set for integral_divide: mirrors Spark's IntegralDivide.checkDivideOverflow (LONG operands with failOnError). When this is set and eval_mode is ANSI, a quotient that does not fit in a LONG (Long.MinValue div -1) throws ARITHMETIC_OVERFLOW instead of wrapping around. Never set for DECIMAL operands, which wrap around on the cast to LONG even in ANSI mode, matching Spark.

Used in: AggExpr

• optional Expr child = 1

• optional DataType datatype = 2

Used in: AggExpr

• optional Expr child = 1

• optional DataType datatype = 2

Used in: Expr

• optional Expr child = 1

• string timezone = 2

Used in: Expr

• optional Expr child = 1

• optional DataType datatype = 2

Used in: SortOrder

• NullsFirst = 0

• NullsLast = 1

Used in: AggExpr

• optional Expr child = 1

• optional Expr percentage = 2

  Single percentile in [0.0, 1.0] as a literal double expression.

• optional DataType datatype = 3

Spark's internal PreciseTimestampConversion, used by time-window grouping to convert between TimestampType/TimestampNTZType and LongType without losing microsecond precision. This is a pure reinterpret: the underlying value is unchanged, only the type is changed.

Used in: Expr

• optional Expr child = 1

• optional DataType datatype = 2

QueryContext provides SQL query context for error messages. Mirrors Spark's SQLQueryContext for rich error reporting.

Used in: AggExpr, Expr

• string sql_text = 1

  Full SQL query text. Left empty when `sql_text_idx` is set, which is the case for a plan serialized through `CometNativeExec.convertBlock`. Still used by the paths that serialize an operator directly without interning (e.g. `CometNativeWriteExec`, the native shuffle writer, the Iceberg scan's `convertBlock` override) and by the bare-`Expr` Parquet filter path, which has no root operator to carry a pool.

• int32 start_index = 2

  Character offset where expression starts (0-based)

• int32 stop_index = 3

  Character offset where expression ends (0-based, inclusive)

• optional string object_type = 4

  Type of SQL object (e.g., "VIEW", "Project", "Filter")

• optional string object_name = 5

  Name of object (e.g., view name, column name)

• int32 line = 6

  Line number in SQL query (1-based)

• int32 start_position = 7

  Column position within the line (0-based)

• optional int32 sql_text_idx = 8

  Index into the root `Operator.sql_text_pool` holding this context's SQL text. Set instead of `sql_text` so that a query text shared by many expressions is serialized once per plan rather than once per expression.

Used in: Expr

• int64 seed = 1

Spark's RandStr (Spark 4.0+) returns a random alphanumeric string of the given length. It is non-deterministic: the resolved random seed is combined with the partition index and seeds an XORShiftRandom, matching org.apache.spark.sql.catalyst.expressions.ExpressionImplUtils.randStr.

Used in: Expr

• int32 length = 1

• int64 seed = 2

Used in: Expr

• string func = 1

• repeated Expr args = 2

• optional DataType return_type = 3

• bool fail_on_error = 4

Used in: Expr

• optional Expr child = 1

• string timezone = 2

Spark's Shuffle returns a random permutation of the given array. It is non-deterministic: the resolved random seed is combined with the partition index and drives a MersenneTwister-based inside-out Fisher-Yates shuffle, matching org.apache.spark.sql.catalyst.util.RandomIndicesGenerator.

Used in: Expr

• optional Expr child = 1

• int64 seed = 2

Used in: SortOrder

• Ascending = 0

• Descending = 1

Used in: Expr

• optional Expr child = 1

• SortDirection direction = 2

• NullOrdering null_ordering = 3

Used in: Covariance, Stddev, Variance

• SAMPLE = 0

• POPULATION = 1

Used in: AggExpr

• optional Expr child = 1

• bool null_on_divide_by_zero = 2

• optional DataType datatype = 3

• StatisticsType stats_type = 4

Used in: Expr

• int64 id = 1

• optional DataType datatype = 2

Used in: AggExpr

• optional Expr child = 1

• optional DataType datatype = 2

• EvalMode eval_mode = 3

Used in: Expr

• optional Expr child = 1

• optional CsvWriteOptions options = 2

Used in: Expr

• optional Expr child = 1

• string timezone = 2

• string date_format = 3

• string timestamp_format = 4

• string timestamp_ntz_format = 5

• bool ignore_null_fields = 6

Used in: Expr

• optional Expr child = 1

• string timezone = 2

• BinaryOutputStyle binaryOutputStyle = 3

Used in: Expr

• optional Expr format = 1

• optional Expr child = 2

• string timezone = 3

Used in: Expr

• optional Expr child = 1

Used in: Expr

• optional Expr child = 1

• bool fail_on_error = 2

Used in: Expr

• string name = 1

• optional DataType datatype = 2

Used in: Expr

• optional Expr child = 1

• string timezone = 2

Spark's Uuid returns a random RFC 4122 version 4 UUID string. It is non-deterministic: the resolved random seed is combined with the partition index and seeds a Commons Math3 MersenneTwister, matching org.apache.spark.sql.catalyst.util.RandomUUIDGenerator.

Used in: Expr

• int64 seed = 1

Used in: AggExpr

• optional Expr child = 1

• bool null_on_divide_by_zero = 2

• optional DataType datatype = 3

• StatisticsType stats_type = 4