package spark.spark_expression

Mouse Melon logoGet desktop application:
View/edit binary Protocol Buffers messages

message AggExpr

expr.proto:147

Used in: spark_operator.HashAggregate, spark_operator.WindowExpr

message ApproxPercentile

expr.proto:328

Used in: AggExpr

message ArrayInsert

expr.proto:700

Array functions

Used in: Expr

message ArrayJoin

expr.proto:707

message ArraysZip

expr.proto:745

Spark's ArraysZip takes children: Seq[Expression] and names: Seq[Expression] https://github.com/apache/spark/blob/branch-4.1/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala#L296

Used in: Expr

message AtLeastNNonNulls

expr.proto:813

Counts non-null and non-NaN children, with per-row short-circuit evaluation.

Used in: Expr

message Avg

expr.proto:215

Used in: AggExpr

message BinaryExpr

expr.proto:494

Used in: Expr

enum BinaryOutputStyle

expr.proto:554

Used in: ToPrettyString

message BitAndAgg

expr.proto:234

Used in: AggExpr

message BitOrAgg

expr.proto:239

Used in: AggExpr

message BitXorAgg

expr.proto:244

Used in: AggExpr

message BloomFilterAgg

expr.proto:344

Used in: AggExpr

message BloomFilterMightContain

expr.proto:659

Used in: Expr

enum BloomFilterVersion

expr.proto:356

Used in: BloomFilterAgg

message BoundReference

expr.proto:507

Bound to a particular vector array in input batch.

Used in: Expr

message CaseWhen

expr.proto:609

Used in: Expr

message Cast

expr.proto:464

Used in: Expr

message CheckOverflow

expr.proto:592

Used in: Expr

message CollectList

expr.proto:381

Used in: AggExpr

message CollectSet

expr.proto:376

Used in: AggExpr

message Correlation

expr.proto:271

Used in: AggExpr

message Count

expr.proto:195

Used in: AggExpr

message Covariance

expr.proto:249

Used in: AggExpr

message CreateNamedStruct

expr.proto:664

Used in: Expr

message CsvWriteOptions

expr.proto:543

Used in: ToCsv

message DataType

types.proto:43

Used in: ApproxPercentile, Avg, BitAndAgg, BitOrAgg, BitXorAgg, BloomFilterAgg, BoundReference, Cast, CheckOverflow, CollectList, CollectSet, Correlation, Covariance, DataType.ListInfo, DataType.MapInfo, DataType.StructInfo, First, FromJson, JvmScalarUdf, Last, Literal, MathExpr, Max, Min, Mode, NativeScalarUdf, NormalizeNaNAndZero, Percentile, PivotFirst, PreciseTimestampConversion, Regr, ScalarFunc, Stddev, Subquery, Sum, UnboundReference, Variance, spark_operator.ArrowPythonFunction, spark_operator.MergeRows, spark_operator.NativeScanCommon, spark_operator.Scan, spark_operator.ShuffleScan, spark_operator.SparkStructField, spark_operator.WindowExpr

enum DataType.DataTypeId

types.proto:44

Used in: DataType

message DataType.DataTypeInfo

types.proto:70

Used in: DataType

message DataType.DecimalInfo

types.proto:79

Used in: DataTypeInfo

message DataType.FieldMetadata

types.proto:110

Used in: StructInfo

message DataType.ListInfo

types.proto:84

Used in: DataTypeInfo

message DataType.MapInfo

types.proto:91

Used in: DataTypeInfo

message DataType.StructInfo

types.proto:100

Used in: DataTypeInfo

message EmptyExpr

expr.proto:503

Used in: Expr

(message has no fields)

enum EvalMode

expr.proto:445

Used in: Avg, Cast, Correlation, MathExpr, Regr, Sum

message Expr

expr.proto:30

The basic message representing a Spark expression.

Used in: AggExpr, ApproxPercentile, ArrayInsert, ArrayJoin, ArraysZip, AtLeastNNonNulls, Avg, BinaryExpr, BitAndAgg, BitOrAgg, BitXorAgg, BloomFilterAgg, BloomFilterMightContain, CaseWhen, Cast, CheckOverflow, CollectList, CollectSet, Correlation, Count, Covariance, CreateNamedStruct, First, FromJson, GetArrayStructFields, GetStructField, HigherOrderFunc, HllPlusPlus, HllSketchAgg, HllUnionAgg, Hour, HoursTransform, IfExpr, In, JvmScalarUdf, Last, ListAgg, ListExtract, MathExpr, Max, MaxBy, Min, MinBy, Minute, Mode, NativeScalarUdf, NormalizeNaNAndZero, Percentile, PivotFirst, PreciseTimestampConversion, Regr, ScalarFunc, Second, Shuffle, SortOrder, Stddev, Sum, ToCsv, ToJson, ToPrettyString, TruncTimestamp, UnaryExpr, UnaryMinus, UnixTimestamp, Variance, spark_operator.ArrowPythonFunction, spark_operator.BroadcastNestedLoopJoin, spark_operator.Expand, spark_operator.Explode, spark_operator.Filter, spark_operator.HashAggregate, spark_operator.HashJoin, spark_operator.MergeInstruction, spark_operator.MergeOutputRow, spark_operator.MergeRows, spark_operator.NativeScanCommon, spark_operator.Projection, spark_operator.Sort, spark_operator.SortMergeJoin, spark_operator.SparkPartitionedFile, spark_operator.Window, spark_operator.WindowExpr, spark_operator.WindowGroupLimit, spark_operator.WindowSpecDefinition, spark_partitioning.BoundaryRow, spark_partitioning.HashPartition, spark_partitioning.RangePartition

message First

expr.proto:222

Used in: AggExpr

message FromJson

expr.proto:532

Used in: Expr

message GetArrayStructFields

expr.proto:684

Used in: Expr

message GetStructField

expr.proto:669

Used in: Expr

message HigherOrderFunc

expr.proto:755

A DataFusion higher-order function applied to one array argument and a one-parameter lambda, e.g. `array_filter(array, x -> body)`. The lambda body is an ordinary expression tree in which `LambdaVariable` nodes refer to the lambda parameter and `BoundReference` nodes refer to columns of the enclosing operator's input.

Used in: Expr

message HllPlusPlus

expr.proto:387

approx_count_distinct (Spark's HyperLogLogPlusPlus)

Used in: AggExpr

message HllSketchAgg

expr.proto:362

Used in: AggExpr

message HllUnionAgg

expr.proto:369

Used in: AggExpr

message Hour

expr.proto:568

Used in: Expr

message HoursTransform

expr.proto:573

Used in: Expr

message IfExpr

expr.proto:637

Used in: Expr

message In

expr.proto:621

Used in: Expr

message JvmScalarUdf

expr.proto:774

Scalar UDF dispatched to the JVM via JNI. Native side exports input arrays through Arrow C Data Interface, calls CometUdfBridge.evaluate, and imports the result.

Used in: Expr

message LambdaVariable

expr.proto:767

Reference to a lambda parameter from inside a `HigherOrderFunc` body. The element type is taken from the array argument at planning time.

Used in: Expr

message Last

expr.proto:228

Used in: AggExpr

message ListAgg

expr.proto:399

Spark 4.0+ LISTAGG / STRING_AGG aggregate. Comet only serializes the simple form: a StringType child with a literal (or NULL) delimiter and no WITHIN GROUP ORDER BY. DISTINCT falls back to Spark because Comet rejects multi-column distinct aggregates in aggExprToProto, so the native side never sees it.

Used in: AggExpr

message ListExtract

expr.proto:674

Used in: Expr

message ListLiteral

types.proto:26

Used in: Literal

message Literal

literal.proto:28

Used in: Expr, spark_operator.Following, spark_operator.Preceding

message MathExpr

expr.proto:451

Used in: Expr

message Max

expr.proto:210

Used in: AggExpr

message MaxBy

expr.proto:407

Used in: AggExpr

message Min

expr.proto:205

Used in: AggExpr

message MinBy

expr.proto:414

Used in: AggExpr

message Minute

expr.proto:577

Used in: Expr

message Mode

expr.proto:421

Used in: AggExpr

message NativeScalarUdf

expr.proto:793

Call to a user-supplied native UDF loaded from a shared library. The native side resolves (library_path, name) against its loaded-library cache, looks up the kernel by name, and invokes it through the Comet UDF C ABI. That ABI is parameterized only by the Arrow C Data Interface, so the message says nothing about the language the library was written in; the Rust SDK is the only supported way to produce one today.

Used in: Expr

message NormalizeNaNAndZero

expr.proto:627

Used in: Expr

enum NullOrdering

expr.proto:694

Used in: SortOrder

message Percentile

expr.proto:313

Used in: AggExpr

message PivotFirst

expr.proto:438

Optimized Pivot second-phase aggregate. Given a `pivot_column` and a pre-computed list of `pivot_values`, buckets the incoming `value_column` into an array whose length is `pivot_values.size()`. Rows whose pivot value is not in `pivot_values` are ignored; if multiple rows in the same group map to the same bucket, the last non-null value wins (matches Spark's `PivotFirst.update`/`merge`). Output type is ArrayType(value_datatype). The pivot_values are serialized as Literal Exprs so the value bytes and data type reach native code faithfully.

Used in: AggExpr

message PreciseTimestampConversion

expr.proto:489

Spark's internal PreciseTimestampConversion, used by time-window grouping to convert between TimestampType/TimestampNTZType and LongType without losing microsecond precision. This is a pure reinterpret: the underlying value is unchanged, only the type is changed.

Used in: Expr

message QueryContext

expr.proto:115

QueryContext provides SQL query context for error messages. Mirrors Spark's SQLQueryContext for rich error reporting.

Used in: AggExpr, Expr

message Rand

expr.proto:713

Used in: Expr

message RandStr

expr.proto:738

Spark's RandStr (Spark 4.0+) returns a random alphanumeric string of the given length. It is non-deterministic: the resolved random seed is combined with the partition index and seeds an XORShiftRandom, matching org.apache.spark.sql.catalyst.expressions.ExpressionImplUtils.randStr.

Used in: Expr

message Regr

expr.proto:284

Simple linear regression aggregates (regr_slope, regr_intercept, regr_r2, regr_sxx, regr_syy, regr_sxy). child1 is the dependent variable (y) and child2 is the independent variable (x).

Used in: AggExpr

enum Regr.RegrType

expr.proto:285

Used in: Regr

message ScalarFunc

expr.proto:598

Used in: Expr

message Second

expr.proto:582

Used in: Expr

message Shuffle

expr.proto:721

Spark's Shuffle returns a random permutation of the given array. It is non-deterministic: the resolved random seed is combined with the partition index and drives a MersenneTwister-based inside-out Fisher-Yates shuffle, matching org.apache.spark.sql.catalyst.util.RandomIndicesGenerator.

Used in: Expr

enum SortDirection

expr.proto:689

Used in: SortOrder

message SortOrder

expr.proto:517

Used in: Expr

enum StatisticsType

expr.proto:190

Used in: Covariance, Stddev, Variance

message Stddev

expr.proto:264

Used in: AggExpr

message Subquery

expr.proto:654

Used in: Expr

message Sum

expr.proto:199

Used in: AggExpr

message ToCsv

expr.proto:538

Used in: Expr

message ToJson

expr.proto:523

Used in: Expr

message ToPrettyString

expr.proto:562

Used in: Expr

message TruncTimestamp

expr.proto:643

Used in: Expr

message UnaryExpr

expr.proto:499

Used in: Expr

message UnaryMinus

expr.proto:632

Used in: Expr

message UnboundReference

expr.proto:512

Used in: Expr

message UnixTimestamp

expr.proto:587

Used in: Expr

message Uuid

expr.proto:730

Spark's Uuid returns a random RFC 4122 version 4 UUID string. It is non-deterministic: the resolved random seed is combined with the partition index and seeds a Commons Math3 MersenneTwister, matching org.apache.spark.sql.catalyst.util.RandomUUIDGenerator.

Used in: Expr

message Variance

expr.proto:257

Used in: AggExpr