Get desktop application:
View/edit binary Protocol Buffers messages
Used in: ,
Optional filter expression for SQL FILTER (WHERE ...) clause. Only set in Partial aggregation mode; absent in Final/PartialMerge.
Optional QueryContext for error reporting (contains SQL text and position)
Unique expression ID for context lookup during error creation
Used in:
Child value expression, already cast to Float64 by the serde.
The percentiles and accuracy are carried as resolved scalars rather than child Exprs (unlike Percentile/BloomFilterAgg) because they are needed at UDAF construction time to drive return_type and accumulator shape. One or more percentiles in [0.0, 1.0].
Spark's accuracy argument; relative_error = 1.0 / accuracy.
True when the percentile argument was an array (output is a list).
Spark's input/output type, used to cast results back from Float64.
Array functions
Used in:
Spark's ArraysZip takes children: Seq[Expression] and names: Seq[Expression] https://github.com/apache/spark/blob/branch-4.1/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala#L296
Used in:
Counts non-null and non-NaN children, with per-row short-circuit evaluation.
Used in:
Used in:
Used in:
Used in:
Used in:
Used in:
Used in:
Used in:
Output serialization version. Spark 4.0 and earlier always wrote V1; Spark 4.1+ defaults to V2 (different bit-scattering algorithm and a `seed` field in the binary format). The JVM serde sets this to the matching version so Comet's aggregate output is byte-equivalent with Spark's.
Used in:
Used in:
Bound to a particular vector array in input batch.
Used in:
Used in:
The expr field is added to be consistent with CaseExpr definition in DataFusion. This field is not really used. When constructing a CaseExpr, this expr field is always set to None. The reason that we always set this expr field to None is because Spark parser converts the expr to a EqualTo conditions. After the conversion, we don't see this expr any more so it's always None.
Used in:
True when running against Spark 4.0+. Controls version-specific cast behaviour such as the handling of leading whitespace before T-prefixed time-only strings.
True when the running Spark casts a zero string with any exponent ("0e39") to decimal as 0, instead of applying the integral-digit limit to it (SPARK-60118).
True when the running Spark casts a string far below the target scale's ulp to decimal as 0 even when the difference to the target scale does not fit in an int, instead of failing in BigDecimal.setScale (SPARK-60119).
True when Spark does not apply the 38 integral-digit limit when it casts a string to decimal, as it does not with spark.sql.legacy.allowNegativeScaleOfDecimal.
Used in:
Used in:
Used in:
Used in:
The mode of the division in Spark's `ck / sqrt(xMk * yMk)`. A zero denominator returns null in LEGACY mode and raises DIVIDE_BY_ZERO in ANSI mode.
Used in:
Used in:
Used in:
Used in:
Used in: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ,
Used in:
Used in:
Used in:
Used in:
Used in:
Parquet field ID for the synthetic list element, when provided by Catalyst metadata.
Used in:
Parquet field IDs for the synthetic map fields, when provided by Catalyst metadata.
Used in:
Parallel to field_names. Each entry holds Spark field metadata for the corresponding nested field. Currently used to carry parquet.field.id through to the native side. Empty when no fields carry metadata.
Used in:
(message has no fields)
Used in: , , , , ,
The basic message representing a Spark expression.
Used in: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ,
Optional QueryContext for error reporting (contains SQL text and position)
Unique expression ID for context lookup during error creation
Used in:
Used in:
Used in:
Used in:
A DataFusion higher-order function applied to one array argument and a one-parameter lambda, e.g. `array_filter(array, x -> body)`. The lambda body is an ordinary expression tree in which `LambdaVariable` nodes refer to the lambda parameter and `BoundReference` nodes refer to columns of the enclosing operator's input.
Used in:
DataFusion higher-order function name, e.g. "array_filter".
Unique name of the lambda parameter. `LambdaVariable.name` nodes in the body resolve against it.
approx_count_distinct (Spark's HyperLogLogPlusPlus)
Used in:
Number of addressing bits, computed from relativeSD to match Spark.
Used in:
Child value expression (integral, string, or binary).
DataSketches lgConfigK (log2 of the number of buckets), Spark default 12.
Used in:
Child sketch expression (Binary column of serialized HLL sketches).
When false, Spark errors if input sketches have differing lgConfigK.
Used in:
Used in:
Used in:
Used in:
Scalar UDF dispatched to the JVM via JNI. Native side exports input arrays through Arrow C Data Interface, calls CometUdfBridge.evaluate, and imports the result.
Used in:
Fully-qualified Java/Scala class name implementing org.apache.comet.udf.CometUDF (must have a public no-arg constructor).
Argument expressions, evaluated by the native side before invocation.
Expected return type. Used to import the result FFI_ArrowArray.
Whether the result column may contain nulls.
Reference to a lambda parameter from inside a `HigherOrderFunc` body. The element type is taken from the array argument at planning time.
Used in:
Used in:
Spark 4.0+ LISTAGG / STRING_AGG aggregate. Comet only serializes the simple form: a StringType child with a literal (or NULL) delimiter and no WITHIN GROUP ORDER BY. DISTINCT falls back to Spark because Comet rejects multi-column distinct aggregates in aggExprToProto, so the native side never sees it.
Used in:
Literal delimiter expression. NULL delimiter is normalized to empty string by Spark's semantics. The result type is always Utf8 (Spark StringType), so no datatype field is needed.
Used in:
Evaluate the ordinal only for the rows where the child is not NULL, as in ScalarFunc
Used in:
Only one of these fields should be populated based on the array type
Used in: , ,
Protobuf doesn't provide int8 and int16, we put them into int32 and convert to int8 and int16 when deserializing.
Used in:
Only set for integral_divide: mirrors Spark's IntegralDivide.checkDivideOverflow (LONG operands with failOnError). When this is set and eval_mode is ANSI, a quotient that does not fit in a LONG (Long.MinValue div -1) throws ARITHMETIC_OVERFLOW instead of wrapping around. Never set for DECIMAL operands, which wrap around on the cast to LONG even in ANSI mode, matching Spark.
Used in:
Used in:
The value returned by the aggregate (associated with the maximum ordering).
The ordering expression whose maximum selects the returned value.
Used in:
Used in:
The value returned by the aggregate (associated with the minimum ordering).
The ordering expression whose minimum selects the returned value.
Used in:
Used in:
Whether `-0.0` should be folded into `0.0` before it is used as a frequency-map key. Spark only started doing this in 4.2.0 (SPARK-57329), so this tracks the Spark version Comet is running against. See `Mode` in the spark-expr crate for the full rationale.
Call to a user-supplied native UDF loaded from a shared library. The native side resolves (library_path, name) against its loaded-library cache, looks up the kernel by name, and invokes it through the Comet UDF C ABI. That ABI is parameterized only by the Arrow C Data Interface, so the message says nothing about the language the library was written in; the Rust SDK is the only supported way to produce one today.
Used in:
Function name as registered through CometNativeUDF.register on the JVM side; matched against names exposed by the cdylib.
Filesystem path of the cdylib.
Argument expressions, evaluated before invocation.
Expected return type, declared at register time on the JVM side.
Whether the call is deterministic (mirrors Spark's deterministic flag). Always true today: registration rejects deterministic = false, because the native side plans every native UDF with Volatility::Immutable and has no way to express a volatile one. The field is carried so that honoring it later does not need a wire change.
Used in:
Used in:
Used in:
Single percentile in [0.0, 1.0] as a literal double expression.
Whether the child was FLOAT before the serde cast it to DOUBLE.
Whether Spark keeps 0.0 and -0.0 as separate keys in its percentile buffer, which it does from 3.5.2 (SPARK-45599), and the input is FLOAT or DOUBLE, so that it can hold both.
Whether the aggregate is a PartialMerge, which serializes the buffer it merged its input into, so its state hands on the partials it merged as one.
Optimized Pivot second-phase aggregate. Given a `pivot_column` and a pre-computed list of `pivot_values`, buckets the incoming `value_column` into an array whose length is `pivot_values.size()`. Rows whose pivot value is not in `pivot_values` are ignored; if multiple rows in the same group map to the same bucket, the last non-null value wins (matches Spark's `PivotFirst.update`/`merge`). Output type is ArrayType(value_datatype). The pivot_values are serialized as Literal Exprs so the value bytes and data type reach native code faithfully.
Used in:
Spark's internal PreciseTimestampConversion, used by time-window grouping to convert between TimestampType/TimestampNTZType and LongType without losing microsecond precision. This is a pure reinterpret: the underlying value is unchanged, only the type is changed.
Used in:
QueryContext provides SQL query context for error messages. Mirrors Spark's SQLQueryContext for rich error reporting.
Used in: ,
Full SQL query text. Left empty when `sql_text_idx` is set, which is the case for a plan serialized through `CometNativeExec.convertBlock`. Still used by the paths that serialize an operator directly without interning (e.g. `CometNativeWriteExec`, the native shuffle writer, the Iceberg scan's `convertBlock` override) and by the bare-`Expr` Parquet filter path, which has no root operator to carry a pool.
Character offset where expression starts (0-based)
Character offset where expression ends (0-based, inclusive)
Type of SQL object (e.g., "VIEW", "Project", "Filter")
Name of object (e.g., view name, column name)
Line number in SQL query (1-based)
Column position within the line (0-based)
Index into the root `Operator.sql_text_pool` holding this context's SQL text. Set instead of `sql_text` so that a query text shared by many expressions is serialized once per plan rather than once per expression.
Used in:
Spark's RandStr (Spark 4.0+) returns a random alphanumeric string of the given length. It is non-deterministic: the resolved random seed is combined with the partition index and seeds an XORShiftRandom, matching org.apache.spark.sql.catalyst.expressions.ExpressionImplUtils.randStr.
Used in:
Simple linear regression aggregates (regr_slope, regr_intercept, regr_r2, regr_sxx, regr_syy, regr_sxy). child1 is the dependent variable (y) and child2 is the independent variable (x).
Used in:
Only consulted for SLOPE and INTERCEPT. When true (SPARK-48719), VariancePop(x) is computed only over rows where both y and x are non-null. When false, VariancePop(x) includes every row where x is non-null even if y is null, matching earlier Spark patches.
Only consulted for R2. Spark swapped the degenerate-case handling of regr_r2. When true, a constant dependent variable (m2(y) = 0) returns 1.0 and a constant independent variable (m2(x) = 0) returns null. When false those two cases are reversed. The swap is present in the Spark versions Comet builds against (3.5.9, 4.0.3+, 4.1, 4.2).
Only consulted for R2. Division by an underflowed moment product returns null in LEGACY mode and raises DIVIDE_BY_ZERO in ANSI mode.
Used in:
Used in:
Evaluate each argument after the first only for the rows where no earlier argument is NULL, as Spark does for a null-intolerant BinaryExpression or TernaryExpression. See QueryPlanSerde.withNullShortCircuit.
Used in:
Spark's Shuffle returns a random permutation of the given array. It is non-deterministic: the resolved random seed is combined with the partition index and drives a MersenneTwister-based inside-out Fisher-Yates shuffle, matching org.apache.spark.sql.catalyst.util.RandomIndicesGenerator.
Used in:
Used in:
Used in:
Used in: , ,
Used in:
Used in:
Used in:
Used in:
Used in:
Used in:
Used in:
Whether SECOND and MILLISECOND truncation wraps when the result would fall below the smallest timestamp. Spark before 4.2.0 wraps the Long subtraction; 4.2.0 and later (SPARK-56663) raise `long overflow` like every other unit. This tracks the Spark version Comet is running against.
Used in:
Used in:
Used in:
Used in:
Spark's Uuid returns a random RFC 4122 version 4 UUID string. It is non-deterministic: the resolved random seed is combined with the partition index and seeds a Commons Math3 MersenneTwister, matching org.apache.spark.sql.catalyst.util.RandomUUIDGenerator.
Used in:
Used in: