Proto commits in spiraldb/vortex

These 92 commits are when the Protocol Buffers files have changed:

2026-06-23

Add JSON to Parquet Variant conversion with shredding (#8391) ## Summary This PR adds a `vortex.json_to_variant` expression for converting UTF-8 or `vortex.json` extension arrays into Variant arrays. The expression lives in `vortex-json` so JSON-facing expression metadata and serialization are independent of a concrete Variant encoding. `vortex-parquet-variant` provides the concrete execution kernels, parsing JSON through `parquet_variant_compute` and producing Parquet Variant storage. --------- Signed-off-by: Adam Gutglick <adam@spiraldb.com>

The documentation is generated from this commit.

2026-06-20

Numerical aggregate functions have an option to skip or include nans in calculation, skip by default (#8457) Almost all of the time you want to skip nans but for rare cases when you don't we need to be able to configure it

2026-06-17

Json-Variant exprs Signed-off-by: Adam Gutglick <adam@spiraldb.com>

2026-05-15

Updated Variant array and the new VariantGet expression (#7877) ## Summary This PR includes two big changes as Variant moves closer to readiness. 1. Potentially holding the `shredded` child of a variant array in the canonical VariantArray 2. A `VariantGet` expression that can pull extract data out of variant arrays, either in a typed way or as a more opaque `Variant`. For reviewers, some relevant context might be: 1. The [VariantGet](https://github.com/vortex-data/rfcs/pull/58) RFC: this RFC takes some lessons I've learned working on this into account and reflects my updated view of this problem. 2. The original [Variant type](https://vortex-data.github.io/rfcs/rfc/0015.html) RFC I think the Parquet spec is also a pretty good read and a very heavy influence of this work - [`Shredding`](https://parquet.apache.org/docs/file-format/types/variantshredding/) and the [`Variant type`](https://parquet.apache.org/docs/file-format/types/variantencoding/). --------- Signed-off-by: "Adam Gutglick" <adam@spiraldb.com> Signed-off-by: Adam Gutglick <adam@spiraldb.com>

2026-05-13

Add `UnionVariants` type and embed it in `DType::Union` Replaces `DType::Union(Nullability)` with `DType::Union(UnionVariants, Nullability)`. `UnionVariants` carries a `FieldNames` list, per-variant `FieldDType`s, and an `i8` type-id vector (supporting non-consecutive tags like `[0, 5, 7]` so children can be removed without renumbering). The flatbuffer and protobuf `Union` schemas are extended with the `names`/`dtypes`/`type_ids` fields, and serde, flatbuffers, and protobuf paths are filled in with round-trip tests and nullability-constraint checks. All the `todo!()` stubs added in the previous commit for `DType::Union(..)` arms are left in place; concrete implementations land alongside the array work. Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>

2026-05-12

Add `DType::Union` variant carrying just `Nullability` (#7901) ## Summary Tracking issue: https://github.com/vortex-data/vortex/issues/7882 Adds the `Union(Nullability)` enum variant, plumbs it through all existing match arms with `todo!("TODO(connor)[Union]: unimplemented")`, and reserves slot 12 in the flatbuffer and protobuf wire schemas. A follow-up PR will replace `Union(Nullability)` with `Union(UnionVariants, Nullability)` and fill in the serde paths. Note that this PR doesn't have any implementation logic, it is just boilerplate. ## API Changes Adds a new `DType::Union` variant. ## Testing Nothing to test. Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>

2026-05-11

Remove unused protobuf dependency in java bindings (#7886) This is no longer used. Got left behind latest refactors Signed-off-by: Robert Kruszewski <github@robertk.io>

2026-04-29

VariantGet with DType Signed-off-by: Adam Gutglick <adam@spiraldb.com>

[WIP] Re-shaping the canonical variant array to support more general forms of shredding Signed-off-by: Adam Gutglick <adam@spiraldb.com>

2026-04-20

Add SafeDiv binary operator ## Summary Adds `Operator::SafeDiv` — a division variant where a non-null zero divisor yields 0 rather than erroring. Integer overflow still errors (`i32::MIN / -1`), matching `Div`. Null on either side propagates. ## Implementation The array-level kernel is a hand-written single-pass `match_each_native_ptype!` loop (not a composition of existing ops). It runs over primitive slices once, combining validity from both inputs and producing a `PrimitiveArray` directly. Wiring touches the usual places a new binary arithmetic operator hits: proto enum and generated bindings, `Operator` / `NumericOperator` with their conversions, the `Binary` scalar-fn dispatcher (including stat_falsification), scalar-level `checked_binary_numeric` for both primitive and decimal scalars, and the FFI C enum. To combine validity from the two input arrays, the kernel calls `Validity::and` and then eagerly materializes the resulting lazy `Binary(And)` expression via `execute::<BoolArray>(ctx)`. This required threading `ExecutionCtx` through `execute_numeric` from the `Binary` dispatcher. Scope note: array-level SafeDiv is primitive-only. A decimal-array input fails with a clear error rather than silently inheriting the zero-rhs-errs behavior from arrow-arith; scalar-level decimal SafeDiv (used by the constant-constant fast path) is supported. ## Testing - 5 primitive scalar tests (integer, integer-by-zero, overflow still errs, float-by-zero yields 0.0 not Inf, null propagation) - 1 decimal scalar test (by-zero yields zero, not None) - 4 array-level tests (integer-by-zero, float-by-zero, nullable propagation from both sides, constant-zero rhs fast path) - Conformance harness: SafeDiv added to the operator arrays and is exercised with zero-valued scalars (unlike Div, which is skipped) - `cargo test -p vortex-array --lib` → 2556 passed, 0 failed - `cargo clippy -p vortex-array -p vortex-ffi -p vortex-proto --all-targets --all-features` → clean - `cargo fmt --all` → clean (nightly fmt unavailable in this sandbox) Signed-off-by: Claude <noreply@anthropic.com> https://claude.ai/code/session_01MLXfBVfMbrUgRPafnt5GHC Signed-off-by: Claude <noreply@anthropic.com>

2026-04-15

variant get Signed-off-by: Adam Gutglick <adam@spiraldb.com>

Better expr Signed-off-by: Adam Gutglick <adam@spiraldb.com>

2026-04-06

add array variant to scalar value Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>

2026-03-23

Scalar Signed-off-by: Adam Gutglick <adam@spiraldb.com>

Initial work Signed-off-by: Adam Gutglick <adam@spiraldb.com>

Filling out the missing pieces Signed-off-by: Adam Gutglick <adam@spiraldb.com>

2026-03-22

merge Signed-off-by: Nicholas Gates <nick@nickgates.com>

2026-03-19

chore: `AggregateFnRef` [de]serialize (#7070) Signed-off-by: blaginin <github@blaginin.me>

2026-03-18

Variant DType (#6912) This PR includes initial support for the Variant DType, as described in the [Variant RFC](https://github.com/vortex-data/rfcs/blob/develop/accepted/0015-variant-type.md). It includes most of the required boilerplate and initial structure for this new dtype. It includes the following changes: 1. New dtype 2. serialization for the dtype 3. Scalar variant and new scalar value 4. A lot of code paths that aren't supported yet and error accordingly. --------- Signed-off-by: Adam Gutglick <adam@spiraldb.com>

2026-03-14

Split up binary ScalarFn Signed-off-by: Nicholas Gates <nick@nickgates.com>

2026-03-04

feat[expr]: N-ary CASE WHEN expression (#6786) ## Summary Introduces CASE WHEN expression support in Vortex as a scalar function, implementing a true n-ary CASE WHEN cond1 THEN val1 WHEN cond2 THEN val2 ... ELSE default END expression. ## Changes included This pull request introduces support for SQL-style CASE WHEN expressions in Vortex, including both the expression logic and integration with DataFusion. The changes add new expression constructors, conversion logic, and pushdown support for CASE WHEN, as well as comprehensive benchmarks and tests to ensure correctness and performance. **CASE WHEN expression support:** * Added new constructors `case_when`, `case_when_no_else`, and `nested_case_when` to build CASE WHEN expressions in `exprs.rs`, leveraging the new `CaseWhen` scalar function and its options. [[1]](diffhunk://#diff-d1b532c524303a975ccf98376ed880c0b6e4bab2fce91e99157aac49f9c30e66R24-R25) [[2]](diffhunk://#diff-d1b532c524303a975ccf98376ed880c0b6e4bab2fce91e99157aac49f9c30e66R115-R168) * Introduced `CaseWhenOpts` protobuf message to encode CASE WHEN options for serialization. * Registered the new `case_when` scalar function module. **DataFusion integration:** * Implemented conversion from DataFusion's `CaseExpr` to Vortex's CASE WHEN expressions, supporting the "searched CASE" form and mapping WHEN/THEN/ELSE clauses. [[1]](diffhunk://#diff-ca68b66d97eff4b97ef5e70b3c324078ee441d91422f8d3a426f47882a61a202R148-R186) [[2]](diffhunk://#diff-ca68b66d97eff4b97ef5e70b3c324078ee441d91422f8d3a426f47882a61a202R278-R281) * Enhanced pushdown logic to support CASE WHEN expressions, including recursive checks for child expressions and ELSE clauses. [[1]](diffhunk://#diff-ca68b66d97eff4b97ef5e70b3c324078ee441d91422f8d3a426f47882a61a202L383-R432) [[2]](diffhunk://#diff-ca68b66d97eff4b97ef5e70b3c324078ee441d91422f8d3a426f47882a61a202R445-R510) **Benchmarking and testing:** * Added comprehensive benchmarks for various CASE WHEN scenarios in `case_when_bench.rs` and registered them in `Cargo.toml`. [[1]](diffhunk://#diff-c745f3a7a34d4a4f657c8e2e9e47197ec0f36e2886656366721559f36b76eca9R1-R210) [[2]](diffhunk://#diff-b1d55e82b9c9a25ba02540f4e1d46def7467de866587345f75b63f9ac51c04a6R130-R134) * Added an equivalence test to ensure that DataFusion CASE WHEN results match Vortex's results for the same input data. **Other improvements:** * Refined scalar function pushdown checks and utility logic for expression convertibility. * Minor code quality improvements and error handling in the expression module. ## Testing * Added a comprehensive test to verify equivalence between DataFusion and Vortex results for a `CASE WHEN` expression applied to an Arrow `RecordBatch`. ## Benchmarking * Introduced a new benchmark for various `CASE WHEN` scenarios, including simple, nested, all-true, and all-false cases, in `benches/expr/case_when_bench.rs`, and registered it in `Cargo.toml`. [[1]](diffhunk://#diff-c745f3a7a34d4a4f657c8e2e9e47197ec0f36e2886656366721559f36b76eca9R1-R144) [[2]](diffhunk://#diff-b1d55e82b9c9a25ba02540f4e1d46def7467de866587345f75b63f9ac51c04a6R130-R134) Benchmark | 1K rows | 10K rows | 100K rows -- | -- | -- | -- case_when_simple | 5.4 µs | 8.4 µs | 29.5 µs case_when_without_else | 5.5 µs | 8.7 µs | 29.4 µs case_when_all_true | 4.2 µs | 6.3 µs | 20.7 µs case_when_all_false | 4.3 µs | 6.4 µs | 20.1 µs case_when_nary_3_conditions | 15.1 µs | 25.8 µs | 87.3 µs case_when_nary_equality_lookup (5) | 25.2 µs | 39.7 µs | 125.4 µs case_when_nary_10_conditions | 45.5 µs | 73.1 µs | 257.5 µs --------- Signed-off-by: Luke Kim <80174+lukekim@users.noreply.github.com>

2025-11-07

fix: correctly (de)serialize merge with error duplicate handling Signed-off-by: Daniel King <dan@spiraldb.com>

2025-10-03

Expand expression pushdown to DataFusion (#4398) Started as a demo for @danking to show how to add more pushdown to Datafusion, now it includes some more expression (`IS NULL`, `IS NOT NULL` and `IN LIST`), which only leaves `substr` and dynamic expressions. It also allows for pushing more binary operators for arithmetic operations. `vortex-compact` seems really noise, but clickbench still shows nice improvements in a bunch of queries for the btrblocks compression. --------- Signed-off-by: Adam Gutglick <adam@spiraldb.com>

2025-09-26

More ops Signed-off-by: Adam Gutglick <adam@spiraldb.com>

2025-09-25

More ops Signed-off-by: Adam Gutglick <adam@spiraldb.com>

2025-08-27

Feature: `FixedSizeList` infrastructure (#4385) Adds infrastructure to support `FixedSizeList` including: - Arbitrary implementations - Flatbuffers - Protobuf - Some tests (commit granularity is nice if you want to look at that) --------- Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>

add `FixedSizeList` Protobuf Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>

2025-08-15

feat: Support more datafusion features (#4180) After spending some time today debugging a Datafusion issue with @joseph-isaacs and @robert3005 (ended up not being related to us), I figured I'll use the momentum to add some of the missing features: 1. Support for basic arithmetic expression pushdown (which exists in TPC-DS), including subtraction to `vortex-expr`. 2. Support files with different (but compatible) schemas in the same table. This is currently only supported if a `PhysicalExprAdapterFactory` is provided (either through the `FileScanConfig` or directly). 3. Support hive-style partitioning columns. Both features aren't fully fleshed out and probably require more tests, I want to figure out if we can use datafusion's own test suite to close that gap, they have pretty cool tooling to test SQL. One of the changes it required is making the source and opener more datafusion-y, doing most of the conversion work into Vortex in the actual `FileOpenFuture`. It allows us to utilize more of datafusion's tools for rewriting expressions over specific files. --------- Signed-off-by: Adam Gutglick <adam@spiraldb.com>

feat: More Datafusion support Signed-off-by: Adam Gutglick <adam@spiraldb.com>

2025-08-14

feat: More Datafusion support Signed-off-by: Adam Gutglick <adam@spiraldb.com>

2025-08-11

feat: More Datafusion support Signed-off-by: Adam Gutglick <adam@spiraldb.com>

2025-08-09

feat: More Datafusion support Signed-off-by: Adam Gutglick <adam@spiraldb.com>

2025-08-05

fix: error-prone f16 Scalar serde (#4113) ⏺ Summary of Changes The diff shows comprehensive changes to fix scalar casting bugs and ensure backwards compatibility for serialized scalar values. Here are the key changes: 1. Core Bug Fix: Floating-Point Conversion (vortex-scalar/src/pvalue.rs) - Fixed critical bug where u64 to f64 conversion was using bit reinterpretation (f64::from_bits()) instead of numeric conversion - Changed to use NumCast::from() for proper numeric conversions 2. Type Coercion System (vortex-scalar/src/lib.rs) - Added coerce_value() method in Scalar::new() to handle backwards compatibility - Specifically coerces f16 values stored as u64 (for backwards compatibility with serialized data) - Includes recursive coercion for structs and lists containing f16 fields - Protected against overflow (only coerces u64 to f16 if value ≤ u16::MAX) 3. Protobuf Support for f16 (vortex-proto/proto/scalar.proto) - Added dedicated f16_value field to protobuf schema (field 10) - Updated serialization to use this dedicated field instead of storing f16 as u64 4. Consistent Scalar Construction - Updated all direct Scalar { dtype, value } constructions to use Scalar::new() - Ensures coercion logic is always applied - Changed in: - bool.rs: into_scalar(), bool(), From<bool> - primitive.rs: primitive_value(), From<$T> for Scalar - struct_.rs: field_by_idx(), cast(), struct_() - list.rs: element(), list(), list_empty() 5. Test Coverage (vortex-scalar/src/lib.rs and proto.rs) - Fixed test_canonicalize_scalar_values to create f16 values properly - Added comprehensive coercion tests: - f16 coercion from u64 - No coercion for other type combinations - Struct field coercion - List element coercion - Overflow protection - Added ScalarValue protobuf roundtrip tests: - Direct f16 roundtripping - Value preservation for all primitive types - Handling of type consolidation (u8/u16/u32 → u64, i8/i16/i32 → i64) 6. Key Design Decision - Only f16 from u64 is coerced (minimal approach for backwards compatibility) - Other floating-point types (f32, f64) are NOT coerced - This preserves backwards compatibility while fixing the root cause The changes ensure that: 1. Scalar casting now uses numeric conversion instead of bit reinterpretation 2. Old serialized f16 values (stored as u64) are properly handled 3. All scalar constructions go through the coercion logic 4. Comprehensive tests verify the behavior --------- Signed-off-by: Will Manning <will@willmanning.io>

stop tagging f16 as u64 in proto Signed-off-by: Will Manning <will@willmanning.io>

2025-07-31

feat: Add serialization support for Select expression (#4077) Signed-off-by: Adam Gutglick <adam@spiraldb.com>

2025-07-10

Remove var expression (#3829) Fixes #3671 * Removes the Var expression, leaving instead a `root()` expression for resolving the scope root. * The expression scope can hold context variables, useful for passing in auth tokens for example, but not variables that would impact the return_dtype of the expression. * ScopeDType has therefore been removed, because the dtype of the scope _is_ just the dtype of the root array. * Simplifies some transformation / partitioning logic where vars no longer need to be considered. Signed-off-by: Nicholas Gates <nick@nickgates.com> Co-authored-by: Will Manning <will@spiraldb.com>

2025-07-08

VortexExpr VTables (#3713) Adds the same vtable machinery as arrays and layouts already use. It uses the "Encoding" naming scheme from arrays and layouts. I don't particularly like it, but it's consistent. Open to renames later. Further, adds an expression registry to the Vortex session that will be used for deserialization. Expressions only decide their "options" serialization. So in theory, can support many container formats, not just proto, provided each expression can deserialize their own options format. --------- Signed-off-by: Nicholas Gates <nick@nickgates.com>

2025-07-04

Merge

Merge

chore: SPDX license & copyrights (#3753)

SelectExpr Signed-off-by: Nicholas Gates <nick@nickgates.com>

2025-07-01

Fix ocmpile

Extensible expression serde Signed-off-by: Nicholas Gates <nick@nickgates.com>

2025-06-23

chore: Rename CheckedAdd operator to Add (#3612) Signed-off-by: Robert Kruszewski <github@robertk.io>

feat[vortex-expr]: CheckedAdd operator (#3603) Signed-off-by: Daniel King <dan@spiraldb.com>

chore[vortex-expr]: remove `let` expr (#3604)

2025-06-13

feat[vortex-layout]: add row_id layout reader to support row_id expr queries (#3449) Add the following well defined Scope value ``` dtype("$vx.row_id") = {file_index: u64, file_row_number: u64} ``` This can be used in the scan, and file pruning --------- Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>

2025-06-12

update Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>

2025-06-05

list_contains to use any semantics (#3448) Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>

2025-06-04

wip Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>

feat[vortex-expr]: let expression (#3447) add let x = e in e and var(x) expressions, var("") replaces identity Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> --------- Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>

fix Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>

Merge branch 'develop' into ji/let-expr # Conflicts: # vortex-layout/src/layouts/flat/writer.rs # vortex-proto/src/generated/vortex.expr.rs

add let and var expr Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>

2025-06-03

Adds `list_contains` expression (#3410) Co-authored-by: Andrew Duffy <andrew@a10y.dev>

Merge branch 'develop' into ji/let-expr # Conflicts: # vortex-proto/src/generated/vortex.expr.rs

Add `cast` expression (#3440)

2025-06-02

let expr Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>

This commit will be squashed.

2025-05-30

wip Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>

2025-05-29

This commit will be squashed.

2025-05-16

feat: add is_null expression (#3324)

2025-04-30

Use protobuf to serialize scalars (#3141)

fixes

Use protobuf to serialize scalars

2025-04-29

Add decimal with storage size i8..i64 to vortex (#3161)

add decimal i8..i64

2025-04-28

Use protobuf to serialize scalars

2025-04-24

feat: Decimal types (#3058) Co-authored-by: Robert Kruszewski <github@robertk.io>

use uint32 for decimal precision in proto

2025-04-22

closer

feat: Decimal types

2025-04-21

closer

2025-04-18

feat: Decimal types

2025-04-07

Add all vortex expressions to proto serde (#2907) Add serde + proto expression for like and between.

2025-03-21

feat(java): Expression serialization protobufs (#2744)

2025-03-20

Use proto c++ (#2734) Add marshalling from C++ into Expr proto --------- Co-authored-by: Alexander Droste <alexander.droste@protonmail.com>

2025-03-19

Vortex expr protobuf (#2725) Added a first impl sketch for voretx-expr protobufs. I think in the future we will replace the `Kind kind = 3;` with a more extendable system, maybe proto any or json?

2025-02-07

Improve expression performance (#2273) * Only simplify once in the scanner * Manually implement return_dtype instead of expensive Canonical::Empty (FLUP: we should add `return_nullability()` expr too) * Remove `Field::Index` weirdness. * More consistent APIs for Struct{Scalar,Array,DType}

2025-01-30

chore: remove unused Decimal dtype in proto/fbs (#2156)

2024-11-01

feat: specify the storage_dtype in ExtDType (#1007) Adds `storage_dtype` to `ExtDType`. This PR adds `storage_dtype` to `ExtDType`. This is desirable for a few reasons * Makes it possible to canonicalize an empty chunked ExtensionArray * Makes it possible to determine the storage DType for a ConstantArray without examining its value * Makes it possible for Vortex to reason about externally authored extension types. This is still not fully complete, as an ideal experience would allow extension authors to override IntoCanonical, IntoArrow, Display, etc. To avoid duplicating the nullability, we remove top-level `nullability` from the `DType::Extension` variant, instead nullability is accessed through the inner ExtDType. --------- Co-authored-by: Will Manning <will@willmanning.io>

2024-08-21

Push filter schema manipulation into layout reader and reuse ipc message writer in file writer (#651) This is done in preparation for pruning as we need to be able to mutate the filters and read columns based on filters

2024-08-08

Lots of things to try and get publishing working (#557)

2024-06-12

Clean up fields / field paths (#353)

Expression proto serde (#351)

2024-06-09

Implement StructValue proto serde without google.protobuf.Value (#343) Move away from google.protobuf.Value. It makes things unnecessary complicated - number is always f64, look at the previous logic for handling struct.... It seems like the Value models JSON value which is not what we need. I considered using google.protobuf wrappers, e.g. BytesValue, Int32Value, instead of primitives but those are also for usage with JSON or with proto2 (before optional was introduced to enable optional primitive fields), so decided to use proto primitives. Adds to proto and try from proto.

2024-05-16

define ScalarValue in VortexScalar protobuf (#323) more aligned with the flatbuffers definition

Proto Refactor (#325)

2024-05-08

Add ScalarView (#301) Fixes #290 Makes scalar generic over heap allocated data or serialised scalar data. Serialized scalars will no longer include DType information to avoid duplication meaning they require a DType to deserialise.

2024-05-07

DType Serialization (#298) * Clean up DType serde * Create build-vortex package to unify flatbuffer/protobuffer code generation. * Add protobuf DType definitions. FLUP questions: * Should we remove nullable from DType and define a separate `FieldType { dtype: DType, nullable: bool }`? Recursive types such as Struct and List would then use FieldType. It does introduce an interesting quirk of a nullable null... But maybe that's less annoying that duplicating nullability everywhere? * If we don't want a separate FieldType, we could still pull `nullable` up in both flatbuffer and protobuf definitions. Start of #277

2022-01-31

We dont really have to keep proto files, start of the substrait extension

This commit does not contain any .proto files.

Adding substrait as a third party and script to update/gen substrait files