Proto commits in spiraldb/vortex

These commits are when the Protocol Buffers files have changed: (only the last 100 relevant commits are shown)

2026-07-28

inital work Signed-off-by: Adam Gutglick <adam@spiraldb.com>

Add fixed-size binary canonical type Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>

2026-07-27

Add fixed-size binary canonical type Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>

inital work Signed-off-by: Adam Gutglick <adam@spiraldb.com>

2026-07-23

Add fixed-size binary canonical type Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>

2026-07-22

Add fixed-size binary canonical type Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>

2026-07-21

Add fixed-size binary canonical type Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>

Add `UnionValue` and `UnionScalar` (#8838) _Revived from https://github.com/vortex-data/vortex/pull/8791 since too many things have changed._ ## Summary Tracking issue: https://github.com/vortex-data/vortex/issues/8769 Adds scalar implementation for `Union` type in Vortex. ## Changes Adds `UnionValue` which exists inside of `ScalarValue` and the typed view `UnionScalar<'a>`. ```rust pub struct UnionValue { /// The type ID selecting a variant in the enclosing [`DType::Union`]. type_id: u8, /// The selected variant scalar. /// /// This is boxed to break the recursive layout between [`ScalarValue`] /// and [`UnionValue`]. value: `Option<Box<ScalarValue>>`, } ``` ### Inner Nulls with Type IDs The type above implies that type IDs MATTER for equality of nulls. 2 values with the same union type might both be null, **but if the type IDs are different, we consider them NOT equal.** Please respond below if you think this is the incorrect behavior. I think that this might be different from arrow's semantics, but because we are already distinguishing from them with the outer nullability, I think this is fine. ### Value type ~~I was debating if we should instead store a `Option<Box<ScalarValue>>` instead of just `Box<Scalar>` (which carries an extra dtype and an extra allocation even if the value is null). I think it is nice to have the dtype in there for implementation ease. Not sure if it's worth it.~~ We use `Option<Box<ScalarValue>>` instead of `Box<Scalar>` so that we do not store `DType` in multiple places. If we did so, we run into synchronization issues with casting. ### Casting For casting, this only allows identity and null casting (like everything else), otherwise it just panics with unimplemented. We probably want to figure this out later as it is quite valuable to cast union types. ## Default and Zero value semantics Finally, the "default value" and "zero value" semantics here are as such: - `default_value(nullable Union)` -> returns an outer null. - `default_value(non-nullable Union)` -> **panics** because there is no designated default variant. - `zero_value(Union)` -> **panics** because a Union has no canonical zero variant. - `is_zero()` on a null Union -> returns `None`. - `is_zero()` on a valid Union -> returns `Some(false)`, even when its child is numerically zero. Note that none of these behaviors depend on variant ordering. And also I do think that it is possible that we just choose the first variant of a union to be the default, I think this is kind of opinionated, and it also doesn't solve the issue where there are 0 variants (empty union / void type). So I would prefer if we do this in a separate change. --------- Signed-off-by: Connor Tsui <connor.tsui20@gmail.com> Co-authored-by: Joe Isaacs <joe.isaacs@live.co.uk>

The documentation is generated from this commit.

2026-07-20

Add fixed-size binary canonical type Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>

2026-07-17

Add Union scalar support Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>

The documentation is generated from this commit.

inital work Signed-off-by: Adam Gutglick <adam@spiraldb.com>

2026-07-16

Add Union scalar support Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>

Add outer nullability to Union (#8798) Tracking issue: https://github.com/vortex-data/vortex/issues/8769 Adds the breaking Union dtype foundation before scalar and array support. ### Changes - changes `DType::Union` to carry independent outer nullability - keeps outer nullability separate from the nullability of each variant - removes runtime-derived Union nullability - updates dtype coercion, serialization, display, and downstream matches This changes the serialized Serde representation of Union dtypes and will require the compatibility override to be reviewed explicitly. Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>

Use u8 Union type IDs (#8790) Tracking issue: https://github.com/vortex-data/vortex/issues/8769 and https://github.com/vortex-data/vortex/issues/7882 Splits the type-ID foundation for #8769 into its own change. - changes Union variant type IDs from `i8` to `u8` across the dtype APIs, protobuf, FlatBuffers, and generated bindings - supports the full `u8` tag range - supports up to 256 Union variants with consecutive tags `0..=255` Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>

2026-07-15

Add Union scalar support Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>

2026-07-14

Add Union scalar support Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>

2026-07-13

Add Union scalar support Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>

2026-07-10

Remove stored `Union` `Nullability` (#8714) ## Summary Tracking issue: #7882 A union has no independent parent validity. A row is null when its selected child is null, so the union's logical nullability is determined by its variants. Storing the same information on `DType::Union` created two sources of truth and allowed the outer value to disagree with the variants. This change makes inconsistent union DTypes impossible. The remaining question is how an operation such as `mask` should change a union's variants when it introduces nulls. That remains deferred and does not block the canonical sparse `UnionArray`. The Arrow implementation notes and deferred operation scope are in [this comment on #7882](https://github.com/vortex-data/vortex/issues/7882#issuecomment-4935971614). ## Changes - change `DType::Union` to store only `UnionVariants` - derive union nullability at runtime from the variant DTypes - name the non-zero-cost scan `derived_nullability` - leave `with_nullability` unchanged for unions instead of rewriting variant schemas - restrict union least-supertype coercion to identical variants for now - remove Union nullability from Serde, protobuf, and FlatBuffers Supersedes #8713. --------- Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>

Add `UnionVariants` type and embed it in `DType::Union` (#8703) ## Rationale for this change Tracking Issue: https://github.com/vortex-data/vortex/issues/7882 We want to make `DType::Union` usable. This PR has been resurrected from https://github.com/vortex-data/vortex/pull/7902. ## What changes are included in this PR? Replaces `DType::Union(Nullability)` with `DType::Union(UnionVariants, Nullability)`. `UnionVariants` carries a `FieldNames` list, per-variant `FieldDType`s, and an `i8` type-id vector (supporting non-consecutive tags like `[0, 5, 7]` so children can be removed without renumbering). The flatbuffer and protobuf `Union` schemas are extended with the `names`/`dtypes`/`type_ids` fields, and serde, flatbuffers, and protobuf paths are filled in with round-trip tests and nullability-constraint checks. The flatbuffer `Union` table uses explicit field ids so that `nullable` keeps the id it had before this change, keeping the schema conformant with develop. Existing `DType::Union(..)` match arms that reject or skip unions are left in place, and concrete implementations will come later. ## What APIs are changed? Are there any user-facing changes? `DType::Union` will now have a type parameter `UnionVariants`. Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>

Add `UnionVariants` type and embed it in `DType::Union` Replaces `DType::Union(Nullability)` with `DType::Union(UnionVariants, Nullability)`. `UnionVariants` carries a `FieldNames` list, per-variant `FieldDType`s, and an `i8` type-id vector (supporting non-consecutive tags like `[0, 5, 7]` so children can be removed without renumbering). The flatbuffer and protobuf `Union` schemas are extended with the `names`/`dtypes`/`type_ids` fields, and serde, flatbuffers, and protobuf paths are filled in with round-trip tests and nullability-constraint checks. The flatbuffer `Union` table uses explicit field ids so that `nullable` keeps the id it had before this change, keeping the schema conformant with develop. Existing `DType::Union(..)` match arms that reject or skip unions are left in place; concrete implementations land alongside the array work. Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>

2026-06-23

Add JSON to Parquet Variant conversion with shredding (#8391) ## Summary This PR adds a `vortex.json_to_variant` expression for converting UTF-8 or `vortex.json` extension arrays into Variant arrays. The expression lives in `vortex-json` so JSON-facing expression metadata and serialization are independent of a concrete Variant encoding. `vortex-parquet-variant` provides the concrete execution kernels, parsing JSON through `parquet_variant_compute` and producing Parquet Variant storage. --------- Signed-off-by: Adam Gutglick <adam@spiraldb.com>

2026-06-20

Numerical aggregate functions have an option to skip or include nans in calculation, skip by default (#8457) Almost all of the time you want to skip nans but for rare cases when you don't we need to be able to configure it

2026-06-17

Json-Variant exprs Signed-off-by: Adam Gutglick <adam@spiraldb.com>

2026-05-15

Updated Variant array and the new VariantGet expression (#7877) ## Summary This PR includes two big changes as Variant moves closer to readiness. 1. Potentially holding the `shredded` child of a variant array in the canonical VariantArray 2. A `VariantGet` expression that can pull extract data out of variant arrays, either in a typed way or as a more opaque `Variant`. For reviewers, some relevant context might be: 1. The [VariantGet](https://github.com/vortex-data/rfcs/pull/58) RFC: this RFC takes some lessons I've learned working on this into account and reflects my updated view of this problem. 2. The original [Variant type](https://vortex-data.github.io/rfcs/rfc/0015.html) RFC I think the Parquet spec is also a pretty good read and a very heavy influence of this work - [`Shredding`](https://parquet.apache.org/docs/file-format/types/variantshredding/) and the [`Variant type`](https://parquet.apache.org/docs/file-format/types/variantencoding/). --------- Signed-off-by: "Adam Gutglick" <adam@spiraldb.com> Signed-off-by: Adam Gutglick <adam@spiraldb.com>

2026-05-13

Add `UnionVariants` type and embed it in `DType::Union` Replaces `DType::Union(Nullability)` with `DType::Union(UnionVariants, Nullability)`. `UnionVariants` carries a `FieldNames` list, per-variant `FieldDType`s, and an `i8` type-id vector (supporting non-consecutive tags like `[0, 5, 7]` so children can be removed without renumbering). The flatbuffer and protobuf `Union` schemas are extended with the `names`/`dtypes`/`type_ids` fields, and serde, flatbuffers, and protobuf paths are filled in with round-trip tests and nullability-constraint checks. All the `todo!()` stubs added in the previous commit for `DType::Union(..)` arms are left in place; concrete implementations land alongside the array work. Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>

2026-05-12

Add `DType::Union` variant carrying just `Nullability` (#7901) ## Summary Tracking issue: https://github.com/vortex-data/vortex/issues/7882 Adds the `Union(Nullability)` enum variant, plumbs it through all existing match arms with `todo!("TODO(connor)[Union]: unimplemented")`, and reserves slot 12 in the flatbuffer and protobuf wire schemas. A follow-up PR will replace `Union(Nullability)` with `Union(UnionVariants, Nullability)` and fill in the serde paths. Note that this PR doesn't have any implementation logic, it is just boilerplate. ## API Changes Adds a new `DType::Union` variant. ## Testing Nothing to test. Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>

2026-05-11

Remove unused protobuf dependency in java bindings (#7886) This is no longer used. Got left behind latest refactors Signed-off-by: Robert Kruszewski <github@robertk.io>

2026-04-29

VariantGet with DType Signed-off-by: Adam Gutglick <adam@spiraldb.com>

[WIP] Re-shaping the canonical variant array to support more general forms of shredding Signed-off-by: Adam Gutglick <adam@spiraldb.com>

2026-04-20

Add SafeDiv binary operator ## Summary Adds `Operator::SafeDiv` — a division variant where a non-null zero divisor yields 0 rather than erroring. Integer overflow still errors (`i32::MIN / -1`), matching `Div`. Null on either side propagates. ## Implementation The array-level kernel is a hand-written single-pass `match_each_native_ptype!` loop (not a composition of existing ops). It runs over primitive slices once, combining validity from both inputs and producing a `PrimitiveArray` directly. Wiring touches the usual places a new binary arithmetic operator hits: proto enum and generated bindings, `Operator` / `NumericOperator` with their conversions, the `Binary` scalar-fn dispatcher (including stat_falsification), scalar-level `checked_binary_numeric` for both primitive and decimal scalars, and the FFI C enum. To combine validity from the two input arrays, the kernel calls `Validity::and` and then eagerly materializes the resulting lazy `Binary(And)` expression via `execute::<BoolArray>(ctx)`. This required threading `ExecutionCtx` through `execute_numeric` from the `Binary` dispatcher. Scope note: array-level SafeDiv is primitive-only. A decimal-array input fails with a clear error rather than silently inheriting the zero-rhs-errs behavior from arrow-arith; scalar-level decimal SafeDiv (used by the constant-constant fast path) is supported. ## Testing - 5 primitive scalar tests (integer, integer-by-zero, overflow still errs, float-by-zero yields 0.0 not Inf, null propagation) - 1 decimal scalar test (by-zero yields zero, not None) - 4 array-level tests (integer-by-zero, float-by-zero, nullable propagation from both sides, constant-zero rhs fast path) - Conformance harness: SafeDiv added to the operator arrays and is exercised with zero-valued scalars (unlike Div, which is skipped) - `cargo test -p vortex-array --lib` → 2556 passed, 0 failed - `cargo clippy -p vortex-array -p vortex-ffi -p vortex-proto --all-targets --all-features` → clean - `cargo fmt --all` → clean (nightly fmt unavailable in this sandbox) Signed-off-by: Claude <noreply@anthropic.com> https://claude.ai/code/session_01MLXfBVfMbrUgRPafnt5GHC Signed-off-by: Claude <noreply@anthropic.com>

2026-04-15

Better expr Signed-off-by: Adam Gutglick <adam@spiraldb.com>

variant get Signed-off-by: Adam Gutglick <adam@spiraldb.com>

2026-04-06

add array variant to scalar value Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>

2026-03-23

Initial work Signed-off-by: Adam Gutglick <adam@spiraldb.com>

Filling out the missing pieces Signed-off-by: Adam Gutglick <adam@spiraldb.com>

Scalar Signed-off-by: Adam Gutglick <adam@spiraldb.com>

2026-03-22

merge Signed-off-by: Nicholas Gates <nick@nickgates.com>

2026-03-19

chore: `AggregateFnRef` [de]serialize (#7070) Signed-off-by: blaginin <github@blaginin.me>

2026-03-18

Variant DType (#6912) This PR includes initial support for the Variant DType, as described in the [Variant RFC](https://github.com/vortex-data/rfcs/blob/develop/accepted/0015-variant-type.md). It includes most of the required boilerplate and initial structure for this new dtype. It includes the following changes: 1. New dtype 2. serialization for the dtype 3. Scalar variant and new scalar value 4. A lot of code paths that aren't supported yet and error accordingly. --------- Signed-off-by: Adam Gutglick <adam@spiraldb.com>

2026-03-14

Split up binary ScalarFn Signed-off-by: Nicholas Gates <nick@nickgates.com>

2026-03-04

feat[expr]: N-ary CASE WHEN expression (#6786) ## Summary Introduces CASE WHEN expression support in Vortex as a scalar function, implementing a true n-ary CASE WHEN cond1 THEN val1 WHEN cond2 THEN val2 ... ELSE default END expression. ## Changes included This pull request introduces support for SQL-style CASE WHEN expressions in Vortex, including both the expression logic and integration with DataFusion. The changes add new expression constructors, conversion logic, and pushdown support for CASE WHEN, as well as comprehensive benchmarks and tests to ensure correctness and performance. **CASE WHEN expression support:** * Added new constructors `case_when`, `case_when_no_else`, and `nested_case_when` to build CASE WHEN expressions in `exprs.rs`, leveraging the new `CaseWhen` scalar function and its options. [[1]](diffhunk://#diff-d1b532c524303a975ccf98376ed880c0b6e4bab2fce91e99157aac49f9c30e66R24-R25) [[2]](diffhunk://#diff-d1b532c524303a975ccf98376ed880c0b6e4bab2fce91e99157aac49f9c30e66R115-R168) * Introduced `CaseWhenOpts` protobuf message to encode CASE WHEN options for serialization. * Registered the new `case_when` scalar function module. **DataFusion integration:** * Implemented conversion from DataFusion's `CaseExpr` to Vortex's CASE WHEN expressions, supporting the "searched CASE" form and mapping WHEN/THEN/ELSE clauses. [[1]](diffhunk://#diff-ca68b66d97eff4b97ef5e70b3c324078ee441d91422f8d3a426f47882a61a202R148-R186) [[2]](diffhunk://#diff-ca68b66d97eff4b97ef5e70b3c324078ee441d91422f8d3a426f47882a61a202R278-R281) * Enhanced pushdown logic to support CASE WHEN expressions, including recursive checks for child expressions and ELSE clauses. [[1]](diffhunk://#diff-ca68b66d97eff4b97ef5e70b3c324078ee441d91422f8d3a426f47882a61a202L383-R432) [[2]](diffhunk://#diff-ca68b66d97eff4b97ef5e70b3c324078ee441d91422f8d3a426f47882a61a202R445-R510) **Benchmarking and testing:** * Added comprehensive benchmarks for various CASE WHEN scenarios in `case_when_bench.rs` and registered them in `Cargo.toml`. [[1]](diffhunk://#diff-c745f3a7a34d4a4f657c8e2e9e47197ec0f36e2886656366721559f36b76eca9R1-R210) [[2]](diffhunk://#diff-b1d55e82b9c9a25ba02540f4e1d46def7467de866587345f75b63f9ac51c04a6R130-R134) * Added an equivalence test to ensure that DataFusion CASE WHEN results match Vortex's results for the same input data. **Other improvements:** * Refined scalar function pushdown checks and utility logic for expression convertibility. * Minor code quality improvements and error handling in the expression module. ## Testing * Added a comprehensive test to verify equivalence between DataFusion and Vortex results for a `CASE WHEN` expression applied to an Arrow `RecordBatch`. ## Benchmarking * Introduced a new benchmark for various `CASE WHEN` scenarios, including simple, nested, all-true, and all-false cases, in `benches/expr/case_when_bench.rs`, and registered it in `Cargo.toml`. [[1]](diffhunk://#diff-c745f3a7a34d4a4f657c8e2e9e47197ec0f36e2886656366721559f36b76eca9R1-R144) [[2]](diffhunk://#diff-b1d55e82b9c9a25ba02540f4e1d46def7467de866587345f75b63f9ac51c04a6R130-R134) Benchmark | 1K rows | 10K rows | 100K rows -- | -- | -- | -- case_when_simple | 5.4 µs | 8.4 µs | 29.5 µs case_when_without_else | 5.5 µs | 8.7 µs | 29.4 µs case_when_all_true | 4.2 µs | 6.3 µs | 20.7 µs case_when_all_false | 4.3 µs | 6.4 µs | 20.1 µs case_when_nary_3_conditions | 15.1 µs | 25.8 µs | 87.3 µs case_when_nary_equality_lookup (5) | 25.2 µs | 39.7 µs | 125.4 µs case_when_nary_10_conditions | 45.5 µs | 73.1 µs | 257.5 µs --------- Signed-off-by: Luke Kim <80174+lukekim@users.noreply.github.com>

2025-11-07

fix: correctly (de)serialize merge with error duplicate handling Signed-off-by: Daniel King <dan@spiraldb.com>

2025-10-03

Expand expression pushdown to DataFusion (#4398) Started as a demo for @danking to show how to add more pushdown to Datafusion, now it includes some more expression (`IS NULL`, `IS NOT NULL` and `IN LIST`), which only leaves `substr` and dynamic expressions. It also allows for pushing more binary operators for arithmetic operations. `vortex-compact` seems really noise, but clickbench still shows nice improvements in a bunch of queries for the btrblocks compression. --------- Signed-off-by: Adam Gutglick <adam@spiraldb.com>

2025-09-26

More ops Signed-off-by: Adam Gutglick <adam@spiraldb.com>

2025-09-25

More ops Signed-off-by: Adam Gutglick <adam@spiraldb.com>

2025-08-27

Feature: `FixedSizeList` infrastructure (#4385) Adds infrastructure to support `FixedSizeList` including: - Arbitrary implementations - Flatbuffers - Protobuf - Some tests (commit granularity is nice if you want to look at that) --------- Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>

add `FixedSizeList` Protobuf Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>

2025-08-15

feat: Support more datafusion features (#4180) After spending some time today debugging a Datafusion issue with @joseph-isaacs and @robert3005 (ended up not being related to us), I figured I'll use the momentum to add some of the missing features: 1. Support for basic arithmetic expression pushdown (which exists in TPC-DS), including subtraction to `vortex-expr`. 2. Support files with different (but compatible) schemas in the same table. This is currently only supported if a `PhysicalExprAdapterFactory` is provided (either through the `FileScanConfig` or directly). 3. Support hive-style partitioning columns. Both features aren't fully fleshed out and probably require more tests, I want to figure out if we can use datafusion's own test suite to close that gap, they have pretty cool tooling to test SQL. One of the changes it required is making the source and opener more datafusion-y, doing most of the conversion work into Vortex in the actual `FileOpenFuture`. It allows us to utilize more of datafusion's tools for rewriting expressions over specific files. --------- Signed-off-by: Adam Gutglick <adam@spiraldb.com>

feat: More Datafusion support Signed-off-by: Adam Gutglick <adam@spiraldb.com>

2025-08-14

feat: More Datafusion support Signed-off-by: Adam Gutglick <adam@spiraldb.com>

2025-08-11

feat: More Datafusion support Signed-off-by: Adam Gutglick <adam@spiraldb.com>

2025-08-09

feat: More Datafusion support Signed-off-by: Adam Gutglick <adam@spiraldb.com>

2025-08-05

fix: error-prone f16 Scalar serde (#4113) ⏺ Summary of Changes The diff shows comprehensive changes to fix scalar casting bugs and ensure backwards compatibility for serialized scalar values. Here are the key changes: 1. Core Bug Fix: Floating-Point Conversion (vortex-scalar/src/pvalue.rs) - Fixed critical bug where u64 to f64 conversion was using bit reinterpretation (f64::from_bits()) instead of numeric conversion - Changed to use NumCast::from() for proper numeric conversions 2. Type Coercion System (vortex-scalar/src/lib.rs) - Added coerce_value() method in Scalar::new() to handle backwards compatibility - Specifically coerces f16 values stored as u64 (for backwards compatibility with serialized data) - Includes recursive coercion for structs and lists containing f16 fields - Protected against overflow (only coerces u64 to f16 if value ≤ u16::MAX) 3. Protobuf Support for f16 (vortex-proto/proto/scalar.proto) - Added dedicated f16_value field to protobuf schema (field 10) - Updated serialization to use this dedicated field instead of storing f16 as u64 4. Consistent Scalar Construction - Updated all direct Scalar { dtype, value } constructions to use Scalar::new() - Ensures coercion logic is always applied - Changed in: - bool.rs: into_scalar(), bool(), From<bool> - primitive.rs: primitive_value(), From<$T> for Scalar - struct_.rs: field_by_idx(), cast(), struct_() - list.rs: element(), list(), list_empty() 5. Test Coverage (vortex-scalar/src/lib.rs and proto.rs) - Fixed test_canonicalize_scalar_values to create f16 values properly - Added comprehensive coercion tests: - f16 coercion from u64 - No coercion for other type combinations - Struct field coercion - List element coercion - Overflow protection - Added ScalarValue protobuf roundtrip tests: - Direct f16 roundtripping - Value preservation for all primitive types - Handling of type consolidation (u8/u16/u32 → u64, i8/i16/i32 → i64) 6. Key Design Decision - Only f16 from u64 is coerced (minimal approach for backwards compatibility) - Other floating-point types (f32, f64) are NOT coerced - This preserves backwards compatibility while fixing the root cause The changes ensure that: 1. Scalar casting now uses numeric conversion instead of bit reinterpretation 2. Old serialized f16 values (stored as u64) are properly handled 3. All scalar constructions go through the coercion logic 4. Comprehensive tests verify the behavior --------- Signed-off-by: Will Manning <will@willmanning.io>

stop tagging f16 as u64 in proto Signed-off-by: Will Manning <will@willmanning.io>

2025-07-31

feat: Add serialization support for Select expression (#4077) Signed-off-by: Adam Gutglick <adam@spiraldb.com>

2025-07-10

Remove var expression (#3829) Fixes #3671 * Removes the Var expression, leaving instead a `root()` expression for resolving the scope root. * The expression scope can hold context variables, useful for passing in auth tokens for example, but not variables that would impact the return_dtype of the expression. * ScopeDType has therefore been removed, because the dtype of the scope _is_ just the dtype of the root array. * Simplifies some transformation / partitioning logic where vars no longer need to be considered. Signed-off-by: Nicholas Gates <nick@nickgates.com> Co-authored-by: Will Manning <will@spiraldb.com>

2025-07-08

VortexExpr VTables (#3713) Adds the same vtable machinery as arrays and layouts already use. It uses the "Encoding" naming scheme from arrays and layouts. I don't particularly like it, but it's consistent. Open to renames later. Further, adds an expression registry to the Vortex session that will be used for deserialization. Expressions only decide their "options" serialization. So in theory, can support many container formats, not just proto, provided each expression can deserialize their own options format. --------- Signed-off-by: Nicholas Gates <nick@nickgates.com>

2025-07-04

Merge

Merge

chore: SPDX license & copyrights (#3753)

SelectExpr Signed-off-by: Nicholas Gates <nick@nickgates.com>

2025-07-01

Fix ocmpile

Extensible expression serde Signed-off-by: Nicholas Gates <nick@nickgates.com>

2025-06-23

chore: Rename CheckedAdd operator to Add (#3612) Signed-off-by: Robert Kruszewski <github@robertk.io>

feat[vortex-expr]: CheckedAdd operator (#3603) Signed-off-by: Daniel King <dan@spiraldb.com>

chore[vortex-expr]: remove `let` expr (#3604)

2025-06-13

feat[vortex-layout]: add row_id layout reader to support row_id expr queries (#3449) Add the following well defined Scope value ``` dtype("$vx.row_id") = {file_index: u64, file_row_number: u64} ``` This can be used in the scan, and file pruning --------- Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>

2025-06-12

update Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>

2025-06-05

list_contains to use any semantics (#3448) Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>

2025-06-04

wip Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>

feat[vortex-expr]: let expression (#3447) add let x = e in e and var(x) expressions, var("") replaces identity Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> --------- Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>

fix Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>

Merge branch 'develop' into ji/let-expr # Conflicts: # vortex-layout/src/layouts/flat/writer.rs # vortex-proto/src/generated/vortex.expr.rs

add let and var expr Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>

2025-06-03

Adds `list_contains` expression (#3410) Co-authored-by: Andrew Duffy <andrew@a10y.dev>

Merge branch 'develop' into ji/let-expr # Conflicts: # vortex-proto/src/generated/vortex.expr.rs

Add `cast` expression (#3440)

2025-06-02

let expr Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>

This commit will be squashed.

2025-05-30

wip Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>

2025-05-29

This commit will be squashed.

2025-05-16

feat: add is_null expression (#3324)

2025-04-30

Use protobuf to serialize scalars (#3141)

fixes

Use protobuf to serialize scalars

2025-04-29

Add decimal with storage size i8..i64 to vortex (#3161)

add decimal i8..i64

2025-04-28

Use protobuf to serialize scalars

2025-04-24

feat: Decimal types (#3058) Co-authored-by: Robert Kruszewski <github@robertk.io>

use uint32 for decimal precision in proto

2025-04-22

closer

feat: Decimal types

2025-04-21

closer

2025-04-18

feat: Decimal types

2025-04-07

Add all vortex expressions to proto serde (#2907) Add serde + proto expression for like and between.

2025-03-21

feat(java): Expression serialization protobufs (#2744)

2025-03-20

Use proto c++ (#2734) Add marshalling from C++ into Expr proto --------- Co-authored-by: Alexander Droste <alexander.droste@protonmail.com>

2025-03-19

Vortex expr protobuf (#2725) Added a first impl sketch for voretx-expr protobufs. I think in the future we will replace the `Kind kind = 3;` with a more extendable system, maybe proto any or json?

2025-02-07

Improve expression performance (#2273) * Only simplify once in the scanner * Manually implement return_dtype instead of expensive Canonical::Empty (FLUP: we should add `return_nullability()` expr too) * Remove `Field::Index` weirdness. * More consistent APIs for Struct{Scalar,Array,DType}

2025-01-30

chore: remove unused Decimal dtype in proto/fbs (#2156)