🌪️ Vortex

Join the community on Slack! | Documentation | Performance Benchmarks

If you are interested in closer collaboration, please email info@vortex.dev

Overview

Vortex is a next-generation columnar file format and toolkit designed for high-performance data processing. It is the fastest and most extensible format for building data systems backed by object storage. It provides:

Blazing Fast Performance
- 100x faster random access reads (vs. modern Apache Parquet)
- 10-20x faster scans
- 5x faster writes
- Similar compression ratios
- Efficient support for wide tables with zero-copy/zero-parse metadata
Extensible Architecture
- Modeled after Apache DataFusion's extensible approach
- Pluggable encoding system, type system, compression strategy, & layout strategy
- Zero-copy compatibility with Apache Arrow
Open Source, Neutral Governance
- A Linux Foundation (LF AI & Data) Project
- Apache-2.0 Licensed
Integrations
- Arrow, DataFusion, DuckDB, Spark, Pandas, Polars, & more
- Apache Iceberg (coming soon)

🟢 Development Status: Library APIs may change from version to version, but we now consider the file format stable. From release 0.36.0, all future releases of Vortex should maintain backwards compatibility of the file format (i.e., be able to read files written by any earlier version >= 0.36.0).

Key Features

Core Capabilities

Logical Types - Clean separation between logical schema and physical layout
Zero-Copy Arrow Integration - Seamless conversion to/from Apache Arrow arrays
Extensible Encodings - Pluggable physical layouts with built-in optimizations
Cascading Compression - Support for nested encoding schemes
High-Performance Computing - Optimized compute kernels for encoded data
Rich Statistics - Lazy-loaded summary statistics for optimization

Technical Architecture

Logical vs Physical Design

Vortex strictly separates logical and physical concerns:

Logical Layer: Defines data types and schema
Physical Layer: Handles encoding and storage implementation
Built-in Encodings: Compatible with Apache Arrow's memory format
Extension Encodings: Optimized compression schemes (RLE, dictionary, etc.)

Quick Start

Installation

Rust Crate

All features are exported through the main vortex crate.

cargo add vortex

Python Package

uv add vortex-data

Command Line UI (vx)

For browsing the structure of Vortex files, you can use the vx command-line tool.

# Install pre-built binary (fast, recommended)
cargo binstall vortex-tui

# Or build from source
cargo install vortex-tui --locked

# Or run via Python without installing
uvx --from vortex-data vx --help

# Usage
vx browse <file>

Development Setup

Prerequisites (macOS)

# Optional but recommended dependencies
brew install flatbuffers protobuf  # For .fbs and .proto files
brew install duckdb               # For benchmarks

# Install Rust toolchain
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
# or
brew install rustup

# Initialize submodules
git submodule update --init --recursive

# Setup dependencies with uv
uv sync --all-packages

Benchmarking

Use vx-bench to run benchmarks comparing engines (DataFusion, DuckDB) and formats (Parquet, Vortex):

# Install the benchmark orchestrator
uv tool install "bench_orchestrator @ ./bench-orchestrator/"

# Run TPC-H benchmarks
vx-bench run tpch --engine datafusion,duckdb --format parquet,vortex

# Compare results
vx-bench compare --run latest

See bench-orchestrator/README.md for full documentation.

Performance Optimization

For optimal performance, we suggest using MiMalloc:

#[global_allocator]
static GLOBAL_ALLOC: MiMalloc = MiMalloc;

Project Information

License

Licensed under the Apache License, Version 2.0.

Governance

Vortex is an independent open-source project and not controlled by any single company. The Vortex Project is a sub-project of the Linux Foundation Projects. The governance model is documented in CONTRIBUTING.md and is subject to the terms of the Technical Charter.

Contributing

Please do read CONTRIBUTING.md before you contribute.

Reporting Vulnerabilities

If you discover a security vulnerability, please email vuln-report@vortex.dev.

Trademarks

Acknowledgments

The Vortex project benefits enormously from groundbreaking work from the academic & open-source communities.

Research in Vortex

BtrBlocks - Efficient columnar compression
FastLanes & FastLanes on GPU - High-performance integer compression
FSST - Fast random access string compression
ALP & G-ALP - Adaptive lossless floating-point compression
Procella - YouTube's unified data system
Anyblob - High-performance access to object storage
ClickHouse - Fast analytics for everyone
MonetDB/X100 - Hyper-Pipelining Query Execution
Morsel-Driven Parallelism: A NUMA-Aware Query Evaluation Format for the Many-Core Age
The FastLanes File Format - Expression Operators

Vortex in Research

Anyblox - A Framework for Self-Decoding Datasets
F3 - Open-Source Data File Format for the Future