Proto commits in lancedb/lance

These commits are when the Protocol Buffers files have changed: (only the last 100 relevant commits are shown)

2026-07-29

perf(cache)!: use fixed-size cache keys (#7878) ## Summary - Replace heap-allocated, repeatedly hashed logical string keys with one opaque, canonical 16-byte `InternalCacheKey`. - Derive keys from stable type/schema identity plus typed fields using domain-separated BLAKE3, and migrate every in-tree cache-key producer to allocation-free typed encoding. - Keep `CacheBackend` object-safe while simplifying it around one physical key type; preserve codecs, accounting, `clear`, and Moka single-flight behavior. - Add deterministic cache/concurrency contracts, allocation guards, persistent-backend restart coverage, and paired Criterion benchmarks. Closes #7832. ## Stable key format `CACHE_KEY_FORMAT` is `blake3-128-v1`: 1. The root 32-byte namespace is generated with BLAKE3 `derive_key` using a fixed Lance context. 2. Each `with_key_prefix` segment derives a new keyed namespace with explicit domain and length framing. 3. Each entry hashes a stable type ID, author-defined schema ID/version, and tagged logical fields under that namespace. 4. Field tags are defined by a `#[repr(u8)]` enum. Variable-width values are length-framed; integers are fixed-width little-endian; options, variants, sequences, fixed bytes, and variable bytes have distinct one-byte tags. 5. The first 128 bits become the canonical backend key. `InternalCacheKey::{as_bytes,into_bytes,from_bytes}` are the persistence boundary. Changing a key schema version intentionally produces a cold miss. Persistent backends should include `CACHE_KEY_FORMAT` in their physical namespace and allow entries from older formats to age out; there is no runtime legacy-key fallback. The digest is a cache identity, not an authentication or authorization mechanism. Deterministic namespace keys are not secret. A 128-bit digest has approximately 64 bits of generic birthday-collision resistance and 128 bits of targeted preimage resistance. This change does not add a FIPS mode: #7832 selects BLAKE3 and Lance has no existing FIPS configuration surface. ## Intentional backend API break This removes APIs that require retaining logical strings or a secondary inventory: - `CacheBackend::invalidate_prefix` - backend key inventory / `LanceCache::keys` - readable key and prefix accessors - session cache-key inventory methods Use `clear` for explicit invalidation, or derive/version a new namespace when a logical scope changes. Custom backend migration: - Store/copy `InternalCacheKey` directly, or persist `key.into_bytes()` as exactly 16 bytes. - Reconstruct keys with `InternalCacheKey::from_bytes` when needed. - Route serialized values with `CacheCodec::type_id()` instead of inferring value type from a logical key string. - Replace `with_backend_and_prefix` with `with_backend(...).with_key_prefix(...)`. - Remove prefix scans and key-string parsing. The `CacheBackend` trait remains non-generic and object-safe. Existing out-of-tree `CacheKey` / `UnsizedCacheKey` implementations retain a source-compatible default bridge through `key()`. Performance-sensitive implementations should define a stable `CacheKeySchema` and override `write_key`; all current in-tree producers do so. ## Correctness and persistence coverage The proof suite covers: - official and golden BLAKE3 vectors, exact builder output, framing boundaries, endianness, type/schema/namespace separation, and options/variants/sequences; - default sized and unsized string bridges plus schema-version cold misses; - shared strong/weak cache state, expired weak handles, no-cache behavior, custom backends, Moka weights scaled safely above 4 GiB, and contextual type-collision misses/errors; - deterministic single-flight success, error, and owner-cancellation behavior with contenders explicitly parked before release/abort; - zero allocations for complete production-shaped typed page and optional-UUID keys after warm-up; - deletion-file cache identity across distinct storage bases; - a shared serializing backend that retains only bytes and opaque keys across restart and always decodes with the lookup codec; - BTree and IVF restart queries that prove serialized state/partitions are reused, assert vector recall, and perform zero index I/O once non-serializable readers are reconstructed, plus existing FTS/metadata/scalar integration coverage. The removed unstable IVF `cache_key_prefix` protobuf payload is reserved by field number and name. Its codec version is unchanged because protobuf removal is wire-compatible and the new physical-key format already guarantees a cold miss. ## Benchmarks Three independent Criterion passes used `release-with-debug`, 100 samples, Rust 1.97.0, and an AMD Ryzen 9 3900X under x86_64 WSL2. Both paths include the backend's outer hash. Reported ranges compare median time for the fixed path against the benchmark-local implementation of the previous string-key path: - Long production-shaped key preparation: **2.8%–10.3% faster**. - Short isolated key preparation: **139%–156% slower**; this exposes BLAKE3's fixed setup cost instead of hiding it. The motivating long-prefix workload improves. - Strong warmed hits: **7.7%–16.6% faster**. - Weak warmed hits: **1.4%–6.0% faster**. - Bounded rotating inserts with prebuilt values: between **8.9% faster and 3.0% slower** (effectively neutral; median pass was 1.3% faster). - Typed key preparation: **0 allocations** after warm-up. Namespace derivation is benchmarked separately so one-time scope setup is not folded into per-entry preparation. ## Validation Base: `a3c6fce816befb7072505fbe05cc55cd205a171e` Passed after rebasing onto that base and again after review follow-ups: - `cargo fmt --all -- --check` - `CARGO_INCREMENTAL=0 cargo check --workspace --tests --benches --locked` - `CARGO_INCREMENTAL=0 cargo clippy --all --tests --benches -- -D warnings` - `CARGO_INCREMENTAL=0 cargo test --workspace --locked` - `CARGO_INCREMENTAL=0 cargo +1.91.0 check --workspace --tests --benches --locked` - `CARGO_INCREMENTAL=0 cargo check --manifest-path python/Cargo.toml --locked` - `CARGO_INCREMENTAL=0 cargo check --manifest-path java/lance-jni/Cargo.toml --locked` - targeted lance-core cache/allocation and Lance serializing-restart tests on Rust 1.91 - error-path single-flight stress test repeated 1,000 times - three complete, symmetrically hashed paired benchmark passes All three lockfiles contain only the intentional BLAKE3 dependency change (plus the upstream release-version updates already present in the base). ## Interaction with open cache work - #7818 currently relies on readable key inventory and prefix parsing. Cache diagnostics should instead consume aggregate accounting/component metadata that is independent of physical keys; this PR intentionally does not preserve raw key enumeration. - #7828 should expose the key as an opaque 16-byte ABI value and use codec type metadata for serialized-value routing rather than freezing legacy string fields into the ABI. - #7683's registry/configuration model remains applicable, but registered custom backends must adopt the trait migration above. URI/config selection does not need readable physical keys. Made with [Cursor](https://cursor.com) --------- Co-authored-by: Cursor <cursoragent@cursor.com>

2026-07-28

docs(transaction): clarify draft action semantics from self-review Refinements to the Transaction V2 wire draft (protos/transaction/actions.proto), no behavior change: - AddFragment / SetDeletionFile: resolve a contradiction. AddFragment's doc implied a freshly-minted (Local) fragment could take a deletion file via SetDeletionFile, but SetDeletionFile.fragment is a committed-only uint64. Clarify that a new fragment has no deletion vector and deletions arrive in a later operation once the id is committed, and document why SetDeletionFile takes no Ref. - data_change: document the marker on every carrier (previously only on AddFragment), cross-referencing the canonical definition. Spell out its non-obvious meaning on AddIndexSegment / RemoveIndexSegment, where it refers to the indexed data rather than table rows. - AssertUniqueKeys: note that key_fields is authoritative and the embedded filter.field_ids (an artifact of the shared KeyExistenceFilter type) is ignored. - AdjustIndexCoverage: rename bare add / remove fields to add_fragments / remove_fragments for self-documentation and consistency with AddIndexSegment.covered_fragments. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

refactor(transaction): split V2 action protos into transaction/ directory Reorganize the action-based transaction (Transaction V2) wire draft into its own files under protos/transaction/ so the design reads clearly and the shared building blocks have a proper home: - protos/transaction/actions.proto: the V2 vocabulary (Ref, UserOperation, UserAction, Action + action messages), as top-level messages. The design rationale is summarized in an in-tree file header (deltas vs post-images, minting vs reference-stable, Ref/Local resolution, field-level schema, index segments, what stays off the wire) so it stands on its own. - protos/transaction/common.proto: UpdateMap/UpdateMapEntry and KeyExistenceFilter/ExactKeySetFilter/BloomFilter, promoted from nested Transaction messages to top-level so both the legacy operations and the V2 actions can reference them without a circular import. Wire-compatible: field numbers unchanged and none of these types are Any-packed, so the fully-qualified name change is invisible on the wire. - protos/transaction.proto moves to protos/transaction/transaction.proto and keeps only the Transaction envelope, legacy operations, and the user_operation oneof arm. Comments use block style for IDE folding. Rust references to the promoted types are repointed from pb::transaction::X to pb::X (mechanical, compiler-checked); the hand-written dataset::transaction domain types are unaffected. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

feat(transaction): draft action-based transaction wire format (Transaction V2) Draft the full action vocabulary for action-based transactions (Transaction V2) directly in canonical `transaction.proto`, so it can drive the OSS-1529 squash/merge spike and the OSS-757 PMC vote. A `UserOperation` (a new `Transaction.operation` oneof arm, field 116) is an ordered list of `UserAction` steps, each expanding to granular `Action` deltas. Actions record the *change* to the manifest (not a post-image), which is what makes compound commits and branch merge fall out uniformly. Minted identifiers (field/fragment/base ids) carry a `Local` token via a single `Ref { committed | local }` so they relocate on merge/rebase; reference-stable changes key off stable coordinates. Computed conflict footprints and large derivable row-level deltas stay off the wire. Library support is intentionally READ-side fail-closed only: a transaction carrying a `UserOperation` is rejected on load with a clear "not supported" error, and there is no write path, no `apply`, no translation, and no conflict resolution yet. This keeps older writers safe (a concurrent V2 commit in the conflict window aborts an in-flight commit rather than being silently skipped). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

feat(encoding): add range and delta block codecs

Merge remote-tracking branch 'origin/main' into xuanwo/fts-element-document-phase1 # Conflicts: # rust/lance-index/src/scalar/inverted/index.rs # rust/lance/src/dataset/overlay.rs # rust/lance/src/dataset/scanner.rs # rust/lance/src/index/append.rs # rust/lance/src/io/exec/fts.rs

2026-07-27

feat(overlay): gate overlay writes on an explicit dataset setting Data overlay files were permitted on any dataset, which leaves nowhere to record that a table wants overlays before its first one exists. That is what a high-level writer (update, merge_insert) needs to consult when deciding whether to store an update as an overlay. Add `lance.overlays.enabled`, a table config key resolved as: an explicit boolean wins; otherwise a dataset already carrying overlays counts as enabled (so datasets written before the key existed stay writable); otherwise the library default, currently off. Enablement lives in config rather than in feature flag 64 so it survives writers that predate overlays, and so an enabled-but-overlay-free dataset stays open to pre-overlay readers. The flag keeps its existing meaning: overlays are present. Disabling is a two-way door, allowed only while no fragment carries an overlay. Both that check and the converse -- no overlay commit while disabled -- are one invariant enforced in `build_manifest`, which re-runs on conflict rebase, so a disable that races a concurrent `DataOverlay` is rejected rather than silently committed. Surfaces: `WriteParams::enable_overlays` (an `Option<bool>`, since the default is expected to flip and a bare bool could not distinguish "off" from "unspecified"), `Dataset::set_overlays_enabled`, and `Dataset::remove_overlays`, which compacts overlaid fragments and nothing else via a new `CompactionOptions::overlays_only`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

feat(format): promote data overlay files to GA Data overlay files were gated behind a build-profile check plus the `LANCE_ENABLE_UNSTABLE_DATA_OVERLAY_FILES` escape hatch: release builds treated feature flag 64 as unknown and refused any dataset carrying an overlay. Remove the gate so release builds read and write overlays, and rename `FLAG_UNSTABLE_DATA_OVERLAY_FILES` to `FLAG_DATA_OVERLAY_FILES`. `apply_feature_flags` already derives the flag from the manifest's fragments on every commit, so the flag is set by the commit that attaches the first overlay and cleared by the commit that removes the last one. Add tests pinning that lifecycle end to end, covering both ways overlays disappear: deleting the overlaid fragments, and compacting them into base data. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

feat(encoding): add generic block compression

2026-07-24

refactor: rename MemWAL compaction progress (#7957) ## Summary - document generations as SSTable properties rather than MemTable properties - rename MemWAL compaction progress to `CompactedSsTable` / `compacted_sstables` across protobuf, Rust, Python, and Java - use SSTable Compaction terminology and replace the MemWAL overview and shard diagrams ## Validation - `cargo clippy --all --tests --benches -- -D warnings` - `cargo test -p lance compacted_sstables` - Python build, lint, and MemWAL tests (20 passed) - Java and JNI tests (424 passed, 26 skipped) and clippy - `uv run mkdocs build`

2026-07-23

docs(transaction): clarify draft action semantics from self-review Refinements to the Transaction V2 wire draft (protos/transaction/actions.proto), no behavior change: - AddFragment / SetDeletionFile: resolve a contradiction. AddFragment's doc implied a freshly-minted (Local) fragment could take a deletion file via SetDeletionFile, but SetDeletionFile.fragment is a committed-only uint64. Clarify that a new fragment has no deletion vector and deletions arrive in a later operation once the id is committed, and document why SetDeletionFile takes no Ref. - data_change: document the marker on every carrier (previously only on AddFragment), cross-referencing the canonical definition. Spell out its non-obvious meaning on AddIndexSegment / RemoveIndexSegment, where it refers to the indexed data rather than table rows. - AssertUniqueKeys: note that key_fields is authoritative and the embedded filter.field_ids (an artifact of the shared KeyExistenceFilter type) is ignored. - AdjustIndexCoverage: rename bare add / remove fields to add_fragments / remove_fragments for self-documentation and consistency with AddIndexSegment.covered_fragments. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

refactor(transaction): split V2 action protos into transaction/ directory Reorganize the action-based transaction (Transaction V2) wire draft into its own files under protos/transaction/ so the design reads clearly and the shared building blocks have a proper home: - protos/transaction/actions.proto: the V2 vocabulary (Ref, UserOperation, UserAction, Action + action messages), as top-level messages. The design rationale is summarized in an in-tree file header (deltas vs post-images, minting vs reference-stable, Ref/Local resolution, field-level schema, index segments, what stays off the wire) so it stands on its own. - protos/transaction/common.proto: UpdateMap/UpdateMapEntry and KeyExistenceFilter/ExactKeySetFilter/BloomFilter, promoted from nested Transaction messages to top-level so both the legacy operations and the V2 actions can reference them without a circular import. Wire-compatible: field numbers unchanged and none of these types are Any-packed, so the fully-qualified name change is invisible on the wire. - protos/transaction.proto moves to protos/transaction/transaction.proto and keeps only the Transaction envelope, legacy operations, and the user_operation oneof arm. Comments use block style for IDE folding. Rust references to the promoted types are repointed from pb::transaction::X to pb::X (mechanical, compiler-checked); the hand-written dataset::transaction domain types are unaffected. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

refactor(mem-wal): rename flushed MemTable/generation to SSTable (#7943) ## What A flushed MemTable generation always carries a BTree primary-key index and is a persisted, immutable Lance dataset — i.e. an **SSTable** in LSM terms. This renames the internal terminology for that unit. - The persisted-unit noun — `FlushedGeneration` / `flushed_generations` / `flushed MemTable` — becomes `SsTable` / `sstables`. "sstable" already implies flushed, so the qualifier is dropped. - **Kept unchanged:** the flush *verb*, WAL-durability terms (`all_flushed_to_wal`, `rows_flushed`, `unflushed_memtable_bytes`), and the generation *number* concept (`LsmGeneration`, `MergedGeneration`, `current_generation`, on-disk `_gen_{i}`). An SSTable is *identified by* its generation number. ## Surfaces `protos/table.proto` (message `SsTable`, field `sstables` — field numbers preserved), the `lance-table` core types, `rust/lance/src/dataset/mem_wal/` (incl. `LsmDataSource::SsTable`, `SsTableCache`, `open_sstable`, `sstable_cache.rs`), the Python and Java bindings, the mem_wal benches, and `docs/src/format/table/mem_wal.md`. ## Compatibility MemWAL is experimental. Proto field numbers are unchanged (wire-compatible), and `ShardManifest` persists as protobuf, so there is no on-disk change. Experimental binding APIs are renamed directly without deprecation shims.

feat(transaction): draft action-based transaction wire format (Transaction V2) Draft the full action vocabulary for action-based transactions (Transaction V2) directly in canonical `transaction.proto`, so it can drive the OSS-1529 squash/merge spike and the OSS-757 PMC vote. A `UserOperation` (a new `Transaction.operation` oneof arm, field 116) is an ordered list of `UserAction` steps, each expanding to granular `Action` deltas. Actions record the *change* to the manifest (not a post-image), which is what makes compound commits and branch merge fall out uniformly. Minted identifiers (field/fragment/base ids) carry a `Local` token via a single `Ref { committed | local }` so they relocate on merge/rebase; reference-stable changes key off stable coordinates. Computed conflict footprints and large derivable row-level deltas stay off the wire. Library support is intentionally READ-side fail-closed only: a transaction carrying a `UserOperation` is rejected on load with a clear "not supported" error, and there is no write path, no `apply`, no translation, and no conflict resolution yet. This keeps older writers safe (a concurrent V2 commit in the conflict window aborts an in-flight commit rather than being silently skipped). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

2026-07-22

Merge remote-tracking branch 'origin/main' into xuanwo/fts-element-document-phase1 # Conflicts: # python/python/lance/dataset.py # rust/lance/src/dataset/mem_wal/index/fts.rs # rust/lance/src/index/append.rs

2026-07-21

feat(format): read and write sparse structural pages (#7889) This PR replays #7754 unchanged against `main`. #7754 was accidentally merged into `xuanwo/sparse-stack-2-empty-inline-bitpacked` instead of `main`. This PR only corrects that target mistake and introduces no changes beyond the original PR. All design discussion, review history, approvals, and validation are recorded in #7754. --------- Co-authored-by: Weston Pace <weston.pace@gmail.com>

feat(index): write zone map seeds into data file footers during append (#7427) This PR introduces the concept of "index seeds". Indexes can opt-in to planting seeds during ingestion. The seeds are placed into the data files as global buffers. Later, when updating the index to include these data files, we can harvest the seeds instead of scanning the data itself. This is primarily intended to avoid a potentially expensive data scan to update the index. As an example this PR adds index seeds for wide (binary, fixed-size-list, string) columns when creating a zone map index. Now we calculate the min/max/nulls during ingestion, when the data is already present and flowing through the system. Then, when we go to update the index, all we are doing is reading back those counts and adding them to the index (instead of scanning the large column all over again). <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Zone-map indexes can now use stored seed statistics to accelerate incremental updates. * Zone-map configuration now supports optional `rows-per-zone` and seed usage (`use-seeds`) with backward-compatible behavior when omitted. * When appending to existing datasets, the system can persist zone-map seed metadata for eligible indexes. * Added a benchmark to compare incremental update performance with and without seeds. * **Bug Fixes** * Incremental updates automatically fall back to full processing when seed data is unavailable or can’t be used. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>

feat(format): read and write sparse structural pages (#7754) Part of #7750 Depends on https://github.com/lance-format/lance/pull/7752. Wire contract discussion: https://github.com/lance-format/lance/discussions/7631. The validity-polarity text and vote remain a merge gate. This PR freezes the Lance 2.3 SparseLayout wire contract and lands its safe, selective reader together with an explicit writer so the contract is exercised end to end. The reader rejects SparseLayout in pre-2.3 files, validates metadata, domains, semantic sets, descriptors, buffer ranges, compression headers, chunk sums, and complete chunk consumption, caches normalized plans, projects range/take selections through nested layers, reads only intersecting value chunks, and rebuilds no-value selections without value I/O. SparseLayout adds no format-specific size or descriptor-complexity quotas for otherwise representable buffers; callers retain responsibility for resource policy. The writer can emit sparse pages only when field metadata explicitly sets `lance-encoding:structural-encoding=sparse` for a Lance 2.3 file. It does not add automatic selection or change the default encoding policy. Lance 2.3 is unstable, so tests generate pages at runtime and round-trip them; this PR contains no checked-in compatibility fixture or generated test artifact. Coverage includes nullable primitive and struct, list/large-list/map/fixed-size-list, null versus empty lists, both validity polarities, all semantic position/count representations, scan, range, take, deeply nested structures, no-value selections, selective value I/O, malformed pages, large representable buffers/descriptors, and pre-2.3 rejection. Validation: - `cargo fmt --all -- --check` - `cargo check -p lance-encoding -p lance-file --features protoc` - `protoc --descriptor_set_out=/dev/null --proto_path=protos protos/encodings_v2_1.proto protos/file2.proto` - `cargo test -p lance-encoding -p lance-file` (470 + 93 passed; 0 failed) - focused sparse reader/writer and malformed-page tests - `uv run mkdocs build` - `cargo clippy --all --tests --benches -- -D warnings` --------- Co-authored-by: Weston Pace <weston.pace@gmail.com>

feat: add code analyzer for FTS (#7681) (cherry picked from commit 252d81acaeaf7644ac9df99a387b63b869934570)

2026-07-20

fix: preserve FTS document identity after merge

Merge remote-tracking branch 'origin/main' into xuanwo/fts-element-document-phase1 # Conflicts: # protos/index_old.proto # python/python/tests/test_scalar_index.py # python/src/dataset.rs # rust/lance-index/src/scalar/inverted.rs # rust/lance-index/src/scalar/inverted/index.rs # rust/lance-index/src/scalar/inverted/lazy_docset.rs # rust/lance-index/src/scalar/inverted/tokenizer.rs # rust/lance/src/dataset/mem_wal/index.rs # rust/lance/src/dataset/mem_wal/index/fts.rs # rust/lance/src/index/scalar/inverted.rs # rust/lance/src/io/exec/fts.rs

2026-07-19

feat: add code analyzer for FTS (#7681)

2026-07-17

feat(transaction): apply, translate, and conflict-check UpdateBases/DataReplacement/Merge as composite actions Wires the new actions through the existing commit path as a feature-gated Operation::UserOperation variant (opaque wire payload, mirroring the Append+AddIndex slice's ExperimentalUserOperation hook): - apply_user_operation resolves AddBases up front (so same-operation BaseRef::Local references can be minted before the manifest exists), applies ReplaceFragmentColumns (with covering-index invalidation), AddFields, ChangeSchema (with dead-data-file pruning), and RefreshRowVersionMetadata. - actions_from_update_bases/actions_from_data_replacement/actions_from_merge translate the three legacy operations into their action-based equivalent; Merge diffs the prior manifest to recover a pure-column-add or recast delta instead of replacing the fragment list wholesale. - Conflict resolution derives a composite's footprint (added bases / replaced fragment-column pairs / merge-class) from its constituent actions and checks it against every other operation and against another composite, reproducing the legacy UpdateBases/DataReplacement/Merge rules per action rather than treating the whole operation as one opaque unit. Round-trip parity tests assert the composite path produces the same fragments/schema/indices as the legacy build_manifest for each operation (including all-NULL backfill, multi-fragment, and stable-row-id cases), and conflict tests cover composite-vs-legacy and composite-vs-composite. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

feat(transaction): add AddBases/ReplaceFragmentColumns/AddFields/ChangeSchema/RefreshRowVersionMetadata actions Extends the experimental Action enum with the actions needed to decompose the legacy UpdateBases, DataReplacement, and Merge operations: - AddBases: mint new base paths from the manifest's base-id counter. - ReplaceFragmentColumns (+ ColumnReplacement): swap or wholesale-replace a fragment's data files. - AddFields / ChangeSchema: append new fields, or replace the schema wholesale (for column recasts). - RefreshRowVersionMetadata: bump row-version metadata for stable-row-id datasets. - BaseRef: a tagged reference (Committed/Local) so a same-operation ReplaceFragmentColumns/AddFields can land files on a base minted earlier in the same operation, resolved at apply time. No apply/translation/conflict wiring yet; that follows in the next commit. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

fix: group code tokenizer index details

2026-07-15

feat(index): add FTS document granularity

fix(format): refine sparse structural layout contract

2026-07-14

feat(index): support element-document FTS targets

Merge remote-tracking branch 'origin/main' into xuanwo/fts-code-analyzer # Conflicts: # protos/index_old.proto # python/src/dataset.rs # rust/lance-index/src/scalar/inverted/index.rs # rust/lance-index/src/scalar/inverted/tokenizer.rs

fix: clean up analyzer profile persistence

2026-07-13

feat: implement stable logical row addresses for format 2.3

fix(format): remove sparse resource quotas

fix(format): harden sparse structural pages

feat(fts): impact skip data for posting lists (#7602) Builds on the merged configurable posting block size work in #7466. Format discussion: #7606. Store per-block `(freq, doc_len)` impact frontiers alongside compressed posting blocks, plus one level-1 entry per 32 blocks, and drive block-max WAND pruning from them instead of build-time scores that can become stale as index statistics drift. - Both 128- and 256-doc blocks use the same compact impact codec: quantized `u8` document-length norms, delta-encoded frequencies, and an omitted norm byte for the common `+1` delta. - Scorer-specific bounds bake once into cache-shared state; query-local caches reuse the slab without sharing stale bounds across different corpus statistics. - Impact data survives packed prewarm and persistent-cache round trips. Posting-list cache versions are bumped while older versions remain readable. - Packed posting views share both impact-derived state and the block-head cache introduced by #7466. - Malformed or missing impact entries fall back to conservative infinite bounds instead of enabling unsafe WAND skips. - Public posting cache-key struct literals remain source-compatible; impact-bearing entries use an internal namespaced key. - V3 indexes without impacts retain the finite BM25 ceiling, while custom scorers without a declared safe bound fall back to `INFINITY`. The query benchmark below predates this codec follow-up; the V3 scoring bounds are unchanged, but impact size and decode cost need to be remeasured. ## Benchmark Measured before this restack against #7466 using per-branch-tip wheels: 200M-doc V3/256 index, 24 partitions, 1000 warm queries at 8 concurrent requests. | query | #7466 list-max fallback | this PR | |---|---|---| | OR 3w k10 | 0.363s / 22 qps | **0.103s / 76 qps (3.5x)** | | OR 3w k100 | 0.392s / 20 qps | **0.198s / 40 qps (2.0x)** | | AND 3w k10 | 0.547s / 15 qps | **0.115s / 69 qps (4.8x)** | | AND 3w k100 | 0.558s / 14 qps | **0.245s / 33 qps (2.3x)** | ## Validation - `cargo test -p lance-index`: 773 passed, 2 ignored; doctest passed - After the final rebase to current `main`, `cargo test -p lance-index --lib scalar::inverted`: 249 passed - `cargo check --workspace --tests --benches` - `cargo clippy --all --tests --benches -- -D warnings` - `cargo fmt --all -- --check` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **New Features** - Full-text search now optionally uses impact skip data to improve block-max scoring and pruning when index partitions provide it. - Impacts are carried through compressed and packed posting data, with impacts-aware WAND routing and scorer-weighted bound caching. - **Bug Fixes** - Improved decoding/validation of impacts envelopes, including safe behavior for malformed, null, or truncated data. - More robust posting component extraction and cache-key isolation to prevent mixing impact vs non-impact partitions. - **Tests** - Expanded roundtrip, cache isolation, and backward/forward compatibility tests for impact-enabled and legacy postings. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Yang Cen <yang@lancedb.com>

feat(format): read sparse structural pages

feat(fts): impact skip data for posting lists Store per-block (freq, doc_len) impact frontiers alongside 256-doc posting blocks (varint-encoded: 2-3 bytes per pair) plus one level1 entry per 32 blocks, and drive block-max WAND pruning from them instead of build-time scores that go stale as index stats drift: - Bounds bake once per cached list into an Arc-shared slab (max doc weight per entry plus the list-wide max); per-query clones reuse it, so query time pays one multiply per bound instead of frontier rescans. - Entry doc_up_tos decode once at construction. - Lagging iterators park in the WAND tail under the data-driven global bound (query_weight x baked list max) instead of INFINITY. 128-doc-block indexes keep fixed-width u32 impact entries.

feat(fts)!: add configurable posting block size (#7466) ## Feature Linear: [OSS-1344](https://linear.app/lancedb/issue/OSS-1344/make-fts-index-block-size-configurable) ### What is the new feature? FTS inverted index creation now accepts a `block_size` parameter for compressed posting blocks. Supported values are `128` and `256`. ### Why do we need this feature? The posting block size was previously fixed at `128`, which made the block-max granularity impossible to tune for different datasets and query profiles. ### How does it work? - Adds `block_size` to `InvertedIndexParams`, protobuf details, posting-list schema metadata, and cache headers. - Uses `128` as the default for newly created indexes. - Treats older serialized params, schema metadata, and cache entries that omit `block_size` as legacy `128`. - Rejects unsupported values, including `512`, with a clear validation error. - Uses Lance-owned `BitPacker4x` for physical 128-value posting blocks and `BitPacker8x` for physical 256-value posting blocks. - Marks `block_size=256` as experimental in public API docs because it may introduce breaking changes. - Keeps position-stream packing on the legacy 128-value block format. - Keeps downgrade compatibility tests on explicit legacy `block_size=128`, since older wheels cannot read current-created physical 256 FTS posting blocks. - Threads the configured block size through FTS build, read, iterator, WAND, cache, and MemWAL flush paths. - Exposes the parameter in Python and Java FTS index creation APIs, with docs and focused tests. ## Validation - `cargo fmt --all` - `cargo fmt --all --check` - `git diff --check` - `CARGO_TARGET_DIR=/tmp/lance-target-a479-no512 cargo test -p lance-index block_size -- --nocapture` - `CARGO_TARGET_DIR=/tmp/lance-target-a479-no512 cargo clippy -p lance-index --tests -- -D warnings` - `uv run make build` from `python/` - `uv run pytest python/tests/test_scalar_index.py::test_create_scalar_index_fts_block_size` from `python/` - `uv run ruff format --check python/tests/test_scalar_index.py python/lance/dataset.py` from `python/` - `uv run ruff check python/tests/test_scalar_index.py python/lance/dataset.py` from `python/` - `CARGO_TARGET_DIR=/tmp/lance-target-a479-merge-main cargo test -p lance-index block_size -- --nocapture` - `CARGO_TARGET_DIR=/tmp/lance-target-a479-merge-main cargo test -p lance-index test_256_posting_block_uses_single_physical_bitpack_chunk -- --nocapture` - `CARGO_TARGET_DIR=/tmp/lance-target-a479-merge-main cargo test -p lance-bitpacking` - `CARGO_TARGET_DIR=/tmp/lance-target-a479-merge-main cargo clippy -p lance-bitpacking -p lance-index --tests -- -D warnings` - `uv run ruff format --check python/tests/compat/test_scalar_indices.py` from `python/` - `uv run ruff check python/tests/compat/test_scalar_indices.py` from `python/` - `uv run pytest --run-compat -vvv -s python/tests/compat/test_scalar_indices.py::test_FtsIndex_downgrade --durations=30` from `python/` - `CARGO_TARGET_DIR=/tmp/lance-a479-target cargo test -p lance-index test_new_training_request_defaults_missing_block_size_to_128` - `CARGO_TARGET_DIR=/tmp/lance-a479-target cargo test -p lance-index block_size` - `uv run ruff format --check python/lance/dataset.py` from `python/` - `uv run ruff check python/lance/dataset.py` from `python/` Not run locally: Java focused test / spotless check, because this machine has no Java Runtime installed (`Unable to locate a Java Runtime`). --- ## Update: all V3 breaking changes consolidated here Per review direction, every breaking change for the 256-doc block format now lands in this single PR (the follow-up stack #7602/#7603/#7604/#7624/#7625/#7629 carries none). On top of the configurable block size and PFOR frequency encoding, this PR now also includes: - **Quantized doc-length scoring (Lucene norm semantics), 256-doc blocks only.** BM25 doc lengths are quantized to a SmallFloat-style byte code (4 mantissa bits: 0-7 exact, <= 6.25% relative error, decode = bucket floor). The byte-norm slab bakes lazily per loaded DocSet and quarters the doc-length bytes scoring pulls through the cache (200M docs: 800MB -> 200MB). 128-block indexes keep exact-length scoring bit-for-bit. Measured top-k overlap vs exact scoring on the (score-clustered, synthetic) mmlb corpus: 98.1% mean for phrase, 89.7% for 3-word AND; corpora with more score spread shift less. - **256-doc posting blocks drop the leading block-max-score f32** (~1.5G on a 200M-doc index; 131G -> 130G). Block layout: `[first_doc u32][doc num_bits u8][docs][pfor freqs]`; `posting_block_score_prefix_len(block_size)` keys every reader/writer. The impact skip data from the stacked #7602 supplies a tighter per-block bound; until it lands, 256-block block-max pruning falls back to the (valid, looser) list-level max score. **BREAKING:** 256-doc-block (v3) indexes must be rebuilt; v3 is unreleased so no migration is provided. BM25 scores on v3 differ from exact-length BM25 by the norm quantization, matching Lucene's norm semantics. The format discussion #7606 documents the final layout and scoring semantics. Additional validation for this update: bulk-vs-classic A/B under quantized scoring is score-identical (both paths quantize identically); the full stack's warm benchmarks vs Lucene 10.4 on mmlb-200m: OR k10 0.0249s/318qps and OR k100 0.0467s/170qps (both ahead of Lucene sliced), AND k10 0.0443s (1.29x), AND k100 0.0883s (1.94x). --- ## Standalone results vs main (per-branch-tip wheels) **Legacy (128) read-path parity.** Threading a runtime `block_size` through `PostingIterator` initially replaced the compile-time `BLOCK_SIZE` division (a shift) with real `div` instructions in the `doc()`/`next()` hot loops, measured as +11-14% on 3-word OR against the 200M legacy index (`PostingIterator::next` grew from 16.5% to 23.6% of the profile). Block sizes are validated powers of two, so the iterator now derives block indices with `trailing_zeros` shifts and masks; after that fix the legacy path is at parity with main: 3-word OR k10 0.131s (main 0.132-0.134s), k100 0.255-0.256s (main 0.256-0.257s), single-term 0.025s (main 0.027s) across 3 warm passes on the 200M legacy index, 400G cache. **block_size=256 index size** (5M-doc controlled build, same wheel, `with_position=false`): postings shrink **3.33 GiB → 2.62 GiB (−21%)** from PFOR frequencies + no per-block max-score prefix + half the block headers. Index build 192s → 210s (+10%, PFOR encode cost). Top-10 overlap vs the 128 exact-length scoring on 10 3-word OR queries: 95% mean (5/10 identical sets; the rest differ by 1-2 near-tie docs, from the quantized-norm scoring). **Query wins for 256 land in the stacked PRs.** A 256 index without impact skip data prunes on the (valid, looser) list-level max and is *slower* than 128 — e.g. classic AND k10 0.547s until #7602's impacts restore block-granular bounds (0.115s), and #7603/#7604/#7624/#7625/#7629 take the same index to 0.025s OR k10 / 0.045s AND k10. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added configurable FTS posting `block_size` (128/256) to scalar index creation, including updated examples and APIs. * Enabled FTS format version 3 (`v3`) for `block_size=256`, with quantized doc-length scoring for v3. * **Bug Fixes** * Enforced `block_size`/`format_version` compatibility (invalid combinations now error). * Persisted and restored FTS metadata for format version and posting block size, with legacy indexes defaulting to `128`. * **Documentation** * Updated full-text-search and quickstart guides and parameter docs for `block_size`, defaults, accepted values, and the experimental `256`/`v3` behavior. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Yang Cen <yang@lancedb.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

feat: add stable logical row addresses for format 2.3

2026-07-10

docs: specify data overlay files for the table format (#7381) Adds a specification for **data overlay files**: small files attached to a fragment that supply new values for a subset of `(row offset, field)` cells without rewriting the base data files. They make cell-level updates cheap when only a small fraction of rows and/or columns change. This PR is **spec + proto only** — no read/write implementation yet. It is also explicitly *experimental*. The released libraries will not produce tables with this feature enabled. Once the implementation is done in the library, we will vote on the final design before releasing. This is similar to how we have done file format updates. ## Changes - **`protos/table.proto`** - Rework `DataOverlayFile`: a `oneof coverage { bytes shared_offset_bitmap | FieldCoverage field_coverage }` to support both dense (rectangular) and sparse overlays; add the `FieldCoverage` message. - Rename `read_version` → `committed_version` (`uint64`), with effective/commit-stamped semantics so overlay-vs-index ordering is correct. - Drop the in-file offset key column in favor of rank-based addressing off the coverage bitmap. - Document reader feature flag `64` (and previously-undocumented `16`/`32`). - **`docs/src/format/table/data_overlay_file.md`** (new): full specification — coverage/resolution, deletion precedence, NULL-override, layout + rank addressing, dense vs. sparse, versioning, field-aware index exclusion with flat re-evaluation, the correctness invariant, both compaction modes, row lineage, a worked example (write → read → index query → sparse write → read → compaction), and a guidance stub with open questions. - **`docs/src/format/table/index.md`**: concise overview + link to the new spec (replacing the earlier inline sketch). ## Out of scope / follow-ups - Write transaction shape (new `Operation` variant in `transaction.proto` + Rust). - Writer support for unequal-length columns (needed for single-file sparse overlays). - Coverage bitmap external spill for very large coverage. - Per-fragment vs. per-table overlays / LSM analogy (open question in the doc). 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added documentation for experimental Data Overlay Files, including storage, versioning, querying, compaction, and transaction behavior. * Added format and transaction schema support for describing data overlays. * Added navigation links to the new documentation. * **Bug Fixes** * Datasets using unsupported data overlays are now explicitly rejected instead of risking stale results. * **Documentation** * Expanded guidance on invalidated index results and overlay-related filtering scenarios. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: Weston Pace <weston.pace@gmail.com>

feat: store manifests in a single Lance file

fix(fts): address review feedback

fix: avoid persisting fts analyzer profiles

2026-07-09

fix: address code analyzer review feedback

2026-07-08

fix: avoid persisting unrelated FTS stop words

feat: add code analyzer for FTS

2026-07-07

docs: relocate data overlay conflict and index rules to canonical docs Move the DataOverlay conflict-resolution rules out of the transaction.proto comment into transaction.md as a DataOverlay operation + Compatibility section, matching the pattern used for every other operation. Consolidate the reader-side overlay handling in the index doc and make re-evaluation explicit. Mark the feature experimental, drop the open-questions section, and cross-link the three docs. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

feat(transaction): add opaque experimental-operation wire hook Add an ExperimentalUserOperation message + oneof arm (field 115) to the canonical transaction.proto. The action list is carried as an opaque `bytes` payload (a serialized experimental UserOperation) so the PMC-voted schema stays free of the unstable action structure. Builds that cannot interpret it reject the transaction; a real Operation variant is wired up in the next commit. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

feat(transaction): add NewFragment placeholder + AddIndex action Extend the experimental action-transaction types for the first compound-commit slice. `AddFragments` now carries `NewFragment`s, each with an operation-local `local_id` placeholder so a later action can reference a fragment before its real id is assigned. Adds the `AddIndex` action, which registers an index segment and expresses coverage as the union of already-committed fragment ids and same-op placeholders. All behind the non-default `unstable-action-transactions` feature. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

feat(transaction): experimental action-based transaction skeleton Skeleton for the Action-based Transactions milestone (discussion #5960), gated behind the non-default `unstable-action-transactions` Cargo feature so it is absent from released artifacts. Stacked on the general experimental-feature mechanism (#7646): rather than claiming a dedicated feature-flag bit, action-based transactions are the first consumer of FLAG_EXPERIMENTAL. A dataset declares the "action-transactions" experimental feature (registered in `known_experimental_features` under the Cargo feature), which sets the bit so libraries without it fail closed. - protos/transaction_experimental.proto: UserOperation / UserAction / Action (AddFragments only), compiled only under the feature. - lance_table::transaction: UserOperation/UserAction/Action/AddFragments Rust types with protobuf conversions and a roundtrip test; FEATURE_NAME constant. - Register "action-transactions" in the experimental-feature registry, with admission tests for both default (rejected) and feature-on (recognized). The wire format carries no compatibility guarantee until finalized by PMC vote, at which point it graduates to its own feature-flag bit. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

perf(lance): honor unordered filtered reads

feat: add sparse validity polarity

2026-07-06

feat(format): general experimental-feature mechanism Proposes and implements a general mechanism for shipping experimental format features without burning a permanent feature-flag bit per experiment. Policy (design doc) and the proto/format change are together so they can be reviewed and voted on as one. Mechanism: - Reserve one persisted bit, FLAG_EXPERIMENTAL (1 << 6, reusing the old first FLAG_UNKNOWN bit), meaning "this dataset uses experimental feature(s)". It is the fail-closed anchor: pre-mechanism libraries reject the dataset on the bit alone. - Carry feature identities in free-form manifest string lists (experimental_reader_features / experimental_writer_features). New libraries admit a dataset iff they understand every declared name. - A compile-time registry (known_experimental_features) lists the experiments a build understands, gated by each experiment's Cargo feature; a default build understands none and so rejects any experimental dataset. This keeps experiments free-flowing (unbounded string namespace, mint/abandon at will) while the bitmap stays conservative (a bit is spent only at graduation). Design, graduation/lifecycle, alternatives, and prior art (Delta Lake table features) in rust/lance-table/design/experimental_feature_flags.md. can_read_dataset / can_write_dataset now take the declared experimental feature list alongside the flags (two internal callers updated). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

feat: add DataOverlay transaction operation Add the `DataOverlay` operation (and `DataOverlayGroup`) to attach overlay files to fragments without rewriting their base data. Mirrors the `DataReplacement` batch shape, appends to each fragment's `overlays` list, and documents permissive conflict semantics: concurrent overlays, appends, deletes, and column rewrites are compatible; row-rewrites, compaction, and overlay->base folds conflict. committed_version is left 0 by the writer and stamped at commit time. Proto only — Rust/Python bindings deferred. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

docs: specify data overlay files for the table format Add a specification for data overlay files: small files attached to a fragment that supply new values for a subset of (row offset, field) cells without rewriting the base data files, for cheap cell-level updates. - protos/table.proto: rework DataOverlayFile with a dense/sparse coverage oneof (shared_offset_bitmap vs new FieldCoverage), rename read_version to committed_version (effective, commit-stamped), and document rank-based addressing with no offset column. Document reader feature flag 64. - docs: add data_overlay_file.md (full spec, worked example, guidance stub) and link it from the table format overview. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

feat: add DataOverlay transaction operation Add the `DataOverlay` operation (and `DataOverlayGroup`) to attach overlay files to fragments without rewriting their base data. Mirrors the `DataReplacement` batch shape, appends to each fragment's `overlays` list, and documents permissive conflict semantics: concurrent overlays, appends, deletes, and column rewrites are compatible; row-rewrites, compaction, and overlay->base folds conflict. committed_version is left 0 by the writer and stamped at commit time. Proto only — Rust/Python bindings deferred. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

docs: specify data overlay files for the table format Add a specification for data overlay files: small files attached to a fragment that supply new values for a subset of (row offset, field) cells without rewriting the base data files, for cheap cell-level updates. - protos/table.proto: rework DataOverlayFile with a dense/sparse coverage oneof (shared_offset_bitmap vs new FieldCoverage), rename read_version to committed_version (effective, commit-stamped), and document rank-based addressing with no offset column. Document reader feature flag 64. - docs: add data_overlay_file.md (full spec, worked example, guidance stub) and link it from the table format overview. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Merge branch 'main' into yang/oss-1344-make-fts-index-block-size-configurable Reconciles #7600's tokens_only DocSet cache with this branch's quantized_scoring flag: the cached num_tokens-only DocSet is built with quantized scoring applied inside the OnceCell init, matching how ensure_loaded stamps the full DocSet.

feat: add RLE v2 run length widths (#7376) ## Summary Adds RLE v2 run-length widths so newly created datasets can write RLE pages with `u16` or `u32` run lengths instead of splitting every run at 255 values. The capability is recorded as a reader feature flag and is only enabled when a new dataset is created with `WriteParams::enable_rle_v2`; existing unflagged datasets reject attempts to turn it on mid-stream. Closes #7327. ## Benchmark Ran on `xuanwo-lance-lazy-metadata-bench` with a #6941-style sorted low-cardinality `asset_id` workload. | workload | Lance default | Lance RLE2 | reduction | |---|---:|---:|---:| | `150M rows / 5k assets / random5 value` | `167.36 MiB` | `164.57 MiB` | `1.67%` | | `150M rows / 5k assets / by-asset5 value` | `7.62 MiB` | `2.03 MiB` | `73.34%` | The first row keeps the random low-cardinality value column from the issue-like workload, which dominates total size. The second row isolates the long-run case RLE2 targets. ## Validation Validated with focused RLE2 tests and full Rust clippy before publishing.

2026-07-04

feat: use semantic sparse structural sets

feat(fts): impact skip data for posting lists Store per-block (freq, doc_len) impact frontiers alongside 256-doc posting blocks (varint-encoded: 2-3 bytes per pair) plus one level1 entry per 32 blocks, and drive block-max WAND pruning from them instead of build-time scores that go stale as index stats drift: - Bounds bake once per cached list into an Arc-shared slab (max doc weight per entry plus the list-wide max); per-query clones reuse it, so query time pays one multiply per bound instead of frontier rescans. - Entry doc_up_tos decode once at construction. - Lagging iterators park in the WAND tail under the data-driven global bound (query_weight x baked list max) instead of INFINITY. 128-doc-block indexes keep fixed-width u32 impact entries.

feat: add sparse structural encoding

2026-07-03

refactor: gate RLE v2 on file version 2.3

feat(fts): impact skip data for posting lists Store per-block (freq, doc_len) impact frontiers alongside 256-doc posting blocks (varint-encoded: 2-3 bytes per pair) plus one level1 entry per 32 blocks, and drive block-max WAND pruning from them instead of build-time scores that go stale as index stats drift: - Bounds bake once per cached list into an Arc-shared slab (max doc weight per entry plus the list-wide max); per-query clones reuse it, so query time pays one multiply per bound instead of frontier rescans. - Entry doc_up_tos decode once at construction. - Lagging iterators park in the WAND tail under the data-driven global bound (query_weight x baked list max) instead of INFINITY. 128-doc-block indexes keep fixed-width u32 impact entries.

2026-07-02

feat(index): add impact skip data for fts

2026-07-01

perf(lance): honor unordered filtered reads

2026-06-26

Merge remote-tracking branch 'origin/main' into yang/oss-1344-make-fts-index-block-size-configurable

feat(bitpacking): add owned bitpacking codecs

feat(mem-wal): add ShardWriter::abort + Sealed manifest fence for drop-table (#7361) ## What The mem-wal primitives sophon's drop-table two-phase commit needs: - **`ShardWriter::abort(&self)`** — shut down the background flush tasks (`task_executor.shutdown_all()`) *without* flushing, discarding buffered memtable state. Unlike `close(self)` it takes `&self` (so it's callable through the `Arc<ShardWriter>` callers hold) and does no object-store IO. The caller must quiesce writes first (documented). Idempotent. Acked data is **not** lost — it's durable in the WAL log and replays on the next claim, which is what makes the drop's prepare phase reversible. - **`ShardStatus { Active | Sealed }` on `ShardManifest`** — a durable, reversible lifecycle marker (proto + struct + serde). `claim_epoch` refuses a `Sealed` manifest with a **distinguishable** error instead of minting a new epoch, so a shard mid-drop can't be re-claimed — even by a caller that skips its own status check — and a reader can tell an in-doubt drop apart from an ordinary epoch fence. Set/cleared through the existing epoch-guarded `commit_update` CAS; carried across claims via `..base`, so only the genuinely-fresh constructions default it to `Active`. ## Why A WAL-enabled table's drop spans two durable resources — the owning pod's fresh-tier state and the catalog/object-store data — so sophon's teardown is a two-phase commit. Before the dataset directory is removed, the owning pod must: - `abort` the writer so its background flush task can't re-create `_mem_wal/` under the just-deleted directory (a graceful `close()` would flush it back), and - durably mark the shard `Sealed` so the drop is in-doubt across a pod crash or a Maglev rehome — which the in-memory fence flag cannot survive. The seal is reversible (rollback clears it back to `Active`), making the prepare phase abortable without data loss. Both build on existing machinery (`shutdown_all`, the manifest CAS) — thin exposure, not new infrastructure. > **Changed since first draft:** this PR previously also added `Session::invalidate_dataset`; it has been **dropped**. Base-table read freshness rides the recreate's new object-store `e_tag` (the QN→PE and lance metadata caches are `e_tag`-keyed and miss the stale entry), so cache invalidation isn't load-bearing — only `abort` and the `Sealed` marker remain. ## Tests - `test_abort_discards_without_flushing_and_is_idempotent` — `abort` leaves no new L0 generation (contrast with `close`), idempotent on a second call. - `test_claim_epoch_refuses_sealed_manifest` — a `Sealed` manifest is refused with the distinguishable error and left untouched (no epoch minted); rolling the status back to `Active` makes the shard claimable again (reversibility). 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

2026-06-25

feat(fts): add configurable posting block size

2026-06-24

refactor!: rename FMIndexIndexDetails to FMIndexDetails (#7397) I'm not sure if the original naming was intentional or not. However, it required a special case in `get_plugin_name_from_details_name` to rename `fmindex` to `fm` so I'm guessing it was accidental? We are trying to convert indexes to be "generic plugins" and this means we cannot have special cases lying around. An index's short-name is the name of the type URL minus the suffix `IndexDetails`. So if we want `fm` then it should be `FmIndexDetails` (which this PR implements). If we want `fmindex` then it should be `FmIndexIndexDetails` (which it had before). If we want both then we should invent some kind of formal alias mechanism where an index plugin can register potential aliases. However, I think it'd be simplest to avoid that. This change would be a breaking change to any existing FM indexes! That index type has not (AFAIK) been formally released yet so I think this is ok. However, if this misses the 8.0.0 release then we will probably need to find a different way (and possibly forever be locked into carrying around this special case). --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>

refactor!: rename FMIndexIndexDetails to FMIndexDetails (#7397) I'm not sure if the original naming was intentional or not. However, it required a special case in `get_plugin_name_from_details_name` to rename `fmindex` to `fm` so I'm guessing it was accidental? We are trying to convert indexes to be "generic plugins" and this means we cannot have special cases lying around. An index's short-name is the name of the type URL minus the suffix `IndexDetails`. So if we want `fm` then it should be `FmIndexDetails` (which this PR implements). If we want `fmindex` then it should be `FmIndexIndexDetails` (which it had before). If we want both then we should invent some kind of formal alias mechanism where an index plugin can register potential aliases. However, I think it'd be simplest to avoid that. This change would be a breaking change to any existing FM indexes! That index type has not (AFAIK) been formally released yet so I think this is ok. However, if this misses the 8.0.0 release then we will probably need to find a different way (and possibly forever be locked into carrying around this special case). --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>

2026-06-23

docs: specify data overlay files for the table format Add a specification for data overlay files: small files attached to a fragment that supply new values for a subset of (row offset, field) cells without rewriting the base data files, for cheap cell-level updates. - protos/table.proto: rework DataOverlayFile with a dense/sparse coverage oneof (shared_offset_bitmap vs new FieldCoverage), rename read_version to committed_version (effective, commit-stamped), and document rank-based addressing with no offset column. Document reader feature flag 64. - docs: add data_overlay_file.md (full spec, worked example, guidance stub) and link it from the table format overview. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

feat: add DataOverlay transaction operation Add the `DataOverlay` operation (and `DataOverlayGroup`) to attach overlay files to fragments without rewriting their base data. Mirrors the `DataReplacement` batch shape, appends to each fragment's `overlays` list, and documents permissive conflict semantics: concurrent overlays, appends, deletes, and column rewrites are compatible; row-rewrites, compaction, and overlay->base folds conflict. committed_version is left 0 by the writer and stamped at commit time. Proto only — Rust/Python bindings deferred. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

2026-06-19

feat: add RLE v2 run length widths

2026-06-17

feat: stabilize cache codec with a versioned envelope (#7163) Implements #7160. Cache entries (FTS posting lists, scalar/vector index state) were serialized with an ad-hoc, unversioned format only safe to read in the same process that wrote it. This stabilizes the format so entries can live in a **node-agnostic, restart-surviving** cache backend. ## Wire format Each entry is an envelope followed by a body: ```text [magic "LCE1"][envelope_version: u8][type_id][type_version: u32] # envelope <body: optional protobuf header, then sections in a fixed, version-keyed order> ``` Body sections, each self-delimiting: ```text HEADER : [len: u32][protobuf bytes] ARROW_IPC : [pad to 64B][self-delimiting IPC stream] RAW_BLOB : [len: u64][bytes] ``` ## Why this shape - **The envelope is hand-framed, not protobuf.** It's the most stability-critical part: it must parse robustly against *any* bytes (including old, pre-stabilization blobs) and never change shape. The magic is chosen so no prior blob can collide with it. - **Decode returns `Hit`/`Miss`, never a hard error.** Wrong/absent magic, an unknown envelope version, a `type_id` mismatch, a future `type_version`, or a body decode failure all become `Miss` → recompute. Old, foreign, or corrupt bytes self-heal with **zero migration code**. - **Bodies use protobuf headers.** Field-number evolution lets us add fields without a format break; only changes protobuf can't express transparently (reordering sections, changing a raw-blob encoding) bump `type_version`, which the reader branches on. - **Arrow IPC sections are 64-byte aligned** so concatenated sections decode zero-copy instead of a realigning `memcpy` on every read — this guards the FTS WAND hot path. - **`RAW_BLOB` is reserved for payloads with their own portable, self-describing encoding** (roaring bitmaps, the shared position stream). A codec with no scalar metadata (e.g. bitmap) simply omits the header — sections are positional, so nothing is written for an absent header. ## Scope All cache codecs migrated: FTS posting lists (compressed/plain/positions + groups), scalar indices (BTree/Bitmap/Flat/LabelList/RowAddrTreeMap), and the five IVF quantizer partitions + IVF state. The cache protos live in `lance-index/protos-cache/cache.proto` (`package lance.index.cache`) — they describe *library serialization*, not the on-disk format spec. ## Tests Envelope round-trip and every miss path; per-codec round-trip + through-envelope zero-copy alignment (incl. RabitQ Matrix rotation, multi-batch SQ, nested bitmap in a label-list entry); additive proto-field compat; existing IVF build+search suites pass through the migrated path. Closes #7160. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

2026-06-10

feat(vector)!: add approx mode for RaBitQ search (#7179) ## Feature This PR adds a public `approx_mode` setting for vector search with `fast`, `normal`, and `accurate` values. The public API avoids exposing RaBitQ/HACC terminology while still allowing callers to choose the speed/accuracy tradeoff when the backing index supports it. ## Implementation - Adds `ApproxMode` to vector queries and threads it through Rust scanner, ANN proto serialization, Python query parsing, and FlatIndex distance calculator options. - Implements RaBitQ behavior for each mode: - `fast`: force 1-bit query-time scoring for RaBitQ. - `normal`: preserve the existing search path. - `accurate`: use a 16-bit LUT accumulation path for the binary RaBitQ estimator. - Extends query scratch with a wider accumulation buffer while keeping the existing `QueryScratchCapacity::new(...)` API compatible. - Adds Python API support via `approx_mode="fast" | "normal" | "accurate"` and rejects invalid values. ## Benchmark IVF_RQ on dbpedia, `num_bits=5`, no `refine_factor`. The plot shows `approx_mode=fast`, `normal`, and `accurate` as separate curves.  ## Breaking Change BREAKING CHANGE: The ANN protobuf schema now includes `VectorApproxMode approx_mode` on vector query serialization. Consumers that regenerate or explicitly match Lance's serialized ANN query proto should update to the new schema. ## Validation - `cargo fmt --all` - `cargo test -p lance-linalg simd::dist_table` - `cargo test -p lance-index vector::bq::storage` - `cargo test -p lance-index vector::storage::tests` - `cargo test -p lance --features substrait test_query_roundtrip` - `cargo test -p lance test_knn_approx_mode_defaults_and_setter` - `cargo check --workspace --tests --benches` - `cd python && make install` - `cd python && uv run pytest python/tests/test_vector_index.py::test_vector_index_with_approx_mode python/tests/test_vector_index.py::test_vector_index_invalid_approx_mode` - `cd python && uv run pytest python/tests/test_vector_index.py::test_create_ivf_rq_multi_bit_searches_l2_and_cosine` - `git diff --check`

2026-06-04

feat(index): implement FM-Index scalar index for exact substring search (#7026) ## Motivation Enable exact substring search at scale for AI pretraining data decontamination — detecting benchmark contamination in trillion-row text corpora, following the [Infini-gram Mini paper](https://arxiv.org/abs/2506.12229). ## Summary - Implement FM-Index following the Infini-gram Mini paper architecture for exact substring search - Huffman-shaped wavelet tree for entropy-compressed BWT rank queries (~0.26N bytes) - Sampled suffix array (D=32) with LF-mapping locate for document resolution (~0.25N bytes) - Partitioned index (10K docs/partition) with blocked storage (32KB blocks) and lazy loading - Wire up `IndexType::FMIndex` in Lance's `create_index` and query paths (`contains()` filter) - Index size ~0.95x of text (paper claims 0.44x; gap is Lance row overhead per block) ## Benchmark (100K gitlake source code files, 1.59 GB text) | Metric | FM-Index | N-Gram | |--------|----------|--------| | Index size | 1,513 MB (0.95x) | 84 MB (0.05x) | | Build time | 132s | 9s | | Short queries (e.g. `fn `) | 9034ms/q | 448ms/q | | Medium queries (e.g. `fn main()`) | **29ms/q** | 480ms/q | | Long queries (~80 chars) | **34ms/q** | 206ms/q | FM-Index is 17x faster than N-Gram on medium queries and returns exact results (N-Gram returns approximate candidates needing recheck). N-Gram cannot find queries shorter than 3 characters (e.g. `fn ` returns 0). ## Test plan - [x] 9 unit tests covering search, locate, wavelet access, serialization, multi-document - [x] End-to-end benchmark through Lance dataset API (`dataset.create_index`, `dataset.count_rows(filter)`) - [x] Verified correct match counts against full-scan baseline on real source code 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Beinan Wang <beinanwang@microsoft.com>

2026-05-20

feat: implement vector index details (#6099) Cache vector index configuration within the index metadata, such as the distance type and build parameters. Previously, to determine things like the distance type or index type of a vector index, the index file itself had to be opened. This PR stores that information in `VectorIndexDetails` within the manifest's `index_details` field, which is fetched and cached eagerly when loading the manifest. Old indexes have this field left blank. When blank, the details are extracted from the index files and cached. This migration happens on the first write with a new library version. ## What's stored in VectorIndexDetails **Core build parameters** (typed fields — required for any runtime to build the index): - `metric_type` - `target_partition_size` (IVF) - `hnsw_index_config` — `max_connections`, `construction_ef`, `max_level` (HNSW) - `compression` — PQ/SQ/RQ/flat, including `num_bits`, `num_sub_vectors`, `rotation_type` **Runtime hints** (`map<string, string> runtime_hints`): Optional build preferences that don't affect index structure. Stored so a background rebuild process can reproduce the original configuration. Runtimes that don't recognize a key must silently ignore it. Only non-default values are written. Keys use reverse-DNS namespacing: `lance.*` for core Lance hints, other prefixes for runtime-specific hints (e.g., `lancedb.accelerator` for GPU acceleration in LanceDB Enterprise). Current `lance.*` hints: `lance.ivf.max_iters`, `lance.ivf.sample_rate`, `lance.ivf.shuffle_partition_batches`, `lance.ivf.shuffle_partition_concurrency`, `lance.pq.max_iters`, `lance.pq.sample_rate`, `lance.pq.kmeans_redos`, `lance.sq.sample_rate`, `lance.hnsw.prefetch_distance`, `lance.skip_transpose`. Also adds `apply_runtime_hints()` to read hints back into build params for future rebuild logic. Closes #5963 --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>

2026-05-19

perf(encoding): inline mini-block chunk metadata into page layout proto The per-chunk "Words" metadata buffer (a few KB per page) is now also embedded in the MiniBlockLayout protobuf via a new optional field `inline_chunk_meta`. ColumnMetadata is read in a single coalesced I/O at file open, so new readers can populate the chunk metadata from the proto and skip the per-column metadata fetch in MiniBlockScheduler::initialize. When the page has no dictionary and no repetition index, initialize issues zero I/O — addressing the per-column read on the cold search-cache path called out in #4888. The writer continues to emit the metadata buffer at buffer_offsets_and_sizes[0], so older readers (which ignore the unknown proto field) keep decoding correctly. No on-disk format version bump required.

2026-05-18

feat: builder-style MemWAL initialization API (#6815) ## Summary Replaces the struct-based MemWAL initialization API with a fluent builder on `Dataset`, and persists default `ShardWriter` configuration in the MemWAL index. `Dataset::initialize_mem_wal()` returns an `InitializeMemWalBuilder`: ```rust dataset .initialize_mem_wal() .bucket_sharding("id", 16) .maintained_indexes(["id_btree"]) .writer_config_defaults(ShardWriterConfig::default().with_durable_write(false)) .execute() .await?; ``` - `bucket_sharding(column, num_buckets)`, `unsharded()`, and `identity_sharding(column)` are high-level sharding strategies. They own the `ShardingSpec` construction and validation that callers previously did by hand; `num_shards` is derived from the sharding choice. - `writer_config_defaults(ShardWriterConfig)` records the tunable writer configuration as the persisted defaults — a new `map<string, string> writer_config_defaults` field on the `MemWalIndexDetails` protobuf message — so every writer, across processes and restarts, starts from the same defaults. `add_writer_config_default` records arbitrary extra keys. - The Python `initialize_mem_wal` binding accepts the full builder surface (sharding, maintained indexes, writer-config defaults) and invokes the builder; a new `mem_wal_index_details` binding reads the recorded details back. - Removed: `MemWalConfig`, `MemWalShardConfig`, `initialize_mem_wal_with_shards`. ## Context While reviewing lancedb/lancedb#3396, @jackye1995 noted that per-writer tuning knobs are runtime configuration and should not be persisted as part of the sharding spec. That holds for a writer's *live* `ShardWriterConfig`, which stays non-persisted. This PR records only the *defaults*: without a persisted default, writers would be configured independently and could silently drift apart. @jackye1995 also asked for a builder style so bucket sharding is easy to set, moving the hash-bucket spec logic out of the LanceDB layer into Lance. The builder owns all three sharding strategies (bucket, unsharded, identity), so LanceDB's `set_lsm_write_spec` can call the builder directly instead of hand-building shard specs.

2026-05-17

refactor: rename ShardSpec to ShardingSpec (#6813) ## Summary Renames the `ShardSpec` / `ShardField` types — and the `shard_spec` / `shard_specs` identifiers and the corresponding protobuf messages — to `ShardingSpec` / `ShardingField` for naming consistency. The protobuf message names change and the `shard_specs` field is renamed to `sharding_specs`; all field numbers and tags are unchanged, so the change is wire-compatible.

2026-05-14

fix(mem_wal): stop WAL replay from re-loading already-compacted entries (#6767) ## Summary After a writer flushed a memtable to L0 and an external compactor merged that generation into the base table — legitimately draining `flushed_generations` to empty — a subsequent restart re-replayed the original WAL entries into the new active memtable, duplicating rows on read. Two bugs were interacting: 1. **Disambiguation:** `replay_memtable_from_wal` distinguished "fresh shard" from "flushed and compacted" via `flushed_generations.is_empty()`. That works in a closed-world deployment but breaks the moment an external compactor enters the picture — and the compactor is the *intended* consumer that drains that vector, so the signal is structurally broken under OSS-WAL. 2. **Cursor never advanced:** `MemTableFlusher::flush` read `covered_wal_entry_position` from `memtable.last_flushed_wal_entry_position()`, but that field is only set by the `mark_wal_flushed` test helper. In production it stayed at 0, so `replay_after_wal_entry_position` never advanced past 0. Under 0-based WAL positions this masked bug #1 — both "fresh" and "post-flush-of-0" produced cursor=0. ## Fix - **WAL positions are now 1-based** (`FIRST_WAL_ENTRY_POSITION = 1`). A cursor of `0` unambiguously means "no flush has stamped this shard," so replay collapses to `cursor.saturating_add(1)` without consulting `flushed_generations`. - **`WalFlushHandler::handle`** writes the just-appended position back into `state.last_flushed_wal_entry_position` under the state lock before signalling the completion cell. - **`MemTableFlusher::flush` / `flush_with_indexes`** now take an explicit `covered_wal_entry_position` arg. The production caller derives it per-memtable from the `WalFlushResult` carried in the completion cell — authoritative under concurrent flushes — falling back to `memtable.frozen_at_wal_entry_position()` when freeze did not trigger a flush. - **State seed at open** uses the post-replay WAL tip, not `manifest.wal_entry_position_last_seen` (the latter is bumped on every tailer read and can sit above any flushed generation). - Proto field docs on `ShardManifest.replay_after_wal_entry_position` / `wal_entry_position_last_seen` updated to spell out the 1-based convention and what default-0 means. ## Test plan - [x] Added `test_memtable_replay_skips_entries_after_external_compaction` in `rust/lance/src/dataset/mem_wal/write.rs`: open writer, put rows, close (flush), simulate the compactor by directly committing a manifest with empty `flushed_generations`, reopen, assert the memtable is empty. Fails on the pre-fix code; passes now. - [x] `cargo test -p lance --lib dataset::mem_wal` — 236/236 pass - [x] `cargo test -p lance --lib` — 1600/1600 pass - [x] `cargo test -p lance-index --lib` — 302/302 pass - [x] `cargo clippy --all --tests --benches -- -D warnings` — clean - [x] `cargo fmt --all -- --check` — clean ## Compatibility WAL position numbering changes from 0-based to 1-based. Existing on-disk manifests / WAL files written by the prior `oss-wal-multiplex` code are not migrated — coordinated with downstream consumers (sophon) to start fresh. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

2026-05-08

feat: add unenforced_clustering_key to format spec (#6552) ## Summary - Add `unenforced_clustering_key` metadata to the Lance schema format, mirroring the existing `unenforced_primary_key` pattern - Clustering keys hint at the physical ordering of data within a table, enabling query engine optimizations such as storage-partitioned joins (SPJ) - Unlike primary keys, clustering key fields may be nullable Changes across all layers: - **Protobuf**: `unenforced_clustering_key` (bool) + `unenforced_clustering_key_position` (uint32) fields 14-15 - **Rust core**: field struct, constants, Arrow metadata parsing, schema method - **Protobuf serialization**: round-trip support with backward compat - **Java JNI + LanceField**: constructor args and getters - **Python bindings + type stubs**: `is_unenforced_clustering_key()` / `unenforced_clustering_key_position()` - **Format docs**: clustering key metadata section ## Motivation This was discussed in the lance-spark SPJ PR (lance-format/lance-spark#445). Rather than using custom table properties, embedding clustering key info in the schema metadata follows the established pattern and avoids migration issues. ## Test plan - [x] `cargo check -p lance-core -p lance-file` passes - [x] `cargo test -p lance-core -p lance-file` passes (all tests including existing primary key tests) - [x] `cargo clippy -p lance-core -p lance-file --tests -- -D warnings` clean 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

2026-05-03

fix: propagate update_columns offsets and partial last_updated for RewriteColumns (#6650) ## Summary * Fixes https://github.com/lance-format/lance/issues/6505 * `FileFragment::update_columns` returns `Result<(Fragment, Vec<u32>)>` (unchanged public shape). `update_columns_with_offsets` returns `FragmentUpdateColumnsResult` (fragment, `fields_modified`, `matched_offsets: RoaringBitmap`) for callers that need physical row indices for stable row-id metadata. * `HashJoiner::matched_join_rows` — boolean mask for hash hits; used by `update_columns_with_offsets` and covered by `test_matched_join_rows`. * `Operation::Update`: optional `updated_fragment_offsets: Option<UpdatedFragmentOffsets>` where `UpdatedFragmentOffsets` wraps `HashMap<u64, RoaringBitmap>` (newtype with `Default`, `PartialEq`, manual `DeepSizeOf`). `None` means the caller did not supply offsets. * Proto (`transaction.proto`): backward-compatible `map<uint64, UInt32List> updated_fragment_offsets = 9` on `Update`; serde round-trip preserves semantics. * `build_manifest`: when stable row IDs are enabled, `update_mode == RewriteColumns`, and `Some(UpdatedFragmentOffsets(..))` includes a non-empty bitmap for a fragment, calls `refresh_row_latest_update_meta_for_partial_frag_rewrite_cols` for those offsets only — unmatched rows and untouched fragments are left unchanged. * JNI / Java: `FragmentUpdateResult` includes matched row offsets; the 2-arg constructor `(FragmentMetadata, long[])` delegates to the 3-arg form with an empty offset array for compatibility. JNI uses `update_columns_with_offsets`. * Python: `update_columns` binding correctly destructures the `(Fragment, Vec<u32>)` tuple. ## Root cause For `Operation::Update` with `RewriteColumns`, commits could advance the dataset version without advancing `_row_last_updated_at_version` for the rows that were actually rewritten. `update_columns` did not report which physical offsets matched, and `build_manifest` had no per-fragment offset map to drive the partial refresh. Without that information the transaction layer cannot distinguish which rows changed, so the version metadata is not updated. ## Implementation notes * `RoaringBitmap` iteration is ascending and duplicate-free; redundant `sort` / `dedup` when building proto lists or offset vectors from bitmaps were removed. * Call sites that do not populate offsets use `updated_fragment_offsets: None`. ## Why the protobuf field exists lance-spark passes `Transaction` through JNI as a protobuf blob: Java builds a `Transaction` proto, Rust deserializes it and runs `build_manifest`. Without `updated_fragment_offsets` on the wire, the decoded `Operation::Update` would always have `updated_fragment_offsets: None` even when matched offsets were computed on the JVM side, and the partial refresh in `build_manifest` would silently do nothing. ## Test plan * `cargo test -p lance test_matched_join_rows` — `HashJoiner::matched_join_rows`. * `cargo test -p lance test_build_manifest_partial_last_updated_rewrite_columns_stable_row_ids` — `Dataset::commit` -> `build_manifest`: two fragments, partial `update_columns_with_offsets`, `Operation::Update` with `RewriteColumns` and an offset map; asserts matched vs unmatched vs untouched row version metadata. * `cargo test -p lance test_fragment_update` — fragment path with `Operation::Update` and offsets. * `cargo test -p lance --tests` (or at least `cargo check -p lance --tests`) and `cargo check --manifest-path java/lance-jni/Cargo.toml`. The `pylance` crate is excluded from the root workspace; validate Python bindings in the usual `maturin` / CI flow if you touch `python/`. ## Compatibility * Rust: `update_columns` signature unchanged; `update_columns_with_offsets` is additive. * Java: 2-arg `FragmentUpdateResult` constructor preserved. * Proto: field 9; older clients ignore unknown fields. --------- Co-authored-by: Jing chen He <jingh@adobe.com>

feat: add write ahead log appender and tailer primitives (#6669) ## Summary Surface a generic read/write surface for MemWAL shards so callers can drive a shard directly without going through the existing flusher. - **Shard self-description.** `ShardManifest` now records the `(shard_spec_id, field_id -> bytes)` assignment that produced the shard, so a manifest found on disk can be mapped back to a shard spec without consulting the inline index. Proto adds `ShardFieldEntry`; manifest carries `shard_field_values: HashMap<String, Vec<u8>>`. - **Idempotent shard initialization.** `ShardManifestStore::initialize_shard()` writes manifest v1 at writer epoch 0 and treats `AlreadyExists` as success when the existing manifest matches. - **Generic WAL primitives.** - `WalAppender::open()` claims a writer epoch via the manifest store; `append(batches)` serializes Arrow IPC and writes with put-if-not-exists, retrying on conflict and fencing on PUT failure. - `WalTailer::read_entry`, `next_position`, `first_position` use `wal_entry_position_last_seen` as a probe hint with a listing fallback. `WalReadEntry` includes the `writer_epoch` recorded in the entry so callers can fence-check on replay. - **Index discovery helpers.** New `Dataset` methods `mem_wal_index_details()` and `list_mem_wal_latest_shard_ids()` (object-storage directory listing). The inline shard snapshot is positioned as a stale read-optimization rather than the source of truth; `docs/src/format/table/mem_wal.md` is updated to match. ## Tests Added unit tests for: - `ShardManifestStore::initialize_shard` happy path, idempotent on match, rejects mismatched conflict. - `WalAppender` / `WalTailer` round-trip including `writer_epoch` propagation; position increment. - Writer-epoch fencing: a stale appender hits the conflict path on append and surfaces the fence error. - Input validation: empty batch list and zero-row batches are rejected. - `WalTailer::with_cursor_updates(true)` updates `wal_entry_position_last_seen` asynchronously, and `next_position()` still resolves correctly. - `mem_wal_path()` helper. cc @jackye1995 for review.

2026-04-29

feat(vector): add partition search parallelism (#6475) ## Feature This PR adds query-time `query_parallelism` for vector search partition execution and wires it through Rust, Python, and Java. Callers can control how many IVF partitions a single query may search concurrently: - `0`: auto policy. The current implementation maps auto to the single-worker sequential path. - `1`: single-worker sequential partition search. - `-1`: use the CPU pool size. - `>= 2`: partition-parallel search, clamped to the CPU pool size. The default is `0`, which currently preserves the optimized sequential execution path. ## Performance Improvement The performance issue is per-query worker fan-out. With many concurrent queries, spawning a CPU task for every partition of every query increases contention on the CPU worker pool and lengthens queueing time. This is especially visible for fixed-`nprobes` IVF workloads where every query searches the same number of partitions. This PR improves that path by making query partition scheduling configurable while keeping the optimized sequential model as the default auto behavior. Sequential fixed-`nprobes` search prepares partitions asynchronously, then searches prepared partitions on one CPU worker using a query-level global top-k heap. Parallel execution remains available for workloads that benefit from intra-query partition parallelism. ## Implementation - Rust, Python, and Java expose `query_parallelism` on vector search APIs. - `ANNIvfSubIndexExec` converts the configured value into an effective partition concurrency. - Effective partition concurrency is clamped to the CPU pool size. - IVF v2 splits partition search into async prepare and sync execute phases. - Sequential fixed-`nprobes` search uses a query-level global top-k heap. - Sequential late-search keeps the existing per-partition output shape so early-stop behavior is preserved. - Parallel execution uses direct per-partition search tasks, matching the original execution model. - IVF_RQ/Flat sub-index search reuses caller-owned scratch buffers for RQ distance-table quantization and top-k accumulation to reduce allocation overhead. ## Developer Impact - Rust callers can call `Scanner::query_parallelism(...)`. - Python callers can pass `query_parallelism` in vector search APIs. - Java callers can use `Query.Builder#setQueryParallelism(...)`. - `parallel_mode` / `ParallelMode` have been replaced by the concurrency-based API. ## Benchmark GCP VM benchmark using the same index for all modes. All rows had matching result checksums. Configuration: `gist / IVF_RQ / target_partition_size=8192 / k=100 / nprobes=20 / columns=[] / prewarm / max_queries=1000`. Percentages are relative to `main`. | Threads | Mode | Avg | P50 | P90 | P95 | P99 | QPS | |---:|---|---:|---:|---:|---:|---:|---:| | 8 | main | 4.98 ms | 4.92 ms | 6.28 ms | 6.86 ms | 7.56 ms | 1584.6 | | 8 | sequential | 4.27 ms (-14.2%) | 4.21 ms (-14.5%) | 5.03 ms (-19.9%) | 5.27 ms (-23.1%) | 5.97 ms (-21.0%) | 1838.9 (+16.1%) | | 8 | parallel | 5.02 ms (+0.7%) | 4.97 ms (+0.9%) | 6.27 ms (-0.2%) | 6.60 ms (-3.7%) | 7.36 ms (-2.7%) | 1575.4 (-0.6%) | | 16 | main | 9.95 ms | 9.74 ms | 13.12 ms | 14.11 ms | 16.83 ms | 1583.1 | | 16 | sequential | 8.09 ms (-18.6%) | 7.98 ms (-18.0%) | 9.84 ms (-25.0%) | 10.47 ms (-25.8%) | 11.67 ms (-30.6%) | 1936.8 (+22.3%) | | 16 | parallel | 9.95 ms (+0.0%) | 9.68 ms (-0.6%) | 13.37 ms (+1.9%) | 14.42 ms (+2.2%) | 16.83 ms (+0.0%) | 1583.6 (+0.0%) | | 32 | main | 18.68 ms | 18.16 ms | 26.66 ms | 28.77 ms | 33.12 ms | 1652.7 | | 32 | sequential | 15.50 ms (-17.0%) | 15.15 ms (-16.6%) | 20.57 ms (-22.9%) | 22.16 ms (-23.0%) | 25.32 ms (-23.6%) | 2000.6 (+21.1%) | | 32 | parallel | 19.13 ms (+2.4%) | 18.58 ms (+2.3%) | 26.49 ms (-0.6%) | 29.12 ms (+1.2%) | 33.92 ms (+2.4%) | 1625.9 (-1.6%) | | 64 | main | 33.98 ms | 33.01 ms | 49.65 ms | 53.87 ms | 63.58 ms | 1718.4 | | 64 | sequential | 29.95 ms (-11.8%) | 29.67 ms (-10.1%) | 42.44 ms (-14.5%) | 46.81 ms (-13.1%) | 54.42 ms (-14.4%) | 1949.4 (+13.4%) | | 64 | parallel | 35.17 ms (+3.5%) | 34.37 ms (+4.1%) | 50.70 ms (+2.1%) | 55.04 ms (+2.2%) | 69.09 ms (+8.7%) | 1650.7 (-3.9%) | | 128 | main | 38.73 ms | 36.07 ms | 68.28 ms | 76.78 ms | 96.81 ms | 1663.3 | | 128 | sequential | 37.48 ms (-3.2%) | 35.90 ms (-0.5%) | 65.26 ms (-4.4%) | 71.82 ms (-6.5%) | 87.48 ms (-9.6%) | 1915.3 (+15.2%) | | 128 | parallel | 41.29 ms (+6.6%) | 39.38 ms (+9.2%) | 71.68 ms (+5.0%) | 80.70 ms (+5.1%) | 96.20 ms (-0.6%) | 1577.2 (-5.2%) | ## Validation - `cargo fmt --all --check` - `cargo check -p lance --tests` - `cd python && cargo check` - `cd java && cargo check --manifest-path ./lance-jni/Cargo.toml` - `uv run --with maturin maturin develop` - `uv run pytest python/tests/test_vector_index.py::test_vector_index_with_query_parallelism python/tests/test_vector_index.py::test_vector_index_invalid_query_parallelism` - Python ruff check / format check for touched Python files

2026-04-26

feat: add prefilter_type to ANNIvfSubIndexExecProto (#6613) ## Summary - Add `PreFilterType` enum (`None`, `FilteredRowIds`, `ScalarIndexQuery`) to `ANNIvfSubIndexExecProto` in `ann.proto` - Encode sets `prefilter_type` based on the exec's `PreFilterSource` variant - Decode uses `prefilter_type` from the proto to reconstruct the correct `PreFilterSource` variant, replacing the schema-sniffing heuristic used by downstream codecs - Errors on inconsistent state: prefilter type set but no child plan provided, or child plan provided but type is `None` This enables downstream codecs (e.g. sophon's `LancePhysicalExtensionCodec`) to correctly reconstruct `PreFilterSource` without guessing from the child plan's schema. ## Test plan - [x] `test_ann_ivf_sub_index_proto_roundtrip` — existing None prefilter roundtrip (updated to new API) - [x] `test_sub_index_proto_roundtrip_filtered_row_ids` — FilteredRowIds encode/decode - [x] `test_sub_index_proto_roundtrip_scalar_index_query` — ScalarIndexQuery encode/decode - [x] `test_sub_index_proto_error_type_none_but_child_provided` — errors on mismatch - [x] `test_sub_index_proto_error_type_set_but_no_child` — errors on mismatch 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

2026-04-24

feat: add ANNIvfPartitionExecProto (#6612)

2026-04-16

feat: add ANN proto codecs and extract table_identifier module (#6503) Add protobuf encode/decode for `ANNIvfSubIndexExec` --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

2026-04-10

feat: add generated row id ranges

2026-04-09

refactor: narrow reserved row id proto scope

docs: clarify reserved row id append semantics

refactor: derive reserved row ids from transactions

2026-04-08

feat: reserve stable row ids for append transactions

2026-04-03

refactor: rename "region" to "shard" in mem_wal implementation (#6367) Avoid confusion with object store regions (e.g., AWS regions) which are unrelated to the MemWAL concept of a unique writer/reader instance. Closes https://github.com/lance-format/lance/issues/6355 Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

2026-03-30

refactor: align proto field names with Rust struct fields, serialize full IndexMetadata - Rename `distance_type` to `metric_type` in VectorQueryProto (matches Query.metric_type) - Replace `index_name` + `segment_uuids` with `repeated bytes indices` in ANNIvfSubIndexExecProto, serializing full IndexMetadata via prost-encoded bytes from lance.table package (avoids lossy UUID-only serialization and removes load_indices_by_name roundtrip on deserialization) - Fix unused variable warning in make_indexed_dataset test helper - Move test imports to top of test module - Remove stale doc comment in table_identifier.rs - Strengthen sub-index roundtrip test to verify IndexMetadata fields Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

refactor: extract table_identifier module, add sub index roundtrip test - Move table_identifier_from_dataset, table_identifier_from_dataset_with_manifest, and open_dataset_from_table_identifier into a shared table_identifier module so both filtered_read_proto and ann_ivf_proto use the same code - Re-export from filtered_read_proto for backwards compatibility - Remove PE-specific comments from ann_ivf.proto (open source doesn't need to know PE) - Add test_ann_ivf_sub_index_proto_roundtrip with a real IVF index Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

feat: add proto serialization for ANNIvfPartitionExec and ANNIvfSubIndexExec Add protobuf definitions and encode/decode functions for distributed execution of ANN IVF plan nodes, following the FilteredReadExec pattern. Key design choices: - VectorQueryProto round-trips ALL Query fields using Arrow IPC for the key array (supports Float16/Float32/Float64/UInt8, not just Float32) - DistanceType uses Display/TryFrom<&str> instead of manual match - from_proto functions take Option<Arc<Dataset>> so callers can pass from cache or let it open from storage (same as FilteredReadExec) - ANNIvfSubIndexExec from_proto takes input + prefilter_source as params — codec on the caller side handles child extraction Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

2026-03-18

feat: add file list with sizes to IndexMetadata (#5497) Start storing the size of index files in the manifest file. Now you get the byte size of an index in `describe_indices()`: ```python >>> ds.describe_indices() [IndexDescription(name='x_idx', ..., total_size_bytes=1475), IndexDescription(name='y_idx', ..., total_size_bytes=2772)] ``` Because we can now skip the `HEAD` request to get file size, this reduces read IO requests for cold queries: | Case | Before | After | |------|-------:|------:| | Cold BTree search | 5 | 3 | | Warm BTree search | 1 | 1 | | Cold FTS search | 10 | 5 | | Warm FTS search | 1 | 1 | (Cold queries now do 1 IOP per index file, plus 1 for taking results. BTree has two files, while FTS has 4 files, so that's why it's 3 and 5 IOPs respectively.) Migration is handled on write: when we commit, we check for any indices missing the files field and will add them as needed. Closes https://github.com/lance-format/lance/issues/5226 --------- Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>

2026-02-26

feat: compress complex all null (#4990) In order to compress complex all null, we need to add additional parameters in the proto so we know what compression are used for definition level and repetition level and the number of values accordingly. resolve https://github.com/lancedb/lance/issues/4885 --------- Co-authored-by: stevie9868 <yingjianwu2@email.com> Co-authored-by: Xuanwo <github@xuanwo.io>