Proto commits in facebookincubator/velox

These 17 commits are when the Protocol Buffers files have changed:

2026-06-02

feat(dwrf): Add Type attributes to dwrf proto for Iceberg interop (#17661) Summary: Pull Request resolved: https://github.com/facebookincubator/velox/pull/17661 DWRF data files written for Iceberg tables today carry no Iceberg field IDs, so renaming a column in the Iceberg schema silently returns NULL on read. DWRF also has no slot for the other per-column type hints the Iceberg spec mandates for ORC (timestamp unit, UUID vs raw binary, fixed-length size, etc.). This change adds a generic key-value attributes bag to the `Type` proto in the Velox DWRF proto (`velox/dwio/dwrf/proto/dwrf_proto.proto`): ``` message StringPair { optional string key = 1; optional string value = 2; } message Type { ... repeated StringPair attributes = 4; } ``` The mechanism mirrors Apache ORC's `Type.attributes` slot. Iceberg's ORC integration carries per-column metadata under these keys, all of which apply to DWRF without further proto changes: - `iceberg.id` — field id for schema evolution - `iceberg.required` — schema-level nullability - `iceberg.timestamp-unit` — MICROS vs NANOS for `timestamp` and `timestamp_ns` - `iceberg.long-type` — TIME marker on a long - `iceberg.binary-type` — UUID / FIXED / BINARY disambiguator - `iceberg.length` — length L for `fixed(L)` - `iceberg.struct-type` — VARIANT marker on a struct Proto2 wire compatibility is preserved: `repeated StringPair` defaults to an empty list, old readers ignore the new field, and existing DWRF files have no attributes set. Writer and reader plumbing land in follow-up diffs; the Iceberg manifest continues to report DWRF files as `"ORC"` until the Iceberg spec adds a DWRF enum. Per zzhao0's review feedback (V8): the Java-side equivalent does NOT belong in `fbcode/fbjava/bbio/hive-dwrf/proto/orc_proto.proto` — that proto has been in deprecation since the 2022 BBIO-Spark-decoupling effort and is not on the fb-spark Iceberg-DWRF read path. The fb-spark read path (`PlasmaIcebergDwrfBatchRecordIterable` → `com.facebook.xldb.orc.presto.reader.Reader` → `com.facebook.xldb.presto.shaded.com.facebook.presto.orc.proto.DwrfProto`) is Presto-shaded, so Java parity will be added via a follow-up PR to `prestodb/orc-protobuf` upstream — XLDB will then pick it up on the next shaded-presto bump. Reviewed By: apurva-meta, zzhao0 Differential Revision: D106713217 fbshipit-source-id: 7086a2cce273c382c9db4386af22e003045578f3

The documentation is generated from this commit.

2026-02-12

Fix license header: dwio (#16372) Summary: Pull Request resolved: https://github.com/facebookincubator/velox/pull/16372 Update copyright header from "Facebook, Inc. and its affiliates" to "Meta Platforms, Inc. and affiliates" and update legacy "Copyright 2004-present Facebook" headers across the velox/dwio directory. Differential Revision: D93129962

Fix license header: dwio Summary: Update copyright header from "Facebook, Inc. and its affiliates" to "Meta Platforms, Inc. and affiliates" and update legacy "Copyright 2004-present Facebook" headers across the velox/dwio directory. Differential Revision: D93129962

2025-08-29

feat: Add gzip compression support (#14593) Summary: Add gzip support for ZlibCompressor. Pull Request resolved: https://github.com/facebookincubator/velox/pull/14593 Reviewed By: kewang1024 Differential Revision: D80961679 Pulled By: zacw7 fbshipit-source-id: 275aeba62482d295e6eb9c03dfa1236c23a808f4

2025-07-01

refactor: Delete substrait (#13938) Summary: Pull Request resolved: https://github.com/facebookincubator/velox/pull/13938 Fixes https://github.com/facebookincubator/velox/issues/8895 Reviewed By: xiaoxmeng Differential Revision: D77556643 fbshipit-source-id: 7db14fcd90a2599975fc8abd1d7946b470b7cd8f

2025-06-24

misc: Enable Spark query runner as reference in aggregation fuzzer test (#9559) Summary: - Enables Spark query runner as reference DB in Spark aggregation fuzzer test. - Removes duplicate Spark aggregation fuzzer test from experimental run. - Implements `executeAndReturnVector` in Spark query runner and removes the below two APIs from the reference query runner as they are not in use since https://github.com/facebookincubator/velox/commit/8519fad1170975ab32ee75e186a88f91d3098495. ``` virtual std::multiset<std::vector<velox::variant>> execute( const std::string& /*sql*/, const std::vector<velox::RowVectorPtr>& /*input*/, const velox::RowTypePtr& /*resultType*/) { VELOX_UNSUPPORTED(); } virtual std::vector<RowVectorPtr> executeVector( const std::string& /*sql*/, const std::vector<RowVectorPtr>& /*input*/, const RowTypePtr& /*resultType*/) { VELOX_UNSUPPORTED(); } ``` - Removes the Spark source file download. Instead, photo files are introduced in Velox and need to be updated for every Spark version. Currently, these files are imported from Spark 3.5.1 version. Fixes https://github.com/facebookincubator/velox/issues/9270. Pull Request resolved: https://github.com/facebookincubator/velox/pull/9559 Reviewed By: Yuhta Differential Revision: D65420347 Pulled By: kevinwilfong fbshipit-source-id: b6afd1c554a000fea3f187a024f3f8a794770579

2024-02-28

Delete dead codegen code Reviewed By: xiaoxmeng Differential Revision: D54269600 fbshipit-source-id: adc6120e85c63678449572ac93f71372de8a13d1

2023-02-08

Update substrait to 0.23.0 (#3959) Summary: Substrait supports physical join and expr evaluation in v0.23.0 Pull Request resolved: https://github.com/facebookincubator/velox/pull/3959 Reviewed By: Yuhta Differential Revision: D43123510 Pulled By: pedroerp fbshipit-source-id: f2c53a7e42a43445eed9c6c8fffa7e53b5106d94

2023-01-13

Update substrait to 0.22.0 (#3697) Summary: Update substrait to newest release version 0.22.0 Pull Request resolved: https://github.com/facebookincubator/velox/pull/3697 Reviewed By: Yuhta Differential Revision: D42497470 Pulled By: mbasmanova fbshipit-source-id: 4b78f03f1f4c806981a4e4ac36a9008e36b58afe

2022-07-25

Update substrait to 0.7.0 (#2002) Summary: Also, rename tests/data/sub.json to q6_first_stage.json. Pull Request resolved: https://github.com/facebookincubator/velox/pull/2002 Reviewed By: pedroerp Differential Revision: D37988016 Pulled By: mbasmanova fbshipit-source-id: 163e17ba9fbd19f89ba431c8e95797c1486f0dc8

2022-06-23

Add limited ORC reader support (#1746) Summary: This PR is an attempt to adding ORC reader support based on `DwrfReader`. For more context, please refer to https://github.com/facebookincubator/velox/issues/1747. This PR introduces the following changes: - Import `.proto` file of ORC from [apache/orc](https://github.com/apache/orc) - Distinguish ORC and DWRF in `FileFormat` and make `DwrfReader` aware of the file format it's processing - Adding the new `PostScript` class that is shared by of DWRF and ORC* - Don't try to read cache when parsing ORC's `PostScript` - For ORC's int DIRECT decoding, use RLE insterad of `DirectDecoder` Currently I use ad hoc method to test the change, that is, try reading and printing the content of [apache/orc's example files](https://github.com/apache/orc/tree/main/examples) (excluding corrupted and encrypted) using modified `DwrfReader` and comparing the output with the output of [apache/orc's orc-contents](https://orc.apache.org/docs/cpp-tools.html#orc-contents), with above changes, I've got some of tests emitting correct output(24 out of 39). For the incorrect / crashed test cases, I think they are due to missing type support (DECIMAL, DATE etc) or feature support (bloom filter). <details> <summary>Detailed test cases</summary> <google-sheets-html-origin><meta name="sheets-banding" data-sheets-range="A1:Z1000" data-sheets-banding-properties="{&quot;1&quot;:{&quot;1&quot;:{&quot;4&quot;:{&quot;1&quot;:2,&quot;2&quot;:12434877}},&quot;2&quot;:{&quot;4&quot;:{&quot;1&quot;:2,&quot;2&quot;:16777215}},&quot;3&quot;:1,&quot;4&quot;:{&quot;4&quot;:{&quot;1&quot;:2,&quot;2&quot;:15987699}},&quot;5&quot;:1}}"> Test | Correct | Issue | Compression | Type -- | -- | -- | -- | -- TestOrcFile.testSeek | Y | | zlib | struct<boolean1:boolean,byte1:tinyint,short1:smallint,int1:int,long1:bigint,float1:float,double1:double,bytes1:binary,string1:string,middle:struct<list:array<struct<int1:int,string1:string>>>,list:array<struct<int1:int,string1:string>>,map:map<string,struct<int1:int,string1:string>>> TestOrcFile.test1 | Y | | zlib | struct<boolean1:boolean,byte1:tinyint,short1:smallint,int1:int,long1:bigint,float1:float,double1:double,bytes1:binary,string1:string,middle:struct<list:array<struct<int1:int,string1:string>>>,list:array<struct<int1:int,string1:string>>,map:map<string,struct<int1:int,string1:string>>> TestOrcFile.testStringAndBinaryStatistics | Y | | zlib | struct<bytes1:binary,string1:string> nulls-at-end-snappy | Y | | snappy | struct<_col0:tinyint,_col1:smallint,_col2:int,_col3:bigint,_col4:float,_col5:double,_col6:boolean> TestStringDictionary.testRowIndex | Y | | zlib | struct<str:string> TestOrcFile.testMemoryManagementV11 | Y | | none | struct<int1:int,string1:string> TestOrcFile.columnProjection | Y | | none | struct<int1:int,string1:string> TestOrcFile.emptyFile | Y | | none | none TestOrcFile.metaData | Y | | none | struct<boolean1:boolean,byte1:tinyint,short1:smallint,int1:int,long1:bigint,float1:float,double1:double,bytes1:binary,string1:string,middle:struct<list:array<struct<int1:int,string1:string>>>,list:array<struct<int1:int,string1:string>>,map:map<string,struct<int1:int,string1:string>>> demo-12-zlib | Y | | zlib | struct<_col0:int,_col1:string,_col2:string,_col3:string,_col4:int,_col5:string,_col6:int,_col7:int,_col8:int> TestOrcFile.testWithoutIndex | Y | | snappy | struct<int1:int,string1:string> zero | Y | | none | none TestOrcFile.testPredicatePushdown | Y | | none | struct<int1:int,string1:string> complextypes_iceberg | Y | | zlib | struct<id:bigint,int_array:array<int>,int_array_array:array<array<int>>,int_map:map<string,int>,int_map_array:array<map<string,int>>,nested_struct:struct<a:int,b:array<int>,c:struct<d:array<array<struct<e:int,f:string>>>>,g:map<string,struct<h:struct<i:array<double>>>>>> TestOrcFile.testStripeLevelStats | Y | | zlib | struct<int1:int,string1:string> TestOrcFile.testMemoryManagementV12 | Y | | none | struct<int1:int,string1:string> TestVectorOrcFile.testLzo | Y | | lzo | struct<x:bigint,y:int,z:bigint> TestVectorOrcFile.testZstd.0.12 | Y | | zstd | struct<x:bigint,y:int,z:bigint> demo-11-zlib | Y | | zlib | struct<_col0:int,_col1:string,_col2:string,_col3:string,_col4:int,_col5:string,_col6:int,_col7:int,_col8:int> TestVectorOrcFile.testLz4 | Y | | lz4 | struct<x:bigint,y:int,z:bigint> TestOrcFile.testSnappy | Y | | snappy | struct<int1:int,string1:string> version1999 | Y | | none | struct<> orc_no_format | Y | | zlib | struct<_col0:array<string>,_col1:map<int,string>,_col2:struct<name:string,score:int>> demo-11-none | Y | | none | struct<_col0:int,_col1:string,_col2:string,_col3:string,_col4:int,_col5:string,_col6:int,_col7:int,_col8:int> TestOrcFile.testTimestamp | N | TIMESTAMP | zlib | timestamp orc_split_elim_new | N | DECIMAL | zlib | struct<userid:bigint,string1:string,subtype:double,decimal1:decimal(16,6),ts:timestamp> decimal64_v2 | N | DECIMAL | zlib | struct<a:bigint,b:decimal(12,0),c:decimal(20,2),d:decimal(12,2),e:decimal(2,2)> orc-file-11-format | N | DECIMAL | none | struct<boolean1:boolean,byte1:tinyint,short1:smallint,int1:int,long1:bigint,float1:float,double1:double,bytes1:binary,string1:string,middle:struct<list:array<struct<int1:int,string1:string>>>,list:array<struct<int1:int,string1:string>>,map:map<string,struct<int1:int,string1:string>>,ts:timestamp,decimal1:decimal(0,0)> TestOrcFile.testUnionAndTimestamp | N | TIMESTAMP, UNION | none | struct<time:timestamp,union:uniontype<int,string>,decimal:decimal(38,18)> orc_split_elim | N | DECIMAL | none | struct<userid:bigint,string1:string,subtype:double,decimal1:decimal(0,0),ts:timestamp> TestOrcFile.testDate1900 | N | TIMESTAMP, DATE | zlib | struct<time:timestamp,date:date> bad_bloom_filter_1.6.11 | N | bloom filter | zlib | struct<id:bigint,name:string> TestOrcFile.testDate2038 | N | TIMESTAMP, DATE | zlib | struct<time:timestamp,date:date> over1k_bloom | N | DECIMAL, bloom filter | zlib | struct<_col0:tinyint,_col1:smallint,_col2:int,_col3:bigint,_col4:float,_col5:double,_col6:boolean,_col7:string,_col8:timestamp,_col9:decimal(4,2),_col10:binary> orc_split_elim_cpp | N | TIMESTAMP, DECIMAL | zlib | struct<userid:bigint,string1:string,subtype:double,decimal1:decimal(16,6),ts:timestamp> decimal64_v2_cplusplus | N | DECIMAL | zlib | struct<a:bigint,b:decimal(10,2),c:decimal(2,2),d:decimal(2,2),e:decimal(2,2)> bad_bloom_filter_1.6.0 | N | bloom filter | zlib | struct<id:bigint,name:string> decimal | N | DECIMAL | none | struct<_col0:decimal(10,5)> orc_index_int_string | N | VARCHAR | zlib | struct<_col0:int,_col1:varchar(4)> </details> To print an ORC file ``` _build/release/velox/examples/velox_example_scan_orc {path_to_orc_file} ``` A more important point I wanna discuss is the modeling of `proto`: > Adding the new `PostScript` class that is shared by of DWRF and ORC As we introduce ORC's `.proto` file, a mechanism is needed to switching between two `.proto` files flexibly. My current solution is to create a class that mimic the `proto` class' intefaces (I only implemented `PostScript` in this PR), which contains attributes of both DWRF and ORC. Pull Request resolved: https://github.com/facebookincubator/velox/pull/1746 Reviewed By: HuamengJiang Differential Revision: D37220861 Pulled By: zzhao0 fbshipit-source-id: 545c4834e0816220de4d3d93b7146589da62b349

2022-05-20

Add velox-to-substrait converter (#1212) Summary: Implement Velox-to-Substrait pan conversion for Values, Filter and Project plan nodes. Expression conversion is limited to handful of hard-coded functions. Proper integration with function registry as well as support for other plan nodes will come in a future PR. Pull Request resolved: https://github.com/facebookincubator/velox/pull/1212 Reviewed By: kgpai Differential Revision: D36536485 Pulled By: mbasmanova fbshipit-source-id: 972ce70f6c681bd4a98a44db6b3594514c807530

2022-03-11

Add Substrait-to-Velox conversion (#1048) Summary: This pr is targeted to add the base layout for Substrait-to-Velox conversion. With this change, the base layout of Substrait-to-Velox conversion would be decided. Some of Substrait plan representations can be converted into Velox plan for computing. More conversions support will be added in following pull requests. Based on [Substrait Commit](https://github.com/substrait-io/substrait/commit/9d9805be19c1d606dc2811395f203857db782872). The used function names are based on: [Substrait PR](https://github.com/substrait-io/substrait/pull/147). Pull Request resolved: https://github.com/facebookincubator/velox/pull/1048 Reviewed By: kagamiori Differential Revision: D34773052 Pulled By: pedroerp fbshipit-source-id: c4c56a43486519b567aa1d715a0d64ce02a447d1

2021-08-20

Fixing leftover copyright headers Summary: Adding copyright headers to a couple of remaining files. Reviewed By: amitkdutta Differential Revision: D30444573 fbshipit-source-id: 2af861a0a9f569707c79e0cb271b3b6b34f31dfd

2021-08-19

Rename ChecksumAlgorithm NULL option to NULL_ (#60) Summary: Pull Request resolved: https://github.com/facebookincubator/velox/pull/60 gcc on linux (9.3 on Ubuntu 20.04) seems to have a problem with the `NULL` option name and fails the compilation. Renaming the flag to NULL_ instead. Interestingly enough, all other usages of the flag I could find already use "NULL_", so I'm a little confused about how this worked in the first place: Reviewed By: zzhao0 Differential Revision: D30396818 fbshipit-source-id: 7def0b77955f76fd45af6235a802cf60ac1240e7

2021-08-09

Remove a quip link in comments (#2) Summary: Pull Request resolved: https://github.com/facebookincubator/Velox/pull/2 Reviewed By: mbasmanova Differential Revision: D30185776 fbshipit-source-id: 3a8311cdd581c6f95e6a2a79580da387be3b9b18

Initial commit fbshipit-source-id: 7c98b4413a035219f8a2949626de33424f6182e5