Proto commits in Eventual-Inc/Daft

These 54 commits are when the Protocol Buffers files have changed:

2025-12-16

remove unused files

This commit does not contain any .proto files.

add gravitino filesystem and gvfs support

2025-12-03

chore!: remove spark connect (#5743) # Remove Spark Connect support ## Motivation We've maintained Spark Connect compatibility for about a year with minimal adoption or interest from users. ## Changes removes spark connect from codebase ## Rationale - No user traction: Haven't seen meaningful adoption despite being available for ~1 year - Maintenance burden: Substantial codebase to maintain, test, and keep compatible This simplifies the codebase and lets us focus on what makes Daft unique.

This commit does not contain any .proto files.

chore: remove ir and proto crates (#5742) ## Changes Made They are unused and taking up space, lets just remove them. ## Related Issues https://github.com/Eventual-Inc/Daft/pull/5736

The documentation is generated from this commit.

2025-11-25

feat: Allow users to disable the suffix range request (#5188) At present, there are still many storage backends that do not support the `suffix range` request. I hope to still retain a parameter for users to control for themselves whether to use `suffix range` for data reading. > Why not directly tag the `suffix range` feature for each backend at the IOClient level? Because some are in the form of protocols, such as S3, which is a standard protocol, many different object storage implementations will attempt to follow the S3 protocol. But, there are also many specific implementations that do not support the `suffix range` capability. We cannot directly control at the S3 level whether `suffix range` can be supported ## Changes Made 1. Allow users to disable the suffix range request because some storage backends do not support suffix range ## Related Issues ## Checklist - [x] Documented in API Docs (if applicable) - [x] Documented in User Guide (if applicable) - [x] If adding a new documentation page, doc is added to `docs/mkdocs.yml` navigation - [x] Documentation builds and is formatted properly (tag @/ccmao1130 for docs review) --------- Co-authored-by: Cory Grinstead <cory.grinstead@gmail.com> Co-authored-by: Colin Ho <colin.ho99@gmail.com>

2025-11-12

Merge branch 'main' into cory/udf-func-sig

feat: support customized retries error message of S3 request (#5447) ## Changes Made support configure some customized keywords of s3 error message, if the configured keywords matched the actual s3 error message, which will retry the s3 request based on S3 ClassifyRetry. Notes: we retried `UnexpectedEof` error last time, and we meet a new timeout error mentioned in https://github.com/Eventual-Inc/Daft/issues/5043 during UploadPart, I think we might not list all potential transient errors, so it's better to support configured these error messages, so that we don't need modify code once meet another error. ## Related Issues https://github.com/Eventual-Inc/Daft/issues/5043 ## Checklist - [ ] Documented in API Docs (if applicable) - [ ] Documented in User Guide (if applicable) - [ ] If adding a new documentation page, doc is added to `docs/mkdocs.yml` navigation - [ ] Documentation builds and is formatted properly --------- Co-authored-by: wuzhiping <wuzhiping.007@bytedance.com>

2025-11-11

Merge branch 'main' of https://github.com/Eventual-Inc/Daft into cory/udf-func-sig

feat: add tos object source (#5372) ## Changes Made Add a new object source based on tos object store. This is the first PR to implement a e2e io flow via tos object source. ## Related Issues https://github.com/Eventual-Inc/Daft/issues/5371 ## Checklist - [ ] Documented in API Docs (if applicable) - [ ] Documented in User Guide (if applicable) - [ ] If adding a new documentation page, doc is added to `docs/mkdocs.yml` navigation - [ ] Documentation builds and is formatted properly --------- Co-authored-by: wuzhiping <wuzhiping.007@bytedance.com>

2025-10-30

feat: add planning time validation on rowwise function signatures

feat: Async batch func (#5459) ## Changes Made Async batch func. Implementation is pretty much the same as async row wise func ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> ## Checklist - [ ] Documented in API Docs (if applicable) - [ ] Documented in User Guide (if applicable) - [ ] If adding a new documentation page, doc is added to `docs/mkdocs.yml` navigation - [ ] Documentation builds and is formatted properly

2025-10-24

feat: retryable udfs (#5392) ## Changes Made currently only implemented for rowwise scalar udf. TODO: the error message could be a bit better as the row index is kinda useless without being able to correlate it back to a batch. example usage ```py image_tensor = np.random.randint(0, 256, (224, 224, 3), dtype=np.uint8) images = [image_tensor] * 10_000 df = daft.from_pydict({"images": images}) @daft.func(use_process=True, return_dtype=daft.DataType.image(), max_retries=3, on_error="raise") def random_fail(x): # Randomly fail approximately 20% of the time if random.random() < 0.2: raise ValueError("Random failure in processing") return x print(df.select(random_fail(df["images"])).collect()) ``` ``` daft.exceptions.DaftCoreException: DaftError::ComputeError Failed to execute Python UDF on some rows : 746: DaftError::PyO3Error ValueError: Random failure in processing 1171: DaftError::PyO3Error ValueError: Random failure in processing 1791: DaftError::PyO3Error ValueError: Random failure in processing 1831: DaftError::PyO3Error ValueError: Random failure in processing 2263: DaftError::PyO3Error ValueError: Random failure in processing 2364: DaftError::PyO3Error ValueError: Random failure in processing 2896: DaftError::PyO3Error ValueError: Random failure in processing 3030: DaftError::PyO3Error ValueError: Random failure in processing 3342: DaftError::PyO3Error ValueError: Random failure in processing 3484: DaftError::PyO3Error ValueError: Random failure in processing 4389: DaftError::PyO3Error ValueError: Random failure in processing 4600: DaftError::PyO3Error ValueError: Random failure in processing 4936: DaftError::PyO3Error ValueError: Random failure in processing 5018: DaftError::PyO3Error ValueError: Random failure in processing 6056: DaftError::PyO3Error ValueError: Random failure in processing 7825: DaftError::PyO3Error ValueError: Random failure in processing 8853: DaftError::PyO3Error ValueError: Random failure in processing 9291: DaftError::PyO3Error ValueError: Random failure in processing 9609: DaftError::PyO3Error ValueError: Random failure in processing ``` ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> ## Checklist - [ ] Documented in API Docs (if applicable) - [ ] Documented in User Guide (if applicable) - [ ] If adding a new documentation page, doc is added to `docs/mkdocs.yml` navigation - [ ] Documentation builds and is formatted properly --------- Co-authored-by: Colin Ho <colin.ho99@gmail.com>

2025-10-17

wip

2025-10-16

wip

2025-10-14

add retries

2025-10-11

feat: batch UDF with `@daft.func.batch` (#5362) ## Changes Made Add support for batch UDFs in our new UDF decorators with `@daft.func.batch` as well as `@daft.method.batch`. ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> ## Checklist - [x] Documented in API Docs (if applicable) - [x] Documented in User Guide (if applicable) - [x] If adding a new documentation page, doc is added to `docs/mkdocs.yml` navigation - [x] Documentation builds and is formatted properly

2025-10-10

feat: batch UDF with

2025-10-09

feat: @daft.cls decorator for new class UDFs (#5350) ## Changes Made Add the `@daft.cls` and `@daft.method` decorators for constructing a Daft class. Some additional changes: - Did a bit of refactoring on the Python side because I did not want to have to implement a whole matrix of UDFs (func vs class, row-wise vs async vs generator). Ended up combining all the function variants into a `Func` class and the code is decently clean. - Looks like we added `use_process` in #5323 but I don't think it was actually hooked up to use processes under the hood. My changes in this PR should fix that - Restructured the class-type decorators since overriding `__new__` seems to make type checkers unhappy. Resolves #5284 I initially implemented a `@daft.setup` that users would use in lieu of having an `__init__` method, but after writing several examples with it, it just felt like a very obtuse way to introduce this whole new thing just to tell users that initialization is lazy. We'd have to explain that the arguments in the constructor are passed into the setup method, and that you have to add this decorator. We had some discussion about this in the working session and I don't think others thought a lazy init would be an issue anyway. ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> ## Checklist - [x] Documented in API Docs (if applicable) - [x] Documented in User Guide (if applicable) - [x] If adding a new documentation page, doc is added to `docs/mkdocs.yml` navigation - [x] Documentation builds and is formatted properly

2025-10-06

feat: Lazy `from_glob_path` (#5235) ## Changes Made Integrate `from_glob_path` into dataframe execution instead of immediately materializing. This code used to hang (> 5 mins and then i killed it) now takes 5s ``` import daft from daft.expressions import col df = (daft .from_glob_path( "s3://eventual-data-test-bucket/digitalcorpora/pdfs/") # 8M pdfs .with_column('image', col('path').url.download()) ) df.show() ``` glob_path go brr ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> ## Checklist - [ ] Documented in API Docs (if applicable) - [ ] Documented in User Guide (if applicable) - [ ] If adding a new documentation page, doc is added to `docs/mkdocs.yml` navigation - [ ] Documentation builds and is formatted properly (tag @/ccmao1130 for docs review)

feat: add `use_process` flag for `@daft.func(...)` (#5323) ## Changes Made adds `use_process` to daft.func to allow same performance benefits as `use_process` for `daft.udf`. ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> ## Checklist - [ ] Documented in API Docs (if applicable) - [ ] Documented in User Guide (if applicable) - [ ] If adding a new documentation page, doc is added to `docs/mkdocs.yml` navigation - [ ] Documentation builds and is formatted properly

2025-10-04

protos

2025-10-02

add use_process flag to daft.func

2025-08-07

refactor: move literal to daft-core (#4928) ## Changes Made This is part of my overall work to [unify type conversions](https://www.notion.so/Daft-Python-Type-Conversions-240091bd8634808abf18e00abc9b7fc8?source=copy_link) and is in particular cleanup that I wanted to get done before adding more literal variants. The crux of this PR is the move of `LiteralValue` from `daft-dsl` to `daft-core`. This serves several purposes: 1. Series -> Literal conversion can now be delegated to the individual array types. This is a lot cleaner than doing manual downcasts based on dtype, and will allow for conversion into literal to work even if a data type has multiple physical implementations. 2. `Vec<Literal>` -> Series conversion can now be implemented as a `TryFrom`, which is more idiomatic. Literal -> Series can also be a `From` now. 3. This creates a cleaner separation between literals and expressions, whereas before they could depend on each other. Instead, literals and series are more tightly associated now, which is more sensible as literals are pretty much the scalar counterparts to series. Additional changes: - Renamed `LiteralValue` to `Literal` and the original `Literal` trait was removed in favor of implementing `From<Literal>` - Removed `Literal::FixedSizeBinary`. I don't plan on adding the fixed size/shape variants for the other types so this was kind of an odd one out. Removing it does not remove any user-facing functionality anyway, but it does simplify the implementation so I chose to go with it. - Added `Literal::Float32` - Moved `PyObjectWrapper` from `daft-dsl` to `common-py-serde` so we can use it in `daft-core` ## Related Issues https://www.notion.so/Daft-Python-Type-Conversions-240091bd8634808abf18e00abc9b7fc8?source=copy_link ## Checklist - [x] Documented in API Docs (if applicable) - [x] Documented in User Guide (if applicable) - [x] If adding a new documentation page, doc is added to `docs/mkdocs.yml` navigation - [x] Documentation builds and is formatted properly (tag @/ccmao1130 for docs review)

refactor: move literal to daft-core

2025-08-01

feat: Add offset operator support to DataFrame for Native Runner (#4582)

2025-07-31

Merge branch 'main' of https://github.com/Eventual-Inc/Daft into sammy/add-buf-publishing

add go workspace

2025-07-28

chore: new code path for scalar UDF (#4814) ## Changes Made Was trying to get kwargs to work with scalar UDFs, decided to bite the bullet and build out the entire code path for the new UDFs so that it no longer relies on the legacy UDF code. Turned out to be quite simple actually, the complexity of the legacy UDF code is largely from the class UDF stuff we did. Additional changes: - add ability to specify kwargs to UDF - remove `@daft.func.batch`. It was just an alias to `@daft.udf` anyway, I think we should actually think about it a bit more since I don't think I want the batch UDF to behave exactly like the current UDF either. - Rename Python UDF to legacy Python UDF ## Related Issues Closes https://github.com/Eventual-Inc/Daft/issues/4729 ## Checklist - [x] Documented in API Docs (if applicable) - [x] Documented in User Guide (if applicable) - [x] If adding a new documentation page, doc is added to `docs/mkdocs.yml` navigation - [x] Documentation builds and is formatted properly (tag @/ccmao1130 for docs review)

2025-07-22

perf: Parallel stateless UDFs in native runner (#4477)

2025-07-18

Addressed comments and CI

2025-07-01

commit this so far

2025-06-23

Override v1 proto instead

More issues

Compiling

2025-06-20

Figure out proto

2025-06-19

feat: creates daft-ir and daft-proto (#4539) ## Changes Made * Creates a `daft-ir` crate for a consolidate IR view + from/to proto implementations (WIP). * Creates a `daft-proto` crate which holds daft's IR protobuf definitions. * Adds initial schema, datatype, expression (rex), and logical plan (rel) definitions. ## TODOs. * The exports (and structure) are not finalized. * Effectively all relational operator (rel) definitions. * Testing. testing. testing. ## Related Issues * n/a ## Checklist - [n/a] Documented in API Docs (if applicable) - [n/a] Documented in User Guide (if applicable) - [n/a] If adding a new documentation page, doc is added to `docs/mkdocs.yml` navigation - [n/a] Documentation builds and is formatted properly (tag @/ccmao1130 for docs review)

2025-06-17

proto aggregations

proto functions and columns

proto proto proto

2025-06-16

address pr factoring feedback

2025-06-14

updates project structure

2025-06-13

basic expr protos

create .run with the tron service

schema protos

2025-06-12

backup

creates daft-proto

2025-04-10

revert accidental change

The documentation is generated from this commit.

pr cleanup

2024-11-08

[FEAT] add spark-connect protocol (#3189) This PR adds a script-based approach for generating the Spark Connect protobuf code, eliminating the need for developers to have the protobuf compiler (protoc) installed locally. Key changes: - Added a new `spark-connect-script` crate that uses `tonic-build` to compile the .proto files into Rust code at build time - The script outputs the generated code as a string which can then be stored directly in the codebase - Added the generated Rust code to `src/spark-connect/src/lib.rs` - Updated workspace members to include both new crates - The proto files are copied from Spark's codebase and cover the full Connect protocol including base, catalog, commands, common, expressions, relations and types This approach makes it easier for developers to work with the Spark Connect protocol since they don't need to manage protoc installation and versioning. The generated code is checked into the repo and can be reviewed/updated as needed. Dependencies added: - prost: For protobuf encoding/decoding - prost-types: Common protobuf type definitions - tonic: For gRPC functionality

The documentation is generated from this commit.

2024-11-06

[FEAT] add spark-connect protocol

base daft connect

2024-10-14

split

fmt/etc

stash