Proto commits in delta-io/delta

These 9 commits are when the Protocol Buffers files have changed:

2025-05-09

[Spark] Add `vacuum`, `upgrade`, `generate`, `clone` command APIs to Delta Connect (Scala client + server) (#4516) <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md 2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP] Your PR title ...'. 3. Be sure to keep the PR description updated to reflect all changes. 4. Please write your PR title to summarize what this PR proposes. 5. If possible, provide a concise example to reproduce the issue for a faster review. 6. If applicable, include the corresponding issue number in the PR title and link it in the body. --> #### Which Delta project/connector is this regarding? <!-- Please add the component selected below to the beginning of the pull request title For example: [Spark] Title of my pull request --> - [x] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [] Other ## Description <!-- - Describe what this PR changes. - Describe why we need the change. If this PR resolves an issue be sure to include "Resolves #XXX" to correctly link and close the issue upon merge. --> - Adds the following new API to the Scala Delta Connect client : `clone`, `vacuum`, `generate` and `upgradeTableProtocol`. - Adds the proto definitions for the aforementioned APIs. - Adds the server-side implementation of the aforementioned APIs. ## How was this patch tested? <!-- If tests were added, say they were added here. Please make sure to test the changes thoroughly including negative and positive cases if possible. If the changes were tested in any way other than unit tests, please clarify how you tested step by step (ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future). If the changes were not tested, please explain why. --> New tests in `DeltaConnectPlannerSuite` (server-side changes) and in `DeltaTableSuite` (Note: these tests are set to `ignore` until https://github.com/delta-io/delta/pull/4468 is merged) ## Does this PR introduce _any_ user-facing changes? <!-- If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible. If possible, please also clarify if this is a user-facing change compared to the released Delta Lake versions or within the unreleased branches such as master. If no, write 'No'. --> Yes. Adds the `clone`, `vacuum`, `generate` and `upgradeTableProtocol` APIs to the Scala Delta Connect client.

The documentation is generated from this commit.

2024-08-22

Revert "[Spark] Add Delta Connect Merge Server and Scala Client" (#3592) Reverts delta-io/delta#3580 This PR looks like missing `DeltaMergeBuilder` which is causing build failures. See https://github.com/delta-io/delta/pull/3591

Revert "[Spark] Add Delta Connect Merge Server and Scala Client (#3580)" This reverts commit 0c188c2a3d1e798c0219888bc51a9979887f322f.

2024-08-21

[Spark] Add Delta Connect Merge Server and Scala Client (#3580) ## Description Add support for `merge` for Delta Connect Server and Scala Client. ## How was this patch tested? Added UTs. ## Does this PR introduce _any_ user-facing changes? No.

2024-08-19

[Spark] Add Delta Connect Update/Delete Server and Scala Client (#3545) <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md 2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP] Your PR title ...'. 3. Be sure to keep the PR description updated to reflect all changes. 4. Please write your PR title to summarize what this PR proposes. 5. If possible, provide a concise example to reproduce the issue for a faster review. 6. If applicable, include the corresponding issue number in the PR title and link it in the body. --> #### Which Delta project/connector is this regarding? <!-- Please add the component selected below to the beginning of the pull request title For example: [Spark] Title of my pull request --> - [x] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [ ] Other (fill in here) ## Description <!-- - Describe what this PR changes. - Describe why we need the change. If this PR resolves an issue be sure to include "Resolves #XXX" to correctly link and close the issue upon merge. --> Add support for `update` and `delete` for Delta Connect Server and Scala Client. ## How was this patch tested? Added UTs. <!-- If tests were added, say they were added here. Please make sure to test the changes thoroughly including negative and positive cases if possible. If the changes were tested in any way other than unit tests, please clarify how you tested step by step (ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future). If the changes were not tested, please explain why. --> ## Does this PR introduce _any_ user-facing changes? No. <!-- If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible. If possible, please also clarify if this is a user-facing change compared to the released Delta Lake versions or within the unreleased branches such as master. If no, write 'No'. --> --------- Co-authored-by: Dhruv Arya <dhruv.arya@databricks.com> Co-authored-by: Qianru Lao <55441375+EstherBear@users.noreply.github.com> Co-authored-by: Zihao Xu <xzhseh@gmail.com> Co-authored-by: Lukas Rupprecht <lukas.rupprecht@databricks.com> Co-authored-by: Venki Korukanti <venki.korukanti@gmail.com> Co-authored-by: jackierwzhang <67607237+jackierwzhang@users.noreply.github.com>

2024-08-14

[Spark] Add Delta Connect Describe History/Detail, Convert To Delta, Restore and isDeltaTable (#3490) <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md 2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP] Your PR title ...'. 3. Be sure to keep the PR description updated to reflect all changes. 4. Please write your PR title to summarize what this PR proposes. 5. If possible, provide a concise example to reproduce the issue for a faster review. 6. If applicable, include the corresponding issue number in the PR title and link it in the body. --> #### Which Delta project/connector is this regarding? <!-- Please add the component selected below to the beginning of the pull request title For example: [Spark] Title of my pull request --> - [x] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [ ] Other (fill in here) ## Description <!-- - Describe what this PR changes. - Describe why we need the change. If this PR resolves an issue be sure to include "Resolves #XXX" to correctly link and close the issue upon merge. --> Add support for `describeHistory/Detail`, `convertToDelta`, `RestoreTable` and `isDeltaTable` for Delta Connect Server and Scala Client. ## How was this patch tested? Added UTs. <!-- If tests were added, say they were added here. Please make sure to test the changes thoroughly including negative and positive cases if possible. If the changes were tested in any way other than unit tests, please clarify how you tested step by step (ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future). If the changes were not tested, please explain why. --> ## Does this PR introduce _any_ user-facing changes? No. <!-- If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible. If possible, please also clarify if this is a user-facing change compared to the released Delta Lake versions or within the unreleased branches such as master. If no, write 'No'. -->

2024-05-29

[Spark] Fix license headers for some Spark Connect proto files (#3170) ## Description - Adjusted the formatting of the license headers of some Spark Connect proto files. - Added also the Delta Lake license since even though we copied the Spark Connect proto files, we modified the `option java_package` of each proto files in [Spark Connect](https://github.com/apache/spark/tree/master/connector/connect/common/src/main/protobuf/spark/connect), similar to how it was done [here](https://github.com/delta-io/delta/blob/master/spark/src/main/scala/org/apache/spark/sql/delta/util/DateFormatter.scala) by this [PR](https://github.com/delta-io/delta/commit/e26435bcd787b232c1cf73eb118202971f1e18f1#diff-727ff4ed853adab6032cd9d06895dee225f15dec45fbcb0541676e77f459ef2e).

[Spark] Adding Spark Connect Protobuf Messages to the Delta repository (#3168) ## Description Delta Connect is an extension for Spark Connect that adds support for the DeltaTable API. To be able to communicate back to Spark Connect we need to have access to Spark Connect's Protobufs. Since Buf does not support local dependencies, while Delta and Apache Spark are two separate repositories, the only solution for Delta Connect to get access to [Spark Connect's Protobuf messages](https://github.com/apache/spark/tree/master/connector/connect/common/src/main/protobuf/spark/connect) is to have the copies of the messages in the Delta repository. This may go out of sync, and we would need to update it manually from the Apache Spark repository, but this should be fine and would not break anything since Protobufs are backward compatible. ## How was this patch tested? N/A.

[Spark] Set up Python Protobuf codegen for Delta Connect (#3125) ## Description Added the very first protobuf messages for `DeltaTable`, `Clone` and `Scan`. This is the first PR for Delta Connect, which adds support for Delta's `DeltaTable` interface to Spark Connect. This is needed to support Delta table operations outside of SQL directly on Spark Connect clients. This PR sets up the [Python code generation for the Protobufs of Delta Connect](https://protobuf.dev/getting-started/pythontutorial/). For this I created a new Buf workspace and I added a few initial Protobuf messages to confirm that everything works. This is the ground work of the project, before we move on to setting up the server and client library. What we are doing here is similar to the [Spark Connect's protobuf development guide](https://github.com/apache/spark/tree/master/connector/connect). ## How was this patch tested? Added the `check-delta-connect-codegen-python.py` to the automated testing, making sure the Python Protobuf Generated codes stay in sync with the proto messages.