Proto commits in snowflakedb/snowpark-python

These 62 commits are when the Protocol Buffers files have changed:

2025-04-30

SNOW-2062273: Remove artifact_repository_packages parameter

2025-04-23

SNOW-2036949: Sync artifact repositry ast changes (#3293)

The documentation is generated from this commit.

2025-04-22

SNOW-2036949: Sync artifact repositry ast changes

The documentation is generated from this commit.

2025-04-03

NOSNOW: Update ast.proto (#3214)

2025-03-28

SNOW-1821290: Finalize AST schema v1 by refactoring Assign to Bind, VarId to BindId (#3183) 1. Which Jira issue is this PR addressing? Make sure that there is an accompanying issue to your PR. [SNOW-1821290](https://snowflakecomputing.atlassian.net/browse/SNOW-1821290): Finalize AST schema v1 by refactoring `Assign` to `Bind`, `VarId` to `BindId` 2. Fill out the following pre-review checklist: - [x] I am adding a new automated test(s) to verify correctness of my new code - [ ] If this test skips Local Testing mode, I'm requesting review from @snowflakedb/local-testing - [ ] I am adding new logging messages - [ ] I am adding a new telemetry message - [ ] I am adding new credentials - [ ] I am adding a new dependency - [ ] If this is a new feature/behavior, I'm adding the Local Testing parity changes. - [x] I acknowledge that I have ensured my changes to be thread-safe. Follow the link for more information: [Thread-safe Developer Guidelines](https://github.com/snowflakedb/snowpark-python/blob/main/CONTRIBUTING.md#thread-safe-development) - [x] If adding any arguments to public Snowpark APIs or creating new public Snowpark APIs, I acknowledge that I have ensured my changes include AST support. 3. Please describe how your code solves the related issue. To reflect the new semantics, `AstBatch.assign` was refactored to `AstBatch.bind` and its usage was updated across the client. Usage of `var_id` fields was replaced with `bind_id` fields as well. `AstBatch._init_batch` now provides the UUID to the newly created `Request` message in its binary 16 byte representation provided by `UUID.bytes` under the `Request.request_id` field. `AstBatch.bind` will also take the current `AstBatch._request_id` and fill `Bind.first_request_id` with the same binary 16 byte representation from `UUID.bytes` to keep track of the first `Request` message which this statement will be flushed to the server in. This PR is accompanied by a [server-side PR](https://github.com/snowflakedb/snowflake/pull/270978). [SNOW-1821290]: https://snowflakecomputing.atlassian.net/browse/SNOW-1821290?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ

SNOW-1882075,SNOW-1990301,SNOW-1990302,SNOW-1821290: AST finalizations for stateless batches, and DataframeWriter,DataframeReader new APIs (#3180) 1. Which Jira issue is this PR addressing? Make sure that there is an accompanying issue to your PR. This PR includes all changes required to complete the following tickets. - [SNOW-1882075](https://snowflakecomputing.atlassian.net/browse/SNOW-1882075): AST for DataFrameWriter.insertInto/insert_into - [SNOW-1990301](https://snowflakecomputing.atlassian.net/browse/SNOW-1990301): AST for DataframerWriter.{format,save} - [SNOW-1990302](https://snowflakecomputing.atlassian.net/browse/SNOW-1990302): AST for DataframeReader.{format,load} - [SNOW-1821290](https://snowflakecomputing.atlassian.net/browse/SNOW-1821290): Finalize the AST schema v1 It also includes changes in support of [SNOW-1997136](https://snowflakecomputing.atlassian.net/browse/SNOW-1997136): Ensure AST batch is stateless. 2. Fill out the following pre-review checklist: - [x] I am adding a new automated test(s) to verify correctness of my new code - [ ] If this test skips Local Testing mode, I'm requesting review from @snowflakedb/local-testing - [ ] I am adding new logging messages - [ ] I am adding a new telemetry message - [ ] I am adding new credentials - [ ] I am adding a new dependency - [ ] If this is a new feature/behavior, I'm adding the Local Testing parity changes. - [x] I acknowledge that I have ensured my changes to be thread-safe. Follow the link for more information: [Thread-safe Developer Guidelines](https://github.com/snowflakedb/snowpark-python/blob/main/CONTRIBUTING.md#thread-safe-development) - [x] If adding any arguments to public Snowpark APIs or creating new public Snowpark APIs, I acknowledge that I have ensured my changes include AST support. 3. Please describe how your code solves the related issue. AST generation for `DataframeReader` APIs which return the same `DataframeReader` instance have been updated to follow the same pattern as `DataframeWriter`. Since all such methods are setters, it is only necessary to have the last set attributes in the AST sent to the serveer. This is in support of the new `format` API in both Snowpark classes. AST generation was added for the new APIs `DataframeReader.{format,load}`, and `DataframeWriter.{format,read,insert_into}` along with expectation test updates. Changes were also made in support of AST batch statelessness in the IR, by changing fields typed `VarId` to be `Expr` typed. This allows more consistent usage of `_set_ast_ref` as a helper method across all Snowpark APIs. This PR is accompanied by a [server-side PR](https://github.com/snowflakedb/snowflake/pull/270350). [SNOW-1882075]: https://snowflakecomputing.atlassian.net/browse/SNOW-1882075?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ [SNOW-1990301]: https://snowflakecomputing.atlassian.net/browse/SNOW-1990301?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ [SNOW-1990302]: https://snowflakecomputing.atlassian.net/browse/SNOW-1990302?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ [SNOW-1821290]: https://snowflakecomputing.atlassian.net/browse/SNOW-1821290?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ [SNOW-1997136]: https://snowflakecomputing.atlassian.net/browse/SNOW-1997136?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ

2025-03-21

SNOW-1995618 Store ref to temp table created in cache_result in AST (#3181) To enable server-side execution, cache_result presents a duality as it effectively is a crossover of eval/assign. We resolve the existing TODO by treating cache_result as an expr together with an assign. Client-side optionally a temp table is created, and then stored within the expr. This will allow server-side to skip any nodes that formed lineage to the cache_result expr. Decoder changes done in monorepo/followup once decoder is refreshed to 1.29.

2025-03-19

SNOW-1916224: Ensure that all APIs have AST support (#3170)

2025-03-13

NO-SNOW: release 1.29.1 merge back to main (#3153) (#3155)

SNOW-1918334: Various fixes to capture user code more accurately (#3132)

2025-03-12

NO-SNOW: release 1.29.1 preparation (#3153)

NO-SNOW: release update (#3150)

2025-03-11

NO-SNOW: release update (#3150)

2025-02-28

SNOW-1941023: Remove optionality for collections (#3076) 1. Which Jira issue is this PR addressing? Make sure that there is an accompanying issue to your PR. [SNOW-1941023](https://snowflakecomputing.atlassian.net/browse/SNOW-1941023) 2. Fill out the following pre-review checklist: - [x] I am adding a new automated test(s) to verify correctness of my new code - [ ] If this test skips Local Testing mode, I'm requesting review from @snowflakedb/local-testing - [ ] I am adding new logging messages - [ ] I am adding a new telemetry message - [ ] I am adding new credentials - [ ] I am adding a new dependency - [ ] If this is a new feature/behavior, I'm adding the Local Testing parity changes. - [x] I acknowledge that I have ensured my changes to be thread-safe. Follow the link for more information: [Thread-safe Developer Guidelines](https://github.com/snowflakedb/snowpark-python/blob/main/CONTRIBUTING.md#thread-safe-development) 3. Please describe how your code solves the related issue. This PR contains the necessary changes to ensure correct AST generation after removing optionality from all `Option[List[T]]` and `Option[Map[T1, T2]]` type fields. We use an empty collection as an indicator of a missing value. Slight modifications were made to make `StructType` AST building more concise. It also removes the `eval` statement generation from `dataframe.stat.sample_by` which is not an eager API. This PR is accompanied by [a server-side PR](https://github.com/snowflakedb/snowflake/pull/261101). [SNOW-1941023]: https://snowflakecomputing.atlassian.net/browse/SNOW-1941023?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ

SNOW-1941024: Unify DataframeSelect variants and DataframeUnion variants (#3069) 1. Which Jira issue is this PR addressing? Make sure that there is an accompanying issue to your PR. [SNOW-1941024](https://snowflakecomputing.atlassian.net/browse/SNOW-1941024) [SNOW-1936164](https://snowflakecomputing.atlassian.net/browse/SNOW-1936164) 2. Fill out the following pre-review checklist: - [x] I am adding a new automated test(s) to verify correctness of my new code - [ ] If this test skips Local Testing mode, I'm requesting review from @snowflakedb/local-testing - [ ] I am adding new logging messages - [ ] I am adding a new telemetry message - [ ] I am adding new credentials - [ ] I am adding a new dependency - [ ] If this is a new feature/behavior, I'm adding the Local Testing parity changes. - [x] I acknowledge that I have ensured my changes to be thread-safe. Follow the link for more information: [Thread-safe Developer Guidelines](https://github.com/snowflakedb/snowpark-python/blob/main/CONTRIBUTING.md#thread-safe-development) 3. Please describe how your code solves the related issue. This PR contains the client side changes needed to unify the `DataframeSelect_{Columns,Exprs}` AST entities as well as the `DataframeUnion{ByName,All,AllByName}` AST entities. Also adds new `allow_missing_columns` parameter to the `DataframeUnion` AST and expectation tests for this. Expectation tests have also been updated and this PR is accompanied by this [server-side PR](https://github.com/snowflakedb/snowflake/pull/260468). [SNOW-1941024]: https://snowflakecomputing.atlassian.net/browse/SNOW-1941024?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ [SNOW-1936164]: https://snowflakecomputing.atlassian.net/browse/SNOW-1936164?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ

2025-02-26

SNOW-1918329: Remove specialized Expr subtraits (#3066) 1. Which Jira issue is this PR addressing? Make sure that there is an accompanying issue to your PR. [SNOW-1918329](https://snowflakecomputing.atlassian.net/browse/SNOW-1918329) 2. Fill out the following pre-review checklist: - [x] I am adding a new automated test(s) to verify correctness of my new code - [ ] If this test skips Local Testing mode, I'm requesting review from @snowflakedb/local-testing - [ ] I am adding new logging messages - [ ] I am adding a new telemetry message - [ ] I am adding new credentials - [ ] I am adding a new dependency - [ ] If this is a new feature/behavior, I'm adding the Local Testing parity changes. - [x] I acknowledge that I have ensured my changes to be thread-safe. Follow the link for more information: [Thread-safe Developer Guidelines](https://github.com/snowflakedb/snowpark-python/blob/main/CONTRIBUTING.md#thread-safe-development) 3. Please describe how your code solves the related issue. This PR removes the following specialized `Expr` subtraits, and updates the Unparser to continue handling the roundtrip Expectation tests from the client repo. - RelationalGroupedDataframeExpr - DataframeReader - DataframeWriter - DataframeExpr - FnRefExpr - MatchedClause - Regarding the removed TODO, I believe the extensions of this subtrait should be `Expr`s as they can be constructed independently and assigned. The above met all of the following criteria to be candidates for removal - The trait doesn't carry any data of its own (is empty), and removing the trait doesn't introduce repetition all over the place. - No code ever dispatches on the intermediate trait (i.e. subset the matches based on the intermediate empty trait). - The removal doesn't introduce ambiguity where none used to exist. The changes are accompanied by this [monorepo PR](https://github.com/snowflakedb/snowflake/pull/259834) [SNOW-1918329]: https://snowflakecomputing.atlassian.net/browse/SNOW-1918329?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ

2025-02-24

Add AST changes

2025-02-21

SNOW-1821290: Drop Sp prefix from all AST entities (#3038)

2025-02-17

NOSNOW - Fix Snowpark AST Tests (#3023)

2025-02-15

NOSNOW - Add `n` to the recorded Dataframe.show AST, add Dataframe.show test (#3012)

NOSNOW - Add `n` to the recorded Dataframe.show AST, add Dataframe.show test (#3012)

2025-02-14

SNOW-1918334: Fix missing ColumnIdentifier calls (#3016)

SNOW-1918334: Fix missing ColumnIdentifier calls (#3016)

2025-02-12

SNOW-1918331: Replace all uses of variadic fields with ExprArgList (#3001)

SNOW-1918331: Replace all uses of variadic fields with ExprArgList (#3001)

2025-02-06

Capture the AST unparser STDERR on failure (#2978)

Capture the AST unparser STDERR on failure (#2978)

2025-01-28

SNOW-1794510 refactor decoder (#2909) 1. Which Jira issue is this PR addressing? Make sure that there is an accompanying issue to your PR. Fixes SNOW-1794510 2. Fill out the following pre-review checklist: - [ ] I am adding a new automated test(s) to verify correctness of my new code - [ ] If this test skips Local Testing mode, I'm requesting review from @snowflakedb/local-testing - [ ] I am adding new logging messages - [ ] I am adding a new telemetry message - [ ] I am adding new credentials - [ ] I am adding a new dependency - [ ] If this is a new feature/behavior, I'm adding the Local Testing parity changes. - [x] I acknowledge that I have ensured my changes to be thread-safe. Follow the link for more information: [Thread-safe Developer Guidelines](https://github.com/snowflakedb/snowpark-python/blob/main/CONTRIBUTING.md#thread-safe-development) 3. Please describe how your code solves the related issue. Added docstrings and rearranged the cases to be in alphabetical order.

2025-01-27

changes for fixing sp_struct_type decoding (#2942) <!--- Please answer these questions before creating your pull request. Thanks! ---> 1. Which Jira issue is this PR addressing? Make sure that there is an accompanying issue to your PR. <!--- In this section, please add a Snowflake Jira issue number. Note that if a corresponding GitHub issue exists, you should still include the Snowflake Jira issue number. For example, for GitHub issue https://github.com/snowflakedb/snowpark-python/issues/1400, you should add "SNOW-1335071" here. ---> Fixes SNOW-NNNNNNN 2. Fill out the following pre-review checklist: - [ ] I am adding a new automated test(s) to verify correctness of my new code - [ ] If this test skips Local Testing mode, I'm requesting review from @snowflakedb/local-testing - [ ] I am adding new logging messages - [ ] I am adding a new telemetry message - [ ] I am adding new credentials - [ ] I am adding a new dependency - [ ] If this is a new feature/behavior, I'm adding the Local Testing parity changes. - [ ] I acknowledge that I have ensured my changes to be thread-safe. Follow the link for more information: [Thread-safe Developer Guidelines](https://github.com/snowflakedb/snowpark-python/blob/main/CONTRIBUTING.md#thread-safe-development) 3. Please describe how your code solves the related issue. Please write a short description of how your code change solves the related issue.

2025-01-22

SNOW-1890485 string interning for filenames (#2898) 1. Which Jira issue is this PR addressing? Make sure that there is an accompanying issue to your PR. Fixes SNOW-1890485 2. Fill out the following pre-review checklist: - [ ] I am adding a new automated test(s) to verify correctness of my new code - [ ] If this test skips Local Testing mode, I'm requesting review from @snowflakedb/local-testing - [ ] I am adding new logging messages - [ ] I am adding a new telemetry message - [ ] I am adding new credentials - [ ] I am adding a new dependency - [ ] If this is a new feature/behavior, I'm adding the Local Testing parity changes. - [x] I acknowledge that I have ensured my changes to be thread-safe. Follow the link for more information: [Thread-safe Developer Guidelines](https://github.com/snowflakedb/snowpark-python/blob/main/CONTRIBUTING.md#thread-safe-development) 3. Please describe how your code solves the related issue. For any entity encoded using the AST typically with_src_position is called which stores the filename of the origin where a Snowpark API is called. Likely, there are only a few distinct files but many API calls happen in a single file typically. Encoding for any entity the full filepath leads to a bloated base64 message. To reduce the size, rather the file location is encoded with an integer that represents a lookup into an interned value table. The IR DSL compiler has been previously expanded to provide functionality to allow for interning of values. Modifications for the IR to intern the filename are done in https://github.com/snowflakedb/snowflake/tree/ls-SNOW-1890485.

SNOW-1890485 string interning for filenames (#2898) 1. Which Jira issue is this PR addressing? Make sure that there is an accompanying issue to your PR. Fixes SNOW-1890485 2. Fill out the following pre-review checklist: - [ ] I am adding a new automated test(s) to verify correctness of my new code - [ ] If this test skips Local Testing mode, I'm requesting review from @snowflakedb/local-testing - [ ] I am adding new logging messages - [ ] I am adding a new telemetry message - [ ] I am adding new credentials - [ ] I am adding a new dependency - [ ] If this is a new feature/behavior, I'm adding the Local Testing parity changes. - [x] I acknowledge that I have ensured my changes to be thread-safe. Follow the link for more information: [Thread-safe Developer Guidelines](https://github.com/snowflakedb/snowpark-python/blob/main/CONTRIBUTING.md#thread-safe-development) 3. Please describe how your code solves the related issue. For any entity encoded using the AST typically with_src_position is called which stores the filename of the origin where a Snowpark API is called. Likely, there are only a few distinct files but many API calls happen in a single file typically. Encoding for any entity the full filepath leads to a bloated base64 message. To reduce the size, rather the file location is encoded with an integer that represents a lookup into an interned value table. The IR DSL compiler has been previously expanded to provide functionality to allow for interning of values. Modifications for the IR to intern the filename are done in https://github.com/snowflakedb/snowflake/tree/ls-SNOW-1890485.

2025-01-17

SNOW-1830541 Decoder functionality for functions1.test and functions2.test (#2881) 1. Which Jira issue is this PR addressing? Make sure that there is an accompanying issue to your PR. Fixes SNOW-1830541 2. Fill out the following pre-review checklist: - [ ] I am adding a new automated test(s) to verify correctness of my new code - [ ] If this test skips Local Testing mode, I'm requesting review from @snowflakedb/local-testing - [ ] I am adding new logging messages - [ ] I am adding a new telemetry message - [ ] I am adding new credentials - [ ] I am adding a new dependency - [ ] If this is a new feature/behavior, I'm adding the Local Testing parity changes. - [x] I acknowledge that I have ensured my changes to be thread-safe. Follow the link for more information: [Thread-safe Developer Guidelines](https://github.com/snowflakedb/snowpark-python/blob/main/CONTRIBUTING.md#thread-safe-development) 3. Please describe how your code solves the related issue. Minor changes to the decoder to make functions1.test and functions2.test work.

SNOW-1873529: Sync structured type AST changes (#2870)

SNOW-1873529: Sync structured type AST changes (#2870)

2025-01-16

SNOW-1830541 update decoder branch with main (#2874) 1. Which Jira issue is this PR addressing? Make sure that there is an accompanying issue to your PR. Fixes SNOW-1830541 2. Fill out the following pre-review checklist: - [ ] I am adding a new automated test(s) to verify correctness of my new code - [ ] If this test skips Local Testing mode, I'm requesting review from @snowflakedb/local-testing - [ ] I am adding new logging messages - [ ] I am adding a new telemetry message - [ ] I am adding new credentials - [ ] I am adding a new dependency - [ ] If this is a new feature/behavior, I'm adding the Local Testing parity changes. - [x] I acknowledge that I have ensured my changes to be thread-safe. Follow the link for more information: [Thread-safe Developer Guidelines](https://github.com/snowflakedb/snowpark-python/blob/main/CONTRIBUTING.md#thread-safe-development) 3. Please describe how your code solves the related issue. Update decoder branch with main. --------- Signed-off-by: Labanya Mukhopadhyay <labanya.mukhopadhyay@snowflake.com> Co-authored-by: Naresh Kumar <113932371+sfc-gh-nkumar@users.noreply.github.com> Co-authored-by: Mahesh Vashishtha <mahesh.vashishtha@snowflake.com> Co-authored-by: Afroz Alam <afroz.alam@snowflake.com> Co-authored-by: Jianzhun Du <68252326+sfc-gh-jdu@users.noreply.github.com> Co-authored-by: Jonathan Shi <149419494+sfc-gh-joshi@users.noreply.github.com> Co-authored-by: Jamison Rose <Jamison.Rose@snowflake.com> Co-authored-by: Hazem Elmeleegy <hazem.elmeleegy@snowflake.com> Co-authored-by: Labanya Mukhopadhyay <labanya.mukhopadhyay@snowflake.com> Co-authored-by: Adam Ling <adam.ling@snowflake.com> Co-authored-by: Leonhard Spiegelberg <leonhard.spiegelberg@snowflake.com> Co-authored-by: Balachandran Atur <bala.atur@snowflake.com> Co-authored-by: Yuyang Wang <yuyang.wang@snowflake.com> Co-authored-by: Rehan Durrani <rehan.durrani@snowflake.com> Co-authored-by: Zihan Li <63482288+sfc-gh-zli@users.noreply.github.com>

2025-01-14

SNOW-1800374: adding support for options and partition_by to dataframewriter (#2841) <!--- Please answer these questions before creating your pull request. Thanks! ---> 1. Which Jira issue is this PR addressing? Make sure that there is an accompanying issue to your PR. <!--- In this section, please add a Snowflake Jira issue number. Note that if a corresponding GitHub issue exists, you should still include the Snowflake Jira issue number. For example, for GitHub issue https://github.com/snowflakedb/snowpark-python/issues/1400, you should add "SNOW-1335071" here. ---> Fixes SNOW-1800374 2. Fill out the following pre-review checklist: - [x] I am adding a new automated test(s) to verify correctness of my new code - [ ] If this test skips Local Testing mode, I'm requesting review from @snowflakedb/local-testing - [ ] I am adding new logging messages - [ ] I am adding a new telemetry message - [ ] I am adding new credentials - [ ] I am adding a new dependency - [ ] If this is a new feature/behavior, I'm adding the Local Testing parity changes. - [x] I acknowledge that I have ensured my changes to be thread-safe. Follow the link for more information: [Thread-safe Developer Guidelines](https://github.com/snowflakedb/snowpark-python/blob/main/CONTRIBUTING.md#thread-safe-development) 3. Please describe how your code solves the related issue. Please write a short description of how your code change solves the related issue.

SNOW-1800374: adding support for options and partition_by to dataframewriter (#2841) <!--- Please answer these questions before creating your pull request. Thanks! ---> 1. Which Jira issue is this PR addressing? Make sure that there is an accompanying issue to your PR. <!--- In this section, please add a Snowflake Jira issue number. Note that if a corresponding GitHub issue exists, you should still include the Snowflake Jira issue number. For example, for GitHub issue https://github.com/snowflakedb/snowpark-python/issues/1400, you should add "SNOW-1335071" here. ---> Fixes SNOW-1800374 2. Fill out the following pre-review checklist: - [x] I am adding a new automated test(s) to verify correctness of my new code - [ ] If this test skips Local Testing mode, I'm requesting review from @snowflakedb/local-testing - [ ] I am adding new logging messages - [ ] I am adding a new telemetry message - [ ] I am adding new credentials - [ ] I am adding a new dependency - [ ] If this is a new feature/behavior, I'm adding the Local Testing parity changes. - [x] I acknowledge that I have ensured my changes to be thread-safe. Follow the link for more information: [Thread-safe Developer Guidelines](https://github.com/snowflakedb/snowpark-python/blob/main/CONTRIBUTING.md#thread-safe-development) 3. Please describe how your code solves the related issue. Please write a short description of how your code change solves the related issue.

2025-01-13

SNOW-1794509 Merging from main into decoder branch. (#2826) <!--- Please answer these questions before creating your pull request. Thanks! ---> 1. Which Jira issue is this PR addressing? Make sure that there is an accompanying issue to your PR. <!--- In this section, please add a Snowflake Jira issue number. Note that if a corresponding GitHub issue exists, you should still include the Snowflake Jira issue number. For example, for GitHub issue https://github.com/snowflakedb/snowpark-python/issues/1400, you should add "SNOW-1335071" here. ---> Fixes SNOW-1794509 2. Fill out the following pre-review checklist: - [ ] I am adding a new automated test(s) to verify correctness of my new code - [ ] If this test skips Local Testing mode, I'm requesting review from @snowflakedb/local-testing - [ ] I am adding new logging messages - [ ] I am adding a new telemetry message - [ ] I am adding new credentials - [ ] I am adding a new dependency - [ ] If this is a new feature/behavior, I'm adding the Local Testing parity changes. - [x] I acknowledge that I have ensured my changes to be thread-safe. Follow the link for more information: [Thread-safe Developer Guidelines](https://github.com/snowflakedb/snowpark-python/blob/main/CONTRIBUTING.md#thread-safe-development) 3. Please describe how your code solves the related issue. Please write a short description of how your code change solves the related issue. --------- Signed-off-by: sfc-gh-mvashishtha <mahesh.vashishtha@snowflake.com> Co-authored-by: Jonathan Shi <149419494+sfc-gh-joshi@users.noreply.github.com> Co-authored-by: Afroz Alam <afroz.alam@snowflake.com> Co-authored-by: Hazem Elmeleegy <hazem.elmeleegy@snowflake.com> Co-authored-by: Varnika Budati <varnika.budati@snowflake.com> Co-authored-by: Mahesh Vashishtha <mahesh.vashishtha@snowflake.com> Co-authored-by: Jamison Rose <Jamison.Rose@snowflake.com> Co-authored-by: Ovidiu Platon <ovidiu.platon@snowflake.com> Co-authored-by: Naresh Kumar <113932371+sfc-gh-nkumar@users.noreply.github.com>

2025-01-06

Unify Snowflake object name handling in the Snowpark AST (#2789) <!--- Please answer these questions before creating your pull request. Thanks! ---> 1. Which Jira issue is this PR addressing? Make sure that there is an accompanying issue to your PR. <!--- In this section, please add a Snowflake Jira issue number. Note that if a corresponding GitHub issue exists, you should still include the Snowflake Jira issue number. For example, for GitHub issue https://github.com/snowflakedb/snowpark-python/issues/1400, you should add "SNOW-1335071" here. ---> Fixes SNOW-1621205 2. Fill out the following pre-review checklist: - [ ] I am adding a new automated test(s) to verify correctness of my new code - [ ] If this test skips Local Testing mode, I'm requesting review from @snowflakedb/local-testing - [ ] I am adding new logging messages - [ ] I am adding a new telemetry message - [ ] I am adding new credentials - [ ] I am adding a new dependency - [ ] If this is a new feature/behavior, I'm adding the Local Testing parity changes. - [x] I acknowledge that I have ensured my changes to be thread-safe. Follow the link for more information: [Thread-safe Developer Guidelines](https://github.com/snowflakedb/snowpark-python/blob/main/CONTRIBUTING.md#thread-safe-development) 3. Please describe how your code solves the related issue. This is the client-side change corresponding to https://github.com/snowflakedb/snowflake/pull/240557 Unify Snowflake object name handling in the Snowpark AST. Remove `FnName` and `SpTableName`. They both had `Flat` and `Structured` variants, but ultimately designate Snowflake object names. Introduce `data SpName` and `entity SpNameRef` for referring to Snowflake objects by relative or fully qualified name. Update `FnNameRefExpr`. Use `SpNameRef` in a few places that used to use `List[String]`.

Unify Snowflake object name handling in the Snowpark AST (#2789) <!--- Please answer these questions before creating your pull request. Thanks! ---> 1. Which Jira issue is this PR addressing? Make sure that there is an accompanying issue to your PR. <!--- In this section, please add a Snowflake Jira issue number. Note that if a corresponding GitHub issue exists, you should still include the Snowflake Jira issue number. For example, for GitHub issue https://github.com/snowflakedb/snowpark-python/issues/1400, you should add "SNOW-1335071" here. ---> Fixes SNOW-1621205 2. Fill out the following pre-review checklist: - [ ] I am adding a new automated test(s) to verify correctness of my new code - [ ] If this test skips Local Testing mode, I'm requesting review from @snowflakedb/local-testing - [ ] I am adding new logging messages - [ ] I am adding a new telemetry message - [ ] I am adding new credentials - [ ] I am adding a new dependency - [ ] If this is a new feature/behavior, I'm adding the Local Testing parity changes. - [x] I acknowledge that I have ensured my changes to be thread-safe. Follow the link for more information: [Thread-safe Developer Guidelines](https://github.com/snowflakedb/snowpark-python/blob/main/CONTRIBUTING.md#thread-safe-development) 3. Please describe how your code solves the related issue. This is the client-side change corresponding to https://github.com/snowflakedb/snowflake/pull/240557 Unify Snowflake object name handling in the Snowpark AST. Remove `FnName` and `SpTableName`. They both had `Flat` and `Structured` variants, but ultimately designate Snowflake object names. Introduce `data SpName` and `entity SpNameRef` for referring to Snowflake objects by relative or fully qualified name. Update `FnNameRefExpr`. Use `SpNameRef` in a few places that used to use `List[String]`.

2024-12-20

SNOW-1866100 Update DataFrame.unpivot AST encoding to include `include_nulls` (#2800) 1. Which Jira issue is this PR addressing? Make sure that there is an accompanying issue to your PR. Fixes SNOW-1866100 2. Fill out the following pre-review checklist: - [ ] I am adding a new automated test(s) to verify correctness of my new code - [ ] If this test skips Local Testing mode, I'm requesting review from @snowflakedb/local-testing - [ ] I am adding new logging messages - [ ] I am adding a new telemetry message - [ ] I am adding new credentials - [ ] I am adding a new dependency - [ ] If this is a new feature/behavior, I'm adding the Local Testing parity changes. - [x] I acknowledge that I have ensured my changes to be thread-safe. Follow the link for more information: [Thread-safe Developer Guidelines](https://github.com/snowflakedb/snowpark-python/blob/main/CONTRIBUTING.md#thread-safe-development) 3. Please describe how your code solves the related issue. Update DataFrame.unpivot AST encoding to include `include_nulls`

SNOW-1866100 Update DataFrame.unpivot AST encoding to include `include_nulls` (#2800) 1. Which Jira issue is this PR addressing? Make sure that there is an accompanying issue to your PR. Fixes SNOW-1866100 2. Fill out the following pre-review checklist: - [ ] I am adding a new automated test(s) to verify correctness of my new code - [ ] If this test skips Local Testing mode, I'm requesting review from @snowflakedb/local-testing - [ ] I am adding new logging messages - [ ] I am adding a new telemetry message - [ ] I am adding new credentials - [ ] I am adding a new dependency - [ ] If this is a new feature/behavior, I'm adding the Local Testing parity changes. - [x] I acknowledge that I have ensured my changes to be thread-safe. Follow the link for more information: [Thread-safe Developer Guidelines](https://github.com/snowflakedb/snowpark-python/blob/main/CONTRIBUTING.md#thread-safe-development) 3. Please describe how your code solves the related issue. Update DataFrame.unpivot AST encoding to include `include_nulls`

2024-12-18

Merge from main 2 (#2778) <!--- Please answer these questions before creating your pull request. Thanks! ---> 1. Which Jira issue is this PR addressing? Make sure that there is an accompanying issue to your PR. <!--- In this section, please add a Snowflake Jira issue number. Note that if a corresponding GitHub issue exists, you should still include the Snowflake Jira issue number. For example, for GitHub issue https://github.com/snowflakedb/snowpark-python/issues/1400, you should add "SNOW-1335071" here. ---> Fixes SNOW-NNNNNNN 2. Fill out the following pre-review checklist: - [ ] I am adding a new automated test(s) to verify correctness of my new code - [ ] If this test skips Local Testing mode, I'm requesting review from @snowflakedb/local-testing - [ ] I am adding new logging messages - [ ] I am adding a new telemetry message - [ ] I am adding new credentials - [ ] I am adding a new dependency - [ ] If this is a new feature/behavior, I'm adding the Local Testing parity changes. - [ ] I acknowledge that I have ensured my changes to be thread-safe. Follow the link for more information: [Thread-safe Developer Guidelines](https://github.com/snowflakedb/snowpark-python/blob/main/CONTRIBUTING.md#thread-safe-development) 3. Please describe how your code solves the related issue. Please write a short description of how your code change solves the related issue. --------- Signed-off-by: sfc-gh-mvashishtha <mahesh.vashishtha@snowflake.com> Signed-off-by: Labanya Mukhopadhyay <labanya.mukhopadhyay@snowflake.com> Co-authored-by: Ovidiu Platon <ovidiu.platon@snowflake.com> Co-authored-by: Andong Zhan <andong.zhan@snowflake.com> Co-authored-by: Leonhard Spiegelberg <leonhard.spiegelberg@snowflake.com> Co-authored-by: Adam Ling <adam.ling@snowflake.com> Co-authored-by: Mahesh Vashishtha <mahesh.vashishtha@snowflake.com> Co-authored-by: Rehan Durrani <rehan.durrani@snowflake.com> Co-authored-by: Jamison Rose <Jamison.Rose@snowflake.com> Co-authored-by: Afroz Alam <afroz.alam@snowflake.com> Co-authored-by: Yun Zou <yun.zou@snowflake.com> Co-authored-by: Hazem Elmeleegy <hazem.elmeleegy@snowflake.com> Co-authored-by: Varnika Budati <varnika.budati@snowflake.com> Co-authored-by: Arthur Zwiegincew <arthur.zwiegincew@snowflake.com> Co-authored-by: Labanya Mukhopadhyay <labanya.mukhopadhyay@snowflake.com> Co-authored-by: Naresh Kumar <113932371+sfc-gh-nkumar@users.noreply.github.com> Co-authored-by: Himanshu Ojha <himanshu.ojha@snowflake.com>

2024-12-16

SNOW-1852779 Fix AST encoding for Column `in_`, `asc`, and `desc` (#2756) 1. Which Jira issue is this PR addressing? Make sure that there is an accompanying issue to your PR. Fixes SNOW-1852779 2. Fill out the following pre-review checklist: - [ ] I am adding a new automated test(s) to verify correctness of my new code - [ ] If this test skips Local Testing mode, I'm requesting review from @snowflakedb/local-testing - [ ] I am adding new logging messages - [ ] I am adding a new telemetry message - [ ] I am adding new credentials - [ ] I am adding a new dependency - [ ] If this is a new feature/behavior, I'm adding the Local Testing parity changes. - [x] I acknowledge that I have ensured my changes to be thread-safe. Follow the link for more information: [Thread-safe Developer Guidelines](https://github.com/snowflakedb/snowpark-python/blob/main/CONTRIBUTING.md#thread-safe-development) 3. Please describe how your code solves the related issue. Fixed AST encoding for Column `in_`, `asc`, and `desc`. Removed an unused entity and renamed `sp_column_in__seq` to `sp_column_in`. Changed the `nulls_first` optional boolean param to be an enum.

SNOW-1852779 Fix AST encoding for Column `in_`, `asc`, and `desc` (#2756) 1. Which Jira issue is this PR addressing? Make sure that there is an accompanying issue to your PR. Fixes SNOW-1852779 2. Fill out the following pre-review checklist: - [ ] I am adding a new automated test(s) to verify correctness of my new code - [ ] If this test skips Local Testing mode, I'm requesting review from @snowflakedb/local-testing - [ ] I am adding new logging messages - [ ] I am adding a new telemetry message - [ ] I am adding new credentials - [ ] I am adding a new dependency - [ ] If this is a new feature/behavior, I'm adding the Local Testing parity changes. - [x] I acknowledge that I have ensured my changes to be thread-safe. Follow the link for more information: [Thread-safe Developer Guidelines](https://github.com/snowflakedb/snowpark-python/blob/main/CONTRIBUTING.md#thread-safe-development) 3. Please describe how your code solves the related issue. Fixed AST encoding for Column `in_`, `asc`, and `desc`. Removed an unused entity and renamed `sp_column_in__seq` to `sp_column_in`. Changed the `nulls_first` optional boolean param to be an enum.

2024-12-12

SNOW-1672579 Encode `DataFrame.to_snowpark_pandas` (#2711) 1. Which Jira issue is this PR addressing? Make sure that there is an accompanying issue to your PR. Fixes SNOW-1672579 2. Fill out the following pre-review checklist: - [x] I am adding a new automated test(s) to verify correctness of my new code - [ ] If this test skips Local Testing mode, I'm requesting review from @snowflakedb/local-testing - [ ] I am adding new logging messages - [ ] I am adding a new telemetry message - [ ] I am adding new credentials - [ ] I am adding a new dependency - [ ] If this is a new feature/behavior, I'm adding the Local Testing parity changes. - [x] I acknowledge that I have ensured my changes to be thread-safe. Follow the link for more information: [Thread-safe Developer Guidelines](https://github.com/snowflakedb/snowpark-python/blob/main/CONTRIBUTING.md#thread-safe-development) 3. Please describe how your code solves the related issue. - Added AST encoding for `DataFrame.to_snowpark_pandas`. - I had to add some Local testing functionality to help my expectation test to pass. Also needed to add some janky logic to create a temp read-only table in lieu of the table that is created by `to_snowpark_pandas`. - I updated the script generating the relevant proto files to create them in the correct directory.

SNOW-1672579 Encode `DataFrame.to_snowpark_pandas` (#2711) 1. Which Jira issue is this PR addressing? Make sure that there is an accompanying issue to your PR. Fixes SNOW-1672579 2. Fill out the following pre-review checklist: - [x] I am adding a new automated test(s) to verify correctness of my new code - [ ] If this test skips Local Testing mode, I'm requesting review from @snowflakedb/local-testing - [ ] I am adding new logging messages - [ ] I am adding a new telemetry message - [ ] I am adding new credentials - [ ] I am adding a new dependency - [ ] If this is a new feature/behavior, I'm adding the Local Testing parity changes. - [x] I acknowledge that I have ensured my changes to be thread-safe. Follow the link for more information: [Thread-safe Developer Guidelines](https://github.com/snowflakedb/snowpark-python/blob/main/CONTRIBUTING.md#thread-safe-development) 3. Please describe how your code solves the related issue. - Added AST encoding for `DataFrame.to_snowpark_pandas`. - I had to add some Local testing functionality to help my expectation test to pass. Also needed to add some janky logic to create a temp read-only table in lieu of the table that is created by `to_snowpark_pandas`. - I updated the script generating the relevant proto files to create them in the correct directory.

SNOW-1830511, SNOW-1830542 Decoder logic for `DataFrame.agg`, `DataFrame.group_By`, and `RelationalGroupedDataFrame.agg` (#2749) 1. Which Jira issue is this PR addressing? Make sure that there is an accompanying issue to your PR. Fixes SNOW-1830511 and SNOW-1830542 2. Fill out the following pre-review checklist: - [x] I am adding a new automated test(s) to verify correctness of my new code - [ ] If this test skips Local Testing mode, I'm requesting review from @snowflakedb/local-testing - [ ] I am adding new logging messages - [ ] I am adding a new telemetry message - [ ] I am adding new credentials - [ ] I am adding a new dependency - [ ] If this is a new feature/behavior, I'm adding the Local Testing parity changes. - [x] I acknowledge that I have ensured my changes to be thread-safe. Follow the link for more information: [Thread-safe Developer Guidelines](https://github.com/snowflakedb/snowpark-python/blob/main/CONTRIBUTING.md#thread-safe-development) 3. Please describe how your code solves the related issue. Added decoder logic for `DataFrame.agg`, `DataFrame.group_By`, and `RelationalGroupedDataFrame.agg`. The corresponding tests are: `DataFrame.agg.test`, `RelationalGroupedDataFrame.test`, and `RelationalGroupedDataFrame.agg.test`.

2024-12-11

SNOW-18844419 Change the aliasing fn type to be an enum instead of optional boolean (#2712) 1. Which Jira issue is this PR addressing? Make sure that there is an accompanying issue to your PR. Fixes SNOW-18844419 2. Fill out the following pre-review checklist: - [ ] I am adding a new automated test(s) to verify correctness of my new code - [ ] If this test skips Local Testing mode, I'm requesting review from @snowflakedb/local-testing - [ ] I am adding new logging messages - [ ] I am adding a new telemetry message - [ ] I am adding new credentials - [ ] I am adding a new dependency - [ ] If this is a new feature/behavior, I'm adding the Local Testing parity changes. - [x] I acknowledge that I have ensured my changes to be thread-safe. Follow the link for more information: [Thread-safe Developer Guidelines](https://github.com/snowflakedb/snowpark-python/blob/main/CONTRIBUTING.md#thread-safe-development) 3. Please describe how your code solves the related issue. Change the aliasing fn type to be an enum instead of optional boolean to differentiate between as_, name, and alias.

SNOW-18844419 Change the aliasing fn type to be an enum instead of optional boolean (#2712) 1. Which Jira issue is this PR addressing? Make sure that there is an accompanying issue to your PR. Fixes SNOW-18844419 2. Fill out the following pre-review checklist: - [ ] I am adding a new automated test(s) to verify correctness of my new code - [ ] If this test skips Local Testing mode, I'm requesting review from @snowflakedb/local-testing - [ ] I am adding new logging messages - [ ] I am adding a new telemetry message - [ ] I am adding new credentials - [ ] I am adding a new dependency - [ ] If this is a new feature/behavior, I'm adding the Local Testing parity changes. - [x] I acknowledge that I have ensured my changes to be thread-safe. Follow the link for more information: [Thread-safe Developer Guidelines](https://github.com/snowflakedb/snowpark-python/blob/main/CONTRIBUTING.md#thread-safe-development) 3. Please describe how your code solves the related issue. Change the aliasing fn type to be an enum instead of optional boolean to differentiate between as_, name, and alias.

2024-11-26

[SNOW-1489960, SNOW-1491199] Snowpark python API client-side changes for IR encoding to protobuf (#2674) Main commit for [SNOW-1489960](https://snowflakecomputing.atlassian.net/browse/SNOW-1489960) bringing in changes to encode Snowpark python API to protobuf. A detailed commit history can be found in https://github.com/snowflakedb/snowpark-python/tree/ls-SNOW-1491199-merge-phase0-server-side. 2. Fill out the following pre-review checklist: - [ ] I am adding a new automated test(s) to verify correctness of my new code - [ ] If this test skips Local Testing mode, I'm requesting review from @snowflakedb/local-testing - [ ] I am adding new logging messages - [ ] I am adding a new telemetry message - [ ] I am adding new credentials - [ ] I am adding a new dependency - [ ] If this is a new feature/behavior, I'm adding the Local Testing parity changes. - [x] I acknowledge that I have ensured my changes to be thread-safe. Follow the link for more information: [Thread-safe Developer Guidelines](https://docs.google.com/document/d/162d_i4zZ2AfcGRXojj0jByt8EUq-DrSHPPnTa4QvwbA/edit#bookmark=id.e82u4nekq80k) 3. Please describe how your code solves the related issue. Please write a short description of how your code change solves the related issue. [SNOW-1489960]: https://snowflakecomputing.atlassian.net/browse/SNOW-1489960?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ --------- Co-authored-by: Arthur Zwiegincew <arthur.zwiegincew@snowflake.com> Co-authored-by: Ovidiu Platon <ovidiu.platon@snowflake.com> Co-authored-by: Hemit Shah <hemit.shah@snowflake.com> Co-authored-by: Varnika Budati <varnika.budati@snowflake.com> Co-authored-by: Eric Vandenberg <eric.vandenberg@snowflake.com> Co-authored-by: Andong Zhan <andong.zhan@snowflake.com>

[SNOW-1489960, SNOW-1491199] Snowpark python API client-side changes for IR encoding to protobuf (#2674) Main commit for [SNOW-1489960](https://snowflakecomputing.atlassian.net/browse/SNOW-1489960) bringing in changes to encode Snowpark python API to protobuf. A detailed commit history can be found in https://github.com/snowflakedb/snowpark-python/tree/ls-SNOW-1491199-merge-phase0-server-side. 2. Fill out the following pre-review checklist: - [ ] I am adding a new automated test(s) to verify correctness of my new code - [ ] If this test skips Local Testing mode, I'm requesting review from @snowflakedb/local-testing - [ ] I am adding new logging messages - [ ] I am adding a new telemetry message - [ ] I am adding new credentials - [ ] I am adding a new dependency - [ ] If this is a new feature/behavior, I'm adding the Local Testing parity changes. - [x] I acknowledge that I have ensured my changes to be thread-safe. Follow the link for more information: [Thread-safe Developer Guidelines](https://docs.google.com/document/d/162d_i4zZ2AfcGRXojj0jByt8EUq-DrSHPPnTa4QvwbA/edit#bookmark=id.e82u4nekq80k) 3. Please describe how your code solves the related issue. Please write a short description of how your code change solves the related issue. [SNOW-1489960]: https://snowflakecomputing.atlassian.net/browse/SNOW-1489960?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ --------- Co-authored-by: Arthur Zwiegincew <arthur.zwiegincew@snowflake.com> Co-authored-by: Ovidiu Platon <ovidiu.platon@snowflake.com> Co-authored-by: Hemit Shah <hemit.shah@snowflake.com> Co-authored-by: Varnika Budati <varnika.budati@snowflake.com> Co-authored-by: Eric Vandenberg <eric.vandenberg@snowflake.com> Co-authored-by: Andong Zhan <andong.zhan@snowflake.com>

2024-11-20

SNOW-1787415 Fix dataframe ast to register udtf for replay execution using client created udtf in same client session (#2620) Add Udtf registration that reuses the client registered UDTF. Doesn't need all the previous parameters as they were only required for original creation of udtf. Fix Full AST Validation to use same session for replay execution of unparser AST, thereby allowing reuse of client defined UDTFs.

2024-11-12

SNOW-1760495 Fix remaining merge gate failures. (#2588) Fixes remaining tests for AST enabled merge gate. Details: - changes window IR to use Expr instead of int to support make_interval. - moves checks for orderby in front. - fixes package moduletype management for sproc. - fixes check for create_map - fixes bug in AST serialization for dataframewriter

2024-11-08

SNOW-1787415 Fix dataframe join with sort deterministic (#2591) SNOW-1787415 Fix dataframe join with sort deterministic

2024-11-07

SNOW-1787415 Fix dataframe AST various bugs and float/decimal infinity, nan etc support (#2580) SNOW-1787415 Fix dataframe AST bugs related to Window, lag, lead, float and decimal infinity support. Fix default Window spec resulting in an error. Fix parameter ordering bug in lag and lead functions. Add support for float and decimal infinity, nan, etc special values

Cherry pick #2518 #2549 (#2577) <!--- Please answer these questions before creating your pull request. Thanks! ---> 1. Which Jira issue is this PR addressing? Make sure that there is an accompanying issue to your PR. <!--- In this section, please add a Snowflake Jira issue number. Note that if a corresponding GitHub issue exists, you should still include the Snowflake Jira issue number. For example, for GitHub issue https://github.com/snowflakedb/snowpark-python/issues/1400, you should add "SNOW-1335071" here. ---> Fixes SNOW-NNNNNNN 2. Fill out the following pre-review checklist: - [ ] I am adding a new automated test(s) to verify correctness of my new code - [ ] If this test skips Local Testing mode, I'm requesting review from @snowflakedb/local-testing - [ ] I am adding new logging messages - [ ] I am adding a new telemetry message - [ ] I am adding new credentials - [ ] I am adding a new dependency - [ ] If this is a new feature/behavior, I'm adding the Local Testing parity changes. - [ ] I acknowledge that I have ensured my changes to be thread-safe. Follow the link for more information: [Thread-safe Developer Guidelines](https://docs.google.com/document/d/162d_i4zZ2AfcGRXojj0jByt8EUq-DrSHPPnTa4QvwbA/edit#bookmark=id.e82u4nekq80k) 3. Please describe how your code solves the related issue. Please write a short description of how your code change solves the related issue. This CP #2518 and #2549 to sync the protobuf generation with main branch and fix some paths. --------- Co-authored-by: Ovidiu Platon <ovidiu.platon@snowflake.com>

2024-10-31

SNOW-1491199 Introduce ast.proto to the Snowpark client (#2549) 1. Which Jira issue is this PR addressing? Make sure that there is an accompanying issue to your PR. Fixes SNOW-1491199 2. Fill out the following pre-review checklist: - [ ] I am adding a new automated test(s) to verify correctness of my new code - [ ] If this test skips Local Testing mode, I'm requesting review from @snowflakedb/local-testing - [ ] I am adding new logging messages - [ ] I am adding a new telemetry message - [ ] I am adding new credentials - [ ] I am adding a new dependency - [ ] If this is a new feature/behavior, I'm adding the Local Testing parity changes. - [x] I acknowledge that I have ensured my changes to be thread-safe. Follow the link for more information: [Thread-safe Developer Guidelines](https://docs.google.com/document/d/162d_i4zZ2AfcGRXojj0jByt8EUq-DrSHPPnTa4QvwbA/edit#bookmark=id.e82u4nekq80k) 3. Please describe how your code solves the related issue. Introduce `ast.proto` under `src/snowflake/snowpark/_internal/proto` and `ast_pb2.py` under the `generated` subdirectory. `ast.proto` gets created from the contents of [Snowpark/ast](https://github.com/snowflakedb/snowflake/tree/snowpark-phase-0/Snowpark/ast), using `ir-dsl-c`. The chain is AST IR -> `ir-dsl-c` -> `ast.proto` -> Python `protoc` -> `ast_pb2.py`. We are merging both `ast.proto` and `ast_pb2.py` because the Snowpark repository is not fully equipped to generate the Python client stubs at packaging time. This change is a work in progress and blocked behind IT tickets for an indeterminate amount of time. A complete description of the AST model is too large to include here. We are currently working on an AST IR developer guide, as well. Co-authored-by: Arthur Zwiegincew <arthur.zwiegincew@snowflake.com> Co-authored-by: Leonhard Spiegelberg <leonhard.spiegelberg@snowflake.com> Co-authored-by: Hemit Shah <hemit.shah@snowflake.com> Co-authored-by: Varnika Budati <varnika.budati@snowflake.com> Co-authored-by: Eric Vandenberg <eric.vandenberg@snowflake.com>

SNOW-1491199 Introduce ast.proto to the Snowpark client (#2549) 1. Which Jira issue is this PR addressing? Make sure that there is an accompanying issue to your PR. Fixes SNOW-1491199 2. Fill out the following pre-review checklist: - [ ] I am adding a new automated test(s) to verify correctness of my new code - [ ] If this test skips Local Testing mode, I'm requesting review from @snowflakedb/local-testing - [ ] I am adding new logging messages - [ ] I am adding a new telemetry message - [ ] I am adding new credentials - [ ] I am adding a new dependency - [ ] If this is a new feature/behavior, I'm adding the Local Testing parity changes. - [x] I acknowledge that I have ensured my changes to be thread-safe. Follow the link for more information: [Thread-safe Developer Guidelines](https://docs.google.com/document/d/162d_i4zZ2AfcGRXojj0jByt8EUq-DrSHPPnTa4QvwbA/edit#bookmark=id.e82u4nekq80k) 3. Please describe how your code solves the related issue. Introduce `ast.proto` under `src/snowflake/snowpark/_internal/proto` and `ast_pb2.py` under the `generated` subdirectory. `ast.proto` gets created from the contents of [Snowpark/ast](https://github.com/snowflakedb/snowflake/tree/snowpark-phase-0/Snowpark/ast), using `ir-dsl-c`. The chain is AST IR -> `ir-dsl-c` -> `ast.proto` -> Python `protoc` -> `ast_pb2.py`. We are merging both `ast.proto` and `ast_pb2.py` because the Snowpark repository is not fully equipped to generate the Python client stubs at packaging time. This change is a work in progress and blocked behind IT tickets for an indeterminate amount of time. A complete description of the AST model is too large to include here. We are currently working on an AST IR developer guide, as well. Co-authored-by: Arthur Zwiegincew <arthur.zwiegincew@snowflake.com> Co-authored-by: Leonhard Spiegelberg <leonhard.spiegelberg@snowflake.com> Co-authored-by: Hemit Shah <hemit.shah@snowflake.com> Co-authored-by: Varnika Budati <varnika.budati@snowflake.com> Co-authored-by: Eric Vandenberg <eric.vandenberg@snowflake.com>

2024-05-15

Updated AST proto. Now matches the server.

2024-05-14

Update ast_pb2.py and include it in the Snowpark wheel (#1564) <!--- Please answer these questions before creating your pull request. Thanks! ---> 1. Which Jira issue is this PR addressing? Make sure that there is an accompanying issue to your PR. <!--- In this section, please add a Snowflake Jira issue number. Note that if a corresponding GitHub issue exists, you should still include the Snowflake Jira issue number. For example, for GitHub issue https://github.com/snowflakedb/snowpark-python/issues/1400, you should add "SNOW-1335071" here. ---> Fixes SNOW-0 2. Fill out the following pre-review checklist: - [ ] I am adding a new automated test(s) to verify correctness of my new code - [ ] I am adding new logging messages - [ ] I am adding a new telemetry message - [ ] I am adding new credentials - [x] I am adding a new dependency 3. Please describe how your code solves the related issue. Update `ast_pb2.py` (already present in the repository). Add the `setuptools` dependencies required for development. Include the module path for `ast_pb2.py` in the manifest, so that the file makes it into the Snowpark wheel.

This commit does not contain any .proto files.

2024-05-03

[private branch] Maintain the AST as we build up Snowpark expressions: table(), filter() (#1468) A very basic initial attempt at serializing the AST. I'm trying to maintain a parallel codebase for phases 0 and 1 for now, since it would be a shame to do this work twice. Once we complete and ship phase 0, we'll be able to drastically simplify the phase 1 client. Unlike what I mentioned before, this implementation doesn't flush dependencies of eagerly evaluated expressions. Instead, any client-side value is appended to the pending batch. This is simpler to implement and will likely work well, although we may need to do some dependency analysis on the server to ensure we don't issue unnecessary queries.