Proto commits in dataform-co/dataform

These commits are when the Protocol Buffers files have changed: (only the last 100 relevant commits are shown)

2025-04-03

Add builtinAssertionNamePrefix field to WorkflowSettings. (#1956) * feat: Add implementation for builtinAssertionNamePrefix. * test: Add unit tests for builtinAssertionNamePrefix. * docs: Update docs with builtinAssertionNamePrefix field info. * test: Cover projectConfig with unit test. * feat: Make prefix as first part of the name and adjust tests. * refactor: Clean up index_test.ts * refactor: Clean up index_test.ts again * refactor: Clean up index_test.ts again and again

The documentation is generated from this commit.

2025-03-05

Add support for schema and dataset configs in data preparation config. (#1935) This allows data preparations to configure override similar to all other actions.

2025-03-03

Remove the unused automatic load mode. This doesn't apply to compiled data preparations.

2025-02-20

Clean up remaining ekrekr todos (#1927) * Start removing todos * Cleanup more todos * Remove another todo * More old TODO tidying * Add fixed filename resolution for filepath of inline assertions * Move inline assertion generation to shared definition * Remove last todo * Fix lint

2025-02-12

Add typedoc generation of the Dataform JS API (#1913) * Experiment with typedoc comments * Add typedoc configuration json. Now can just run 'typedoc' to run * Add working generation, just via npx typedoc * Add documentation for remaining publish session methods * Add docs changes * Add typedoc generation to docs regeneration script, run * Add typedocs to the assertion action, tidy docs structure * Add docs for more actions, add more details on consumption * Add docs for the remaining classes * Tidy, improve JS API references * Fix lint * Update typedoc and typedoc markdown to versions that are in airlock * Fix dataprep docs generation, remove sources from typedoc generation * Add hide private

2025-02-11

Added Load Configuration setting for Data Preparation SQLX format. (#1909) * Added Load Configuration setting for Data Preparation SQLX format. # Conflicts: # core/main_test.ts * Addressed PR comments * Addressed suggestions * Addressed PR comments

2025-02-03

SQLX support for Data Preparations (#1891) * Added SQLX support for data preparation. # Conflicts: # core/utils.ts # Conflicts: # core/actions/data_preparation.ts # core/main_test.ts * Removed config interfaces, using proto config instead. # Conflicts: # core/actions/data_preparation.ts * Removed config interfaces, using proto config instead. Added requested test case. * Removed package.json changes * Addressed PR comments

2025-01-30

Add support for OnSchemaChange field for incremental tables. (#1902) * Add support for OnSchemaChange field for incremental tables. * Fix docs * Fix OnSchemaChange docs section * Update comments and simplify tests.

2025-01-09

Add notebook runtime template resource name field to NotebookRuntimOptions (#1890)

2024-12-12

Add assert config option for the JS api

2024-09-19

Remove data preparation proto definition. (#1849) * Remove data preparation proto definition. Clean up unused code in data preparation action. * Removed missed unused code

2024-09-11

Update Data Preparation processor to remove proto dependency. (#1839) * Update Data Preparation processor to remove proto dependency. The data preparation in the compiled graph now contains the resolved YAML after compiling. * Fixed compilation issue

2024-09-04

Bump dataprep proto to latest config (#1825) * Bump dataprep proto to latest config * Remove license header

2024-08-16

Sync proto structure and message names for data preparation. (#1811) * Sync proto structure and message names for data preparation. Moved data preparation into another package. * Fix broken test.

2024-08-15

Update output for Data Preparation processing (#1808) * Updated data preparation proto to only carry binary encoded data preparation contents after processing the data preparation definition. * Addressed PR Comments. * Addressed final PR comments * Fixed compilation issue.

2024-08-07

Remove some outdated configs references, fix generated formatting (#1799)

2024-07-29

Handling for Data Preparation actions (#1789) * Add logic to handle Data preparation definitions and parse data preparation YAML into the proto representation of the operation. * Addressed PR Comments * Fixed lint warnings

2024-07-18

Added proto definitions for Data Preparations (#1788) * Added proto definitions for Data Preparations * Moved Data preparation protos into a separate file

2024-06-27

Add support for action configs table assertions and hermeticity, add action configs fields tests (#1762) * Progress adding action config tests for different actions * Add remaining table test types, fix support for table assertions and hermeticity * Regenerate docs, tidy merge files

2024-06-26

Declaration configs as protos (#1767)

WIP add support for column descriptors

2024-06-24

Assertions configs as protos (#1731) * Basic POC for assertions configs as protos * Add test for SQLX specific, tidy up session and action builder * Fix assertion schema application via constructor * Tidy * Tidy * Add legacy config type, and use Resolvable * Fix lint

2024-05-10

Make proto object verification give more descriptive error messages (#1733) * Make proto object verification give better details * Fix error messages * Tidy * Add proto type URL to docs URL * Update options to be object based * Update to use enum for error behaviour

2024-05-07

Feature: dependent assertions as dependencies (#1720) * Feature: dependent assertions as dependencies * Action.yaml support for dependent assertions * Review updates * spell check and ActionMap * dependency duplication check O(n) * Formating * includeAssertionsForDependencyMap changed to includeAssertionsForDependency * review fixes * Lint fixes * fix shadow beforeEach call

2024-03-26

Fix CLI tags, allow comma delisted tags, add test

2024-03-12

Remove bucket option

Add notebook runtime options to workflow settings

2024-03-11

Tidy

Make field flat, add core proto tests

Add notebook options to configs proto

2024-02-08

Remove JS context from SQL files (#1672) * Remove JS context from SQL files * Tidy * Update example projects * Tidy * Simpler sql escaping and example

2024-01-23

Update proto to use a separate compiled graph, with UX review changes (#1656) * Update the v3 proto to use a separate compiled graph, with UX review changes * Tidy * Nest bigquery options, tidy action descriptors * Re-add config protos temporarily, to prevent test breakage * Remove columnar level tags, and compiled graph dataform core version * Move action configs outside the compiled graph * Tidy action fields further * Improve action config namespacing * Add notebook config to compiled graph format compilation * More proto tweaks * Progress applying proto updates within file * Fix core main tests * Fix remaining compilation and testing issues

Remove inline tables functionality (#1658)

2024-01-18

Remove more dead code, including navigator column descriptors and tools (#1655) * Remove more dead code, including navigator column descriptors and tools * Remove dead navigator test * Tidy

2024-01-15

Make the CLI use and default to workflow_settings.yaml (#1648) * Make the CLI use and default to workflow_settings.yaml * Tidy * Undo debugging examples change

2024-01-03

Update core proto

Cleanup: remove action hermeticity

Add action config support for the remaining SQL based action types (#1633) * Progress adding remaning action types config support * Add action config support for the remaining SQL based action types * Tidy

2024-01-02

Add support for Tables to action config files (#1631) * Add support for Tables to action config files * Tidy config order * Fix proto

Add support for declarations in action config files (#1630) * Add support for declarations via action config files * Tidy * Fix lint * Tidy constructors

Add basic support for actions.yaml reading SQL files as operations (#1628) * Experiment with basic SQL file support * Fix canonical target population and tidy test * Add proof of context to test * Tidy * Make the action additions to the session use the JS session interface, to preserve JS API compatability * Fix bug with action target names * Fix tslint errors

2023-12-12

Add version to workflow settings (#1610) * Add version to workflow settings * Tidy * Fix import order * Rename version to dataformCoreVersion * Don't populate the project config version if empty * tidying

2023-12-04

Add basic actions.yaml and notebook support (#1595) * Add basic actions.yaml and notebook support * Add test for notebook * Add action details extraction from filename, facilitate requiring notebooks, make tests work * Tidy * Fix lint errors * fix * George comments

2023-12-01

Stop using ProtobufJS's verify method (it doesn't do much) (#1596) * Stop using ProtobufJS's verify method because it doesn't really do anything * Fix proto validation and tests * Remove buganizer reference * Fix lint

2023-11-24

Move concurrency and action retries from project config proto to the CLI (#1585) * Move concurrency and action retries from project config proto to the CLI * Fix typing * Fix tests * Fix optional

2023-11-22

Remove deprecated run cache and gcloud project ID fields (#1584) * Remove deprecated run cache and gcloud project ID fields * Remove run cache test * Tidying

2023-11-15

Remove gen index in favor of main (#1571) * Remove gen index * Fix semver * Simplify error message

Merge main to main_v3

Merge main to main_v3

Merge main into main_v3

Allow ArrayBuffer args and return from the main function, more than doubling compilation speed for large projects (#1570) * Remove base64 from compilation IPC, more than doubling compilation speed * Docs tweak * Export a supported features object, tidy the core index * Bump version * Preserve base64 encoded string backwards compatability * Fix import order * Comments tweaks

2023-11-14

Add initial action config proto definitions (#1576)

2023-10-18

Remove non bigquery warehouses (#1550) * Remove non-bigquery warehouses * Cleaup some dependency usage, make build * Re-add materialized views * Fix some tests

2023-08-11

Fix merge to be from main_v1

2023-05-25

Make actions contextable from inside SQL files, update bigquery stackoverflow example to use it

2023-05-10

Add declarations, fix stackoverflow bigquery example, populate fields used by it, fix canonical ref to ref conversion

2023-05-09

Tidy structure, attempt to attach result to global session

2023-05-02

Add example global session interpreter for includes files

2023-02-16

Working proto-ts

2023-02-15

support writeDisposition option for bigquery

Added enum for table's type field (#1453) * added enum for table's type field Signed-off-by: Nick Nalivaika <ikolina@google.com> * update version Signed-off-by: Nick Nalivaika <ikolina@google.com> * simplify code by filling enumType field for tables when loading compiled graphs * small fix for biquery adapter * make linter happy * fix integration tests * do not call setOrValidateTableEnumType when check for table validity, just check enumType value --------- Signed-off-by: Nick Nalivaika <ikolina@google.com>

2023-01-05

Environments and schedules (#1418)

Environments and schedules (#1418)

2023-01-04

Remove dead code paths and those not relating to ELT management (#1414) * Remove code paths not relating to ELT management * Remove Dataform SQL package * Remove unused workspace imports * Remove common_v1 example * Remove some v1 * Remove @dataform/sql package and common_v1 example * Remove SQL package from common_v2 example * Remove build imports from tests of @dataform/sql that weren't doing anything * Inline timestamp definition in non BQ tests * Fix timestamp function case * Restore v1 backwards_compatibility tests

Remove dead code paths and those not relating to ELT management (#1414) * Remove code paths not relating to ELT management * Remove Dataform SQL package * Remove unused workspace imports * Remove common_v1 example * Remove some v1 * Remove @dataform/sql package and common_v1 example * Remove SQL package from common_v2 example * Remove build imports from tests of @dataform/sql that weren't doing anything * Inline timestamp definition in non BQ tests * Fix timestamp function case * Restore v1 backwards_compatibility tests

2022-05-31

Add defaultLocation field to dataform.json (#1344) * Add defaultLocation field to ProjectConfig * Fix proto number * Remove whitespace

Add defaultLocation field to dataform.json (#1344) * Add defaultLocation field to ProjectConfig * Fix proto number * Remove whitespace

2022-05-17

Additional Options Implementation (#1335) * BigQuery Additional Options Implementation

Additional Options Implementation (#1335) * BigQuery Additional Options Implementation

2022-05-04

Additional BigQuery Options (#1332) Add requirePartitionFilter and partitionExpirationDays to BigQuery config

Additional BigQuery Options (#1332) Add requirePartitionFilter and partitionExpirationDays to BigQuery config

2022-01-27

Add materialized view support for BigQuery and Snowflake (#1302) * Materialized view support for BigQuery and Snowflake

Add materialized view support for BigQuery and Snowflake (#1302) * Materialized view support for BigQuery and Snowflake

2021-11-17

Refactor core.proto into multiple files (#1288) * Multi file protos * Refactor proto files into seperate files

Refactor core.proto into multiple files (#1288) * Multi file protos * Refactor proto files into seperate files

2021-11-16

Remove name and dependencies proto fields (#1278) * Remove name and dependencies proto fields * Fix last tests * A bit more cleanup

Remove name and dependencies proto fields (#1278) * Remove name and dependencies proto fields * Fix last tests * A bit more cleanup

2021-11-15

Remove run caching support (#1286) * Remove support for run caching from the CLI/API * Remove a few more things * Fix merge

Remove run caching support (#1286) * Remove support for run caching from the CLI/API * Remove a few more things * Fix merge

2021-11-12

Update core execution API after design review (#1285) * Update core execution API after design review * Fix

Update core execution API after design review (#1285) * Update core execution API after design review * Fix

2021-11-05

Add dependencyTargets to execution action proto (#1283)

Add dependencyTargets to execution action proto (#1283)

2021-10-15

Remove >12mo old deprecated proto fields (#1276) * Remove >12mo old deprecated proto fields * Fix accidental test logic change

Remove >12mo old deprecated proto fields (#1276) * Remove >12mo old deprecated proto fields * Fix accidental test logic change

2021-08-10

Add a new "core execution" interface for simplified sandbox <> sandboxee communication (#1252) * Add a new compilation main method to replace gen index * Add a new main compilation entry point * Rename main config/result to CoreExecutionX * Address comment * Fix import order

Add a new "core execution" interface for simplified sandbox <> sandboxee communication (#1252) * Add a new compilation main method to replace gen index * Add a new main compilation entry point * Rename main config/result to CoreExecutionX * Address comment * Fix import order

2021-08-06

Remove support for single query compilation (#1251) * Remove support for single query compilation * Fix lint

Remove support for single query compilation (#1251) * Remove support for single query compilation * Fix lint

2021-04-22

Enabling Application Default Credentials for BigQuery (#1210) * Enabling Application Default Credentials for BigQuery * Updated documentation, cleaned up code, and bumped versions. * Fixed documentation * Resolved lint issue. * BigQuery credentials with just a projectId are valid -- remove test.

Enabling Application Default Credentials for BigQuery (#1210) * Enabling Application Default Credentials for BigQuery * Updated documentation, cleaned up code, and bumped versions. * Fixed documentation * Resolved lint issue. * BigQuery credentials with just a projectId are valid -- remove test.

2021-02-03

Add testcases for unknown fields/enum values. (#1182)

Add testcases for unknown fields/enum values. (#1182)

2021-02-02

Fix, and add tests for, repeated Message concatenation (when interpreted as a singular Message). (#1181)

Fix, and add tests for, repeated Message concatenation (when interpreted as a singular Message). (#1181)

2020-12-04

Cleanup: s/protobufts/protoc-gen-ts/g. (#1137) * Cleanup: s/protobufts/protoc-gen-ts/g. * tidy

Cleanup: s/protobufts/protoc-gen-ts/g. (#1137) * Cleanup: s/protobufts/protoc-gen-ts/g. * tidy

2020-12-02

Add initial implementation of protoc-gen-ts. (#1129) * barebones * new plan: one ts file per protobuf * imports written, need root * now with import prefixes * barebone messages/enums * add examples protobufs, got macro working, gazelle * got rid of examples, working macro, fixed enums * enums * null * typing is coming along * imports * indentation and enums * much cleaner generated code * cleanup * cleanup and create start * set params * set params * default values fixed, including repeated * started serialization * started serialization * first test * int32 and cleaned up testcase * moved runtime support * refactored tests * stronger tests * more serialization * more serialization * all non repeated serialization done * repeated field serializatin, needs tests * started adding custom serialization * ongoing new serialization * reorder to match protobuf field order * loads more serialization * loads more serialization * loads more serialization * loads more serialization * serialization all done * get rid of protobufjs dependency * initial repeated serialization * clarify proto3 serialization * fix enum fields to be not-optional, and use new shouldProto3Serialize everywhere * refactor to much cleaner interface * include correct type info for oneof fields * massive cleanup * oneof serialization * oneof typing * ongoing map work * hacking * reserializer * basic deserialize * basic deserialize * tidying up * bugfixes and test tidy * bugfixes and test tidy * oneof deserialization * oneof deserialization * deserialize repeated fields, packed and unpacked * fixes and tests * fixes and tests * unpacked tests * byte equality checks * cleaner tests * cleanup * rewrite the whole bloody thing * toJson support * give create default empty value * starting to use Decoders API to support Maps to support toJson (and better Maps). lots of cleanup needs to happen * omfg so much better * omfg so much better * merge messages upon deserialize * cleanup * add readme * remove gosec * add another todo * make tslint happy * fix reserializer location * lewis comments

Add initial implementation of protoc-gen-ts. (#1129) * barebones * new plan: one ts file per protobuf * imports written, need root * now with import prefixes * barebone messages/enums * add examples protobufs, got macro working, gazelle * got rid of examples, working macro, fixed enums * enums * null * typing is coming along * imports * indentation and enums * much cleaner generated code * cleanup * cleanup and create start * set params * set params * default values fixed, including repeated * started serialization * started serialization * first test * int32 and cleaned up testcase * moved runtime support * refactored tests * stronger tests * more serialization * more serialization * all non repeated serialization done * repeated field serializatin, needs tests * started adding custom serialization * ongoing new serialization * reorder to match protobuf field order * loads more serialization * loads more serialization * loads more serialization * loads more serialization * serialization all done * get rid of protobufjs dependency * initial repeated serialization * clarify proto3 serialization * fix enum fields to be not-optional, and use new shouldProto3Serialize everywhere * refactor to much cleaner interface * include correct type info for oneof fields * massive cleanup * oneof serialization * oneof typing * ongoing map work * hacking * reserializer * basic deserialize * basic deserialize * tidying up * bugfixes and test tidy * bugfixes and test tidy * oneof deserialization * oneof deserialization * deserialize repeated fields, packed and unpacked * fixes and tests * fixes and tests * unpacked tests * byte equality checks * cleaner tests * cleanup * rewrite the whole bloody thing * toJson support * give create default empty value * starting to use Decoders API to support Maps to support toJson (and better Maps). lots of cleanup needs to happen * omfg so much better * omfg so much better * merge messages upon deserialize * cleanup * add readme * remove gosec * add another todo * make tslint happy * fix reserializer location * lewis comments

2020-12-01

Introduce schema proto as a partial target