Proto commits in dataform-co/dataform

These commits are when the Protocol Buffers files have changed: (only the last 100 relevant commits are shown)

2026-07-07

Move DefaultIcebergConfig message to avoid circular dependencies (#2104) * Define DefaultIcebergConfig message Move DefaultIcebergConfig message from configs.proto to core.proto. This now prevents using core.proto messages in config.proto. The way protos are used across the project lets this change work without additional changes in code. * Remove DefaultIcebergConfig message Removed DefaultIcebergConfig message definition from configs.proto. * Remove import of configs.proto from core.proto * Remove unused import from configs.proto Removed import statement for core.proto.

2026-07-03

Implement incrementalPredicates config for BigQuery incremental tables - Replaced legacy 'T' and 'S' MERGE aliases with explicit 'DATAFORM_DEST' and 'DATAFORM_SOURCE' in the BigQuery adapter. - Updated `incremental_table_test.ts` assertions. - Documented the new `incrementalPredicates` property in the OSS API reference.

Support incremental_strategy parameter and new insert_overwrite strategy - updated proto with new parameters - added new tests - added validation for chosen incremental_strategies - added new insert_overwrite strategy logic

2026-07-02

Merge branch 'main' into apilaskowski-patch-1

Proto field additions for JiT flow (#2182)

The documentation is generated from this commit.

2026-06-29

feat: Implement Dataform CLI Data Lineage write path PoC Implement table-level lineage emission from the Dataform CLI to the Dataplex Lineage API (`datalineage.googleapis.com`) using OpenLineage RunEvents. - Introduce `@google-cloud/lineage` library dependency and wire to Bazel. - Build `LineageEmitter` to handle OpenLineage v1-0-2 payload construction, recursive Protobuf Struct serialization, and endpoint localization. - Persist and reuse `runId` maps to consolidate STARTED and terminal execution states under a single run resource lifecycle. - Integrate emission trigger Hooks into the CLI execution engine under `cli/api/commands/run.ts`. - Expose configuration flags via `workflow_settings.yaml` (`lineage: { enabled: true }`) and CLI argument (`--emit-lineage`). - Implement unit tests covering payload serialization, permission denial logic, and error-handling fallback behaviors.

2026-06-22

Merge branch 'main' into dataproc-poc

Support incremental_strategy parameter and new insert_overwrite strategy - updated proto with new parameters - added new tests - added validation for chosen incremental_strategies - added new insert_overwrite strategy logic

2026-06-19

Proto field additions for JiT flow

2026-06-18

feat: Implement Dataform CLI Data Lineage write path PoC Implement table-level lineage emission from the Dataform CLI to the Dataplex Lineage API (`datalineage.googleapis.com`) using OpenLineage RunEvents. - Introduce `@google-cloud/lineage` library dependency and wire to Bazel. - Build `LineageEmitter` to handle OpenLineage v1-0-2 payload construction, recursive Protobuf Struct serialization, and endpoint localization. - Persist and reuse `runId` maps to consolidate STARTED and terminal execution states under a single run resource lifecycle. - Integrate emission trigger Hooks into the CLI execution engine under `cli/api/commands/run.ts`. - Expose configuration flags via `workflow_settings.yaml` (`lineage: { enabled: true }`) and CLI argument (`--emit-lineage`). - Implement unit tests covering payload serialization, permission denial logic, and error-handling fallback behaviors.

Add support for running Airflow operator in Cloud Run service. The choice is controlled using the runner field in orchestration pipeline yaml, which can accept dataform_notebook_cloud_run_service and dataform_notebook_cloud_run_job. The notebooks that run Airflow operator are also more parameterized now.

2026-06-17

Add metadata field to AssertionConfig (#2208) Mirrors TableConfig.metadata / ViewConfig.metadata so callers can attach key/value data that lands on Assertion.action_descriptor.metadata.extra_properties.

2026-06-16

Add support for airflowOperator actions

2026-06-11

Add op-to-dataform extension and related core patches

2026-06-09

Add optional original_file_path to Assertion protos Lets callers record a source declaration path distinct from the existing `file_name`/`filename`. When the new field is unset, consumers continue to use `file_name` as today. Backwards-compatible: additive, optional.

2026-06-08

Support incremental_strategy parameter and new insert_overwrite strategy - updated proto with new parameters - added new tests - added validation for chosen incremental_strategies - added new insert_overwrite strategy logic

Add go_package option to db_adapter.proto and jit.proto (#2197)

2026-06-04

Add Managed Spark execution engine support for Notebook actions - Introduce a nested `ExecutionEngine` enum in both configs and core protos (decoupled design) to allow selecting the engine backend (`COLAB` vs `MANAGED_SPARK`). - Add project-level `default_managed_spark_execution_options` (with required `staging_bucket_uri`) configuration options to configs and core protos. - Update the TypeScript compiler to parse and validate default Managed Spark options at the project level, and map the action-level engine configuration. - Implement compile-time validation in `Notebook.compile()` to check that `defaultManagedSparkExecutionOptions` is defined if a notebook action uses `MANAGED_SPARK`. - Update the VS Code JSON schema to validate the new project options. - Implement unit tests covering parsing, propagation, and validation behaviors.

Add go_package option to db_adapter.proto and jit.proto

2026-06-02

Add JiT compilation support for assertion actions (#2193) Adds JIT_COMPILATION_TARGET_TYPE_ASSERTION alongside the existing TABLE / OPERATION / INCREMENTAL_TABLE target types, plus a matching JitAssertionResult { string query } proto and a jitCompileAssertion() dispatcher in core/jit_compiler.ts. Assertions reuse SqlActionJitContext since AssertionContext implements IActionContext, and the result is a single SELECT query string (matching the existing JitAssertionResult = string alias in core/actions/assertion.ts).

2026-05-27

feat: integrate JiT compilation and finalize test suite feat: enable JiT compilation in CLI run, build and compile commands Connect main CLI execution flow to the JiT worker process Add comprehensive JiT test suite Finalize AoT build logic to preserve JiT-specific fields Update CLI console and entry points for integrated JiT support Add file_name field (=8) to JitCompilationRequest proto for target source file resolution in the JiT worker.

2026-05-26

feat: implement .jitCode() support for Assertion actions (#2170) - Add `jit_code` field to the `Assertion` message in `protos/core.proto` - Fix missing protobuf imports and configuration in `protos/BUILD` - Introduce `JitAssertionResult` type alias and `jitCode` method to `Assertion` builder (`core/actions/assertion.ts`) - Add strictly typed `jitContextable` parameter to `Session.sqlxAction` in `core/session.ts` - Update assertion `compile()` to handle `jitCode` execution bypassing standard string evaluation - Parse and apply `jitContextable` safely for assertions in `core/session.ts` - Add unit test coverage in `core/main_test.ts` to verify compilation output

2026-05-21

feat: add .preOps() and .postOps() support to Assertion action This update brings the Assertion action on par with Table and View by introducing support for pre-operations and post-operations. - Updated `dataform.proto` (via `core.proto` and `configs.proto`) to include `pre_ops`/`post_ops` for `Assertion` and `pre_operations`/ `post_operations` for `AssertionConfig`. - Enhanced `core/actions/assertion.ts` with `contextablePreOps` and `contextablePostOps` arrays. - Exposed `.preOps()` and `.postOps()` methods on both `Assertion` builder and `AssertionContext`. - Added test coverage in `core/main_test.ts` to ensure operations are compiled correctly for assertions.

2026-05-20

JiT Worker in CLI and RPC infrastructure (#2109) * refactor: split index_test.ts into specialized test files * refactor: decompose monolithic cli/index_test.ts into modular components - Extract shared test utilities into cli/index_test_base.ts - Create dedicated test files for help, init, project, compile, and run commands - Update cli/BUILD to reflect the new modular test structure * feat: enhance JiT compiler with action descriptors and metadata support * feat: add action descriptor support to JiT compilation results - Implement action and column descriptor logic in core/jit_compiler.ts - Support overrides for descriptions, labels, and tags in JiT results - Enhance error coercion in common/errors/errors.ts - Add comprehensive JiT metadata tests in core/jit_compiler_test.ts - Sync protos/execution.proto and protos/jit.proto with new result fields * feat: implement JiT worker and RPC infrastructure * feat: add worker process management for JiT compilation - Introduce base worker and JiT-specific child process logic - Implement RPC bridge for database access during JiT - Add VM scripts and loader for isolated execution - Add unit tests for the RPC mechanism

2026-05-01

Populate new unit test properties and add flag to include unit tests in compiled graph (#2063) * Added new properties to Test proto * Simplified code * Simplified code for readability * More cleanup * renamed function * Fixed issue with schema/database overrides and tests * Removed unused code * Cleanup * Further cleanup * Fixed test naming * Added tags support * Removed adding test actions into the DAG until later * Addressed PR comments * Fixed tests * Add new property to configure wether unit tests should be included in the compiled DAG as dependencies to the action being tested

2026-04-30

Bump protobufjs from 7.2.5 to 7.5.5 (#2144) * Bump protobufjs from 7.2.5 to 7.5.5 Bumps [protobufjs](https://github.com/protobufjs/protobuf.js) from 7.2.5 to 7.5.5. - [Release notes](https://github.com/protobufjs/protobuf.js/releases) - [Changelog](https://github.com/protobufjs/protobuf.js/blob/master/CHANGELOG.md) - [Commits](https://github.com/protobufjs/protobuf.js/compare/protobufjs-v7.2.5...protobufjs-v7.5.5) --- updated-dependencies: - dependency-name: protobufjs dependency-version: 7.5.5 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> * fix local proto imports after protobufjs upgrade --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Nick Nalivaika <ikolina@google.com>

2026-04-22

Move extension options (#2147)

2026-04-21

Support generic compilation options (#2145)

2026-04-14

Extend list of supported BQ types (#2134)

2026-03-31

JiT context - Add Running execution metadata (#2132) Extend metadata of JiT context with information about the current execution run.

2026-03-27

Introduce tags to Dataform declarations (#2130)

2026-03-25

Extend ExecuteResponse to contain schema fields. (#2125)

Support extension in workflow_settings.yml (#2122) * Support extension in workflow_settings.yml If extension is defined in workflow_settings.yml, the extension will be executed in selected mode. * Support extension in workflow_settings.yml If extension is defined in workflow_settings.yml, the extension will be executed in selected mode. * Update BUILD. * Update BUILD. * Add BUILD file for utils_test * Add imports to BUILD * Fix test. * Remove node_modules * Update tests. * DRY + move extension preparetion to compilation (from index). * Add newlines at end of files. * Yet another endline.

2026-03-12

JiT: protos update to enable execution in CLI (#2107) - Enhance error coercion in common/errors/errors.ts - Sync protos/execution.proto and protos/jit.proto with new fields

2026-03-10

feat: integrate JiT compilation and finalize test suite * feat: enable JiT compilation in CLI run, build and compile commands - Connect main CLI execution flow to the JiT worker process - Add comprehensive JiT test suite - Finalize AoT build logic to preserve JiT-specific fields - Update CLI console and entry points for integrated JiT support

feat: integrate JiT compilation and finalize test suite

feat: enhance JiT compiler with action descriptors and metadata support * feat: add action descriptor support to JiT compilation results - Implement action and column descriptor logic in core/jit_compiler.ts - Support overrides for descriptions, labels, and tags in JiT results - Enhance error coercion in common/errors/errors.ts - Add comprehensive JiT metadata tests in core/jit_compiler_test.ts - Sync protos/execution.proto and protos/jit.proto with new result fields

JiT: protos update to enable execution in CLI - Enhance error coercion in common/errors/errors.ts - Sync protos/execution.proto and protos/jit.proto with new fields

2026-03-09

Remove unused import from configs.proto Removed import statement for core.proto.

2026-03-07

Add BigQuery reservation configuration (#2096) * feat: Add BigQuery reservation configuration for projects and individual actions. * lint Signed-off-by: Max Ostapenko <1611259+max-ostapenko@users.noreply.github.com> * refactor: rename `bigqueryReservation` to `reservation` across configurations, protos, and related code Signed-off-by: Max Ostapenko <1611259+max-ostapenko@users.noreply.github.com> * docs: clarify BigQuery reservation fallback behavior in protos and CLI Signed-off-by: Max Ostapenko <1611259+max-ostapenko@users.noreply.github.com> * refactor: update default reservation test constant and expand project config assertions Signed-off-by: Max Ostapenko <1611259+max-ostapenko@users.noreply.github.com> * fix test reservation name Signed-off-by: Max Ostapenko <1611259+max-ostapenko@users.noreply.github.com> * added documentation for BigQuery reservation configuration Signed-off-by: Max Ostapenko <1611259+max-ostapenko@users.noreply.github.com> * fix project name Signed-off-by: Max Ostapenko <1611259+max-ostapenko@users.noreply.github.com> * fix reservation location Signed-off-by: Max Ostapenko <1611259+max-ostapenko@users.noreply.github.com> * Add reservation to a profile schema Signed-off-by: Max Ostapenko <1611259+max-ostapenko@users.noreply.github.com> * revert reservation attribute Signed-off-by: Max Ostapenko <1611259+max-ostapenko@users.noreply.github.com> * Note support status --------- Signed-off-by: Max Ostapenko <1611259+max-ostapenko@users.noreply.github.com>

feat: add --verbose flag to expose stateless installation steps of Dataform Core and identify bottlenecks (#2100) * feat: add --verbose argument to expose steps in stateless installation * chore: make verbose and quiet flags mutually exclusive * feat: use performance timing in node to avoid system clock issues * chore: do not export print * refactor: remove redundant double check * chore: print should pipe to stderr

Remove import of configs.proto from core.proto

Remove DefaultIcebergConfig message Removed DefaultIcebergConfig message definition from configs.proto.

Add import for core.proto in configs.proto

Define DefaultIcebergConfig message Move DefaultIcebergConfig message from configs.proto to core.proto. This now prevents using core.proto messages in config.proto. The way protos are used across the project lets this change work without additional changes in code.

2026-02-12

JiT Compilation protos - Add file_paths (#2093) Add file_paths field to pass accessible file paths at JiT compilation stage.

2026-02-09

JiT compilation request format (#2085)

2026-02-04

Jit data in Session (#2080) Adding a JiT data to the session. Renaming the proto field as well, it was submitted before the update of the design, but since it is not used yet, it should be fine to rename it directly.

JiT context RPC service and protos (#2079) Protos for JiT DB adapter and JiT compilation messages.

2026-02-03

Add repository snapshot destination field to NotebookRuntimeOptions (#2078) * Add repository snapshot destination field to NotebookRuntimeOptions * Improve code clarity and add more tests

2026-02-02

Action Protos - Action fields for JiT execution (#2076) Protobuf changes required for a basic JiT execution in @dataform/core.

2026-01-13

Proto changes for unit tests definition (#2065) * Proto changes for unit tests definition * Added property that allows disabling unit tests

2026-01-09

Add extra_properties struct to metadata (#2055) * add custom_attributes struct to metadata & incremental table tests * fix incremental table tests & add view & table test * change custom_attributes to extra_properties

2025-12-18

Added new properties to Test proto

2025-11-24

Allow filename overrides in declaration, tests and other assets (#2048) Making filename configuration supported in all graph nodes, right now this works only partially, for ex. for views and tables, but it is not supported for declarations and breaks for incremental tables. Explicitly adding a support for that and adding a test to prevent future regressions.

2025-11-05

Add overview metadata to sqlxconfig (#2041) * added metadata proto with overview field in view, table & inc table & added unit tests

Merge branch 'main' into add-metadata-to-sqlxconfig

2025-10-30

Compile extension interface (#2037) Introducing a compilation extension interface - the way to plug ad-hoc compilation stages as NPM package without a need to create an explicit JS definition, unblocking the transpilation of additional syntax and formats. The interface at the moment includes a compilation callback, which receives the original request and Session to operate on. Two modes are supported: 1. "prologue" - extension performs compilation in session before the regular Dataform compilation of sources is done. 2. "application code" - extension replaces the normal Dataform sources compilation.

2025-10-28

Dataform Protos - Fix //protos:dataform_proto build (#2036) `//protos:dataform_proto` target has been broken for quite a while with: ``` dataform/protos/BUILD:7:14: Generating Descriptor Set proto_library //protos:dataform_proto failed: (Exit 1): protoc failed: error executing command bazel-out/k8-py2-opt-exec-2B5CBBC6/bin/external/com_google_protobuf/protoc '--descriptor_ set_out=bazel-out/k8-py2-fastbuild/bin/protos/dataform_proto-descriptor-set.proto.bin' ... (remaining 13 arguments skipped) Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging protos/core.proto:29:3: "dataform.DefaultIcebergConfig" seems to be defined in "protos/configs.proto", which is not imported by "protos/core.proto". To use it here, please add the necessary import. ``` This was breaking only this target and not TS/JS code, but we had to fix it manually on the internal imports as well. Adding a missing import, so that the target is passing again and we no longer need to fix it by-hand on internal import.

added metadata proto with overview field in view, table & inc table

2025-10-21

Add --disable-assertions option to skip all assertions during compilation and execution (#2024) * add --disable-assertions option to skip all assertions during compilation and execution * change --disable-assertions behavior to mark assertions as disabled instead of excluding them * move disable_assertions from CompileConfig to ProjectConfig * split disable-assertions test into compile and run test cases * made the integration tests pass * add instructions for running the integration tests * fix incorrect merge conflict resolution * fix linter error for SQL injection

2025-10-07

Enable setting default connection for Iceberg table (#2030) * Enable setting default connection for Iceberg table This change enables a workflow-level default connection to be set inside workflow_settings.yaml. * Change parameters order in getConnectionForIcebergTables

2025-10-06

Enable setting default connection for Iceberg table This change enables a workflow-level default connection to be set inside workflow_settings.yaml.

2025-09-19

Rename Iceberg repo-level configs (#2022)

Rename field with default iceberg configs (#2020) This change renames defaultIcebergConfig to icebergConfig. 'Default' in the name is redundant since all the subfields have 'default_' as a prefix.

2025-09-16

Enable setting default values for storageUri substrings when constructing an Iceberg table. (#2013) * Add fields for Iceberg defaults to protos This change adds default_bucket_name, default_table_folder_root and default_table_folder_subpath to hold default values for respective Iceberg fields. These defaults will be used to construct storageUri for Iceberg tables. * Read Iceberg defaults when constructing ProjectConfig This change uses the Iceberg defaults from workflow_settings.yaml to construct the ProjectConfig. * Use defaults when constructing storageUri This change introduces default values for storageUri substring when creating an Iceberg table. If the substrings are not defined in the config block, we use default values from workflow_settings.yaml. If the substrings are not defined in workflow_settings.yaml either, we fall back to existing defaults. * Add descriptions for proto fields holding defaults This change adds comments to the proto fields holding defaults for storageUri substrings. * Move Iceberg defaults to defaultIcebergConfigs This change extracts proto fields for Iceberg defaults into a separate subblock defaultIcebergConfigs. * Clean up iceberg set up functions for tests This change simplifies the set up functions for Iceberg tests inside table_test and incremental_table_test. * Rename DefaultIcebergConfigs to DefaultIcebergConfig This naming is more consistent with the existing code.

2025-09-15

Move Iceberg defaults to defaultIcebergConfigs This change extracts proto fields for Iceberg defaults into a separate subblock defaultIcebergConfigs.

2025-09-12

Add Merge Incremental config for data preparation. (#2017) Add validation for merge incremental setting on compilation.

2025-09-08

Add descriptions for proto fields holding defaults This change adds comments to the proto fields holding defaults for storageUri substrings.

Add fields for Iceberg defaults to protos This change adds default_bucket_name, default_table_folder_root and default_table_folder_subpath to hold default values for respective Iceberg fields. These defaults will be used to construct storageUri for Iceberg tables.

2025-09-04

Add Iceberg fields to compilation logic (#2009) * Add Apache Iceberg fields to protos This change adds fields that will br used to create Apache Iceberg BigQuery tables to relevant protos. connection, table_format, file_format and storage_uri are added to core.proto. connection, table_format, file_format, storage_uri, bucket_Name, table_folder_root and table_folder_subpath are added to dataform.proto. * Add Iceberg options to compile logic This change ensures Iceberg options provided in the user's config file are correctly compiled. * Set defaults for storageUri, remove it from config This change removes storage_uri as a field users can set in their config files. It also sets defaults for tableFolderRoot and tableFolderSubpath. If tableFolderRoot is not provided to users, we default to `_dataform`. If tableFolderSubpath is not provided to users, we default to `{dataset name}/{table name}`. * Add check for valid file format This change ensures that the file_format in the user's config file is set to "PARQUET" because that is the only value currently supported. * Ensure bucket_name is defined in iceberg config block This change adds a check for bucket_name in the iceberg subblock of the user's config file. bucket_name is required when defining iceberg options. * Alphabetize imports and use T[] instead of Array<T> This fixes the build failures. * Add test for an empty file_format field * Move file_format to iceberg subblock This change moves the file_format config field from the bigquery subblock to the iceberg subblock. * Move FileFormat enum to IcebergTableConfig This change moves the FileFormat enum from TableConfig to IcebergTableConfig because that is where the enum is used. * Rename Iceberg enums in core.proto The new naming is consistent with the naming in configs.proto. * Change fileFormat defaulting logic We now only default to Iceberg if fileFormat is not defined. * Add tests for storageUri and connection validation This change adds a utils_test.ts to hold tests for methods inside core/utils.ts, and adds tests for methods that validate storageUri and connection format. * Remove contextableIcebergOpts This change removes the temporary state variable contextableIcebergOpts and sets Iceberg options directly. * Refactor storageUri construction * Throw regular instead of session errors It is not necessary to throe session errors because regular errors get wrapped into session errors a few layers above. * Add tests for getters in utils This change introduces testing for Iceberg-related getters in utils.ts * Lint fixes * Refactor Iceberg table tests This change refactors the Iceberg table tests to check the entire bigquery object and adds a test with non-Iceberg and Iceberg bigquery options. * Add Iceberg options to IncrementalTableConfig * Add Iceberg options to incremental table compilation This change adds Iceberg options to the compilation logic for incremental tables. Unit tests are added to verify the changes. * Remove validation of iceberg in bigqueryoptions This change removes the validation of iceberg as part of the bigqueryoptions in the unverified config. This validation is not necessary because iceberg is not part of the bigqueryoptions. * Use existing method to set iceberg proto fields This change removes the dedicated method for setting iceberg fields and reuses the existing method for bigquery options.

draft to convert default values from workflow_settings.ts to protos/core.proto - ProjectConfig

2025-09-02

updated rest of files to match {dynamicVar} naming convention

Rename Iceberg enums in core.proto The new naming is consistent with the naming in configs.proto.

Move FileFormat enum to IcebergTableConfig This change moves the FileFormat enum from TableConfig to IcebergTableConfig because that is where the enum is used.

2025-09-01

Move file_format to iceberg subblock This change moves the file_format config field from the bigquery subblock to the iceberg subblock.

2025-08-29

Set defaults for storageUri, remove it from config This change removes storage_uri as a field users can set in their config files. It also sets defaults for tableFolderRoot and tableFolderSubpath. If tableFolderRoot is not provided to users, we default to `_dataform`. If tableFolderSubpath is not provided to users, we default to `{dataset name}/{table name}`.

2025-08-25

Prototype runtime parameters.

Add Apache Iceberg fields to protos This change adds fields that will br used to create Apache Iceberg BigQuery tables to relevant protos. connection, table_format, file_format and storage_uri are added to core.proto. connection, table_format, file_format, storage_uri, bucket_Name, table_folder_root and table_folder_subpath are added to dataform.proto.

2025-05-18

Fix materialized views (#1970) * Add support for partitionBy and clusterBy fields on materialized views. * Update docs to incl;ude new fields on ViewConfig.

2025-05-16

Add support for partitionBy and clusterBy fields on materialized views.

2025-04-03

Add builtinAssertionNamePrefix field to WorkflowSettings. (#1956) * feat: Add implementation for builtinAssertionNamePrefix. * test: Add unit tests for builtinAssertionNamePrefix. * docs: Update docs with builtinAssertionNamePrefix field info. * test: Cover projectConfig with unit test. * feat: Make prefix as first part of the name and adjust tests. * refactor: Clean up index_test.ts * refactor: Clean up index_test.ts again * refactor: Clean up index_test.ts again and again

2025-03-05

Add support for schema and dataset configs in data preparation config. (#1935) This allows data preparations to configure override similar to all other actions.

2025-03-03

Remove the unused automatic load mode. This doesn't apply to compiled data preparations.

2025-02-20

Clean up remaining ekrekr todos (#1927) * Start removing todos * Cleanup more todos * Remove another todo * More old TODO tidying * Add fixed filename resolution for filepath of inline assertions * Move inline assertion generation to shared definition * Remove last todo * Fix lint

2025-02-12

Add typedoc generation of the Dataform JS API (#1913) * Experiment with typedoc comments * Add typedoc configuration json. Now can just run 'typedoc' to run * Add working generation, just via npx typedoc * Add documentation for remaining publish session methods * Add docs changes * Add typedoc generation to docs regeneration script, run * Add typedocs to the assertion action, tidy docs structure * Add docs for more actions, add more details on consumption * Add docs for the remaining classes * Tidy, improve JS API references * Fix lint * Update typedoc and typedoc markdown to versions that are in airlock * Fix dataprep docs generation, remove sources from typedoc generation * Add hide private

2025-02-11

Added Load Configuration setting for Data Preparation SQLX format. (#1909) * Added Load Configuration setting for Data Preparation SQLX format. # Conflicts: # core/main_test.ts * Addressed PR comments * Addressed suggestions * Addressed PR comments

2025-02-03

SQLX support for Data Preparations (#1891) * Added SQLX support for data preparation. # Conflicts: # core/utils.ts # Conflicts: # core/actions/data_preparation.ts # core/main_test.ts * Removed config interfaces, using proto config instead. # Conflicts: # core/actions/data_preparation.ts * Removed config interfaces, using proto config instead. Added requested test case. * Removed package.json changes * Addressed PR comments

2025-01-30

Add support for OnSchemaChange field for incremental tables. (#1902) * Add support for OnSchemaChange field for incremental tables. * Fix docs * Fix OnSchemaChange docs section * Update comments and simplify tests.

2025-01-09

Add notebook runtime template resource name field to NotebookRuntimOptions (#1890)

2024-12-12

Add assert config option for the JS api

2024-09-19

Remove data preparation proto definition. (#1849) * Remove data preparation proto definition. Clean up unused code in data preparation action. * Removed missed unused code

2024-09-11

Update Data Preparation processor to remove proto dependency. (#1839) * Update Data Preparation processor to remove proto dependency. The data preparation in the compiled graph now contains the resolved YAML after compiling. * Fixed compilation issue

2024-09-04

Bump dataprep proto to latest config (#1825) * Bump dataprep proto to latest config * Remove license header

2024-08-16

Sync proto structure and message names for data preparation. (#1811) * Sync proto structure and message names for data preparation. Moved data preparation into another package. * Fix broken test.

2024-08-15

Update output for Data Preparation processing (#1808) * Updated data preparation proto to only carry binary encoded data preparation contents after processing the data preparation definition. * Addressed PR Comments. * Addressed final PR comments * Fixed compilation issue.

2024-08-07

Remove some outdated configs references, fix generated formatting (#1799)

2024-07-29

Handling for Data Preparation actions (#1789) * Add logic to handle Data preparation definitions and parse data preparation YAML into the proto representation of the operation. * Addressed PR Comments * Fixed lint warnings

2024-07-18

Added proto definitions for Data Preparations (#1788) * Added proto definitions for Data Preparations * Moved Data preparation protos into a separate file

2024-06-27

Add support for action configs table assertions and hermeticity, add action configs fields tests (#1762) * Progress adding action config tests for different actions * Add remaining table test types, fix support for table assertions and hermeticity * Regenerate docs, tidy merge files