Proto commits in dbt-labs/dbt-core

These commits are when the Protocol Buffers files have changed: (only the last 100 relevant commits are shown)

2026-06-16

Capture thread start telemetry errors * Capture thread start telemetry errors * Send thread start errors before analytics * Address thread start telemetry review GitOrigin-RevId: d223790a8056072ec41067b2d28c568b0188e087

The documentation is generated from this commit.

telemetry: capture manage_state on invocation span * telemetry: capture manage_state on invocation span Adds a `manage_state` boolean to the InvocationEvalArgs telemetry proto so we can track when users enable dbt State management (auto-deferral). The value is the resolved flag from `--manage-state`, `DBT_ENGINE_MANAGE_STATE`, or `flags.manage_state` in dbt_project.yml / user settings, populated onto the structured invocation span. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * telemetry: source manage_state from existing run_cache_service Per review: drop the redundant EvalArgs.manage_state field. In production `run_cache_service` is only ever assigned the resolved manage-state value (in configure_run_cache) or the default false, so the invocation telemetry span can read it directly instead of shadowing it with a new field. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * changelog: add entry for manage_state telemetry Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * Add issue reference to telemetry change * test: regenerate otel telemetry golden files for manage_state; move changelog to repo root Add `manage_state` to the eval_args in all otel telemetry snapshot golden files (.otel.jsonl / .otel.parquet.jsonl) to reflect the new InvocationEvalArgs field. Move the changelog entry from fs/sa/.changes to the repo-root .changes/unreleased so the changelog-existence check passes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> GitOrigin-RevId: 26113e0c59c52a6d82e55072b5803019ef682c2e

2026-06-13

feat(telemetry): emit Vortex Login event on dbt login * feat(telemetry): emit Vortex Login event on dbt login; sync protos to v1.0.528 Adds a Login Vortex event emitted at the end of every `dbt login` attempt (success or failure). The event captures: - success/failure and which OAuth flow completed (platform or state) - user cookie from ~/.dbt/.user.yml - project_id (MD5 of project name) if run inside a dbt project - platform identity fields (user ID, account ID, account identifier) decoded from the JWT, falling back to the cached session token on failure so expired-credential runs still populate identity context Also syncs proto definitions to v1.0.528 and regenerates Rust bindings. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * chore: add changie for Login Vortex event Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(telemetry): use log_proto instead of log_proto_and_shutdown in login_event log_proto_and_shutdown triggered Vortex flush from inside the instrumented async context, causing the background producer task to emit tracing events with no root span and hitting the debug_assert in data_layer.rs. Using log_proto lets the event get queued and flushed by the normal invocation_end_event shutdown path, which runs inside spawn_blocking instrumented with the invocation span. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: propagate tracing span into license fetch spawned task did_login spawns a tokio task to fetch the platform license without inheriting the caller's span. That task then uses block_in_place + Handle::block_on to drive an async future synchronously; exit_runtime clears the thread-local span context before the closure runs. Two-part fix: - dbt-feature-gating: instrument the spawned task so it enters with the invocation span, covering the warn! on failure too - dbt-licensing: capture Span::current() before block_in_place so the future passed to block_on carries the span across the exit_runtime boundary Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * chore(lint): remove unused import --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> GitOrigin-RevId: a78cfb5b2efff5610c9f0e74f227f252e7878de4

2026-06-11

mv fs/sa/crates/dbt-run-cache fs/sa/crates/dbt-state * mv fs/sa/crates/dbt-run-cache fs/sa/crates/dbt-state * Rename dbt-run-cache crate to dbt-state Update package/lib name and all workspace, dependency, and import references to follow the directory rename. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> GitOrigin-RevId: 008e87e880655d0e4583eba79db66ab646098600

feat(telemetry): emit catalog_type on RunModel event * feat(telemetry): emit catalog_type on RunModel event Add a catalog_type field to the RunModel telemetry event, mirroring how PR #9888 added catalog_name. catalog_type is resolved from the model's catalog_name by following the catalog's active write integration in catalogs.yml (same resolution the adapter uses at materialization time). The proto binding is hand-stubbed (tag 23) pending merge of proto PR dbt-labs/proto#629; it will be regenerated by sync_protos.sh pull. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(telemetry): resolve catalog_type for catalogs.yml v2 Wire up v2 catalog_type resolution in resolve_catalog_type. v2 declares `type` directly on each catalog (no write integrations), so resolve via view_v2() and read catalog.catalog_type. v1 path unchanged. Addresses review feedback on #10761 (VersusFacit: "Add v2 :D"). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore(proto): sync catalog_type from merged proto#629 Replace the hand-stubbed RunModel.catalog_type field with the real generated binding now that dbt-labs/proto#629 has merged. Adds the field to the vendored fusion.proto and regenerates via `cargo xtask protogen`. Scoped to catalog_type only (catalogs count is proto#630 / fs#10762). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> GitOrigin-RevId: fefcef2fa15555d4a14c13b7daacf1dbd8449f8d

2026-06-10

feat(telemetry): emit catalogs count on ResourceCounts event * feat(telemetry): emit catalogs count on ResourceCounts event Add a project-wide `catalogs` count (int32, tag 18) to the parse-time ResourceCounts telemetry event. The count is derived from the parsed catalogs.yml (DbtState.catalogs) at the compilation call site and threaded into resource_counts_event. The proto binding is hand-stubbed pending the merge of dbt-labs/proto#630; once that lands the field will be regenerated via sync_protos.sh pull. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore(proto): sync catalogs count from merged proto#630 Replace the hand-stubbed ResourceCounts.catalogs field with the real generated binding now that dbt-labs/proto#630 has merged. Adds the field to the vendored fusion.proto and regenerates via `cargo xtask protogen`. Scoped to the catalogs count only (catalog_type is proto#629 / fs#10761). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * style: use Vec::len for catalog count, matching surrounding counts resource_counts_event computes every other resource count with `.len() as i32`; catalogs_seq() returns a &yml::Sequence (a Vec), so use `.len()` instead of `.iter().count()` for consistency. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> GitOrigin-RevId: 952b4346a1e73bb975ed29f55a73285fde76827e

2026-06-01

feat(dbt-onboard): drive onboarding auth via AuthChain + dbt login (#10646) * feat(proto): add Login screen variant to OnboardingScreen enums * refactor(dbt-onboard): extract ask_have_project helper * feat(dbt-onboard): add LoginStep that shells out to dbt login * feat(dbt-onboard): replace dbt_cloud.yml autodetect with AuthChain + login prompt * refactor(dbt-onboard): drop LinkAccount step (replaced by LoginStep) * test(dbt-onboard): unit-test retry parsing; split WelcomeStep into named helpers * docs(dbt-onboard): update README flow for login-based onboarding * chore: changie entry for dbt-onboard login flow * fix(dbt-onboard): invoke current binary for dbt login subprocess PATH-based lookup of 'dbt' is intercepted by version-manager shims (asdf, mise) and fails when no version is configured. Use None so run_streaming_program resolves the current_exe(), matching the existing pattern in dbt_parse, compile_sa_off, and other steps. Also commits Cargo.lock for the dbt-platform-auth dep introduced in d17b40a282. * fix(dbt-onboard): split multi-line connect prompt to avoid double-render dialoguer::Confirm re-renders its prompt after the user submits an answer, which doesn't handle embedded \n\n cleanly — the first line of a multi-line prompt prints twice. Move the explanatory text into write_paragraph calls before the Confirm, and keep the Confirm prompt single-line, matching every other prompt in this crate. * feat(dbt-onboard): skip strict static analysis when not signed in to dbt platform Strict mode requires a dbt platform account. When the user declines to connect (or the login flow fails and they choose Continue), the onboarding now skips the 'Want to try strict mode?' prompt at the baseline-compile step and routes straight to CompileSaBaselineSuccess. Track auth state via a new IS_AUTHED context-store key set by: - WelcomeStep::show_authed_path (AuthChain auto-resume succeeded) - LoginStep on successful 'dbt login' exit Absent / false at CompileSaBaseline = skip strict. * fix(sdf-features): exit after `dbt init --fusion-upgrade` onboarding succeeds OnboardingFlow::run returns Ok(()) on the Success terminal state, which will_init_project propagated unchanged - so after onboarding finished cleanly, control returned and the regular `dbt init` flow then ran on top. The intent (per the inline comment) is that onboarding runs INSTEAD of init; collapse both branches into a single exit-with-status path so this is true regardless of success or failure. * feat(dbt-onboard): offer sign-in at baseline-success prompt instead of skipping strict Previously, unauthed users hit a skip message after baseline compile and the strict step was unreachable. Now the baseline-success prompt itself reframes for unauthed users: 'Yes, sign in to dbt platform and try strict mode.' On choosing that, route through LoginStep with a context-stored return-screen pointer; LoginStep honors it on success and goes directly to CompileSaStrict instead of its default ProfileCheck. The authed path (Yes, let's try running in strict mode -> CompileSaStrict) and the EndEarly branch are unchanged. * Set author to 'dakota' in onboarding feature Updated author field in the onboarding feature description. GitOrigin-RevId: 295ba64f3d2ddb28dc2e3ca1a25fd3551d6638df

Populate Vortex invocation distribution (#10626) * Discover proto checkout in sync script * pull updated protos * Populate Vortex invocation distribution * Move invocation distribution onto event emitter * Configure invocation distribution via builders GitOrigin-RevId: 6c2e69d0a71cdb0df719991a7be16583ea7ba316

Bring Renderable and RunCache out to SA (#10578) * Move task-cache into SA fully - refactor TaskRunnerCtx * remove unit test dependency on sdf-frontend pt.1 * Unit test should use TypeOps for formatting types * Fix fmt * changie * Add TypeOps::format_ident and remove sdf-frontend format_ident from unit_test Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> * Fix failing unit tests * move run-cache to oss * fix snowflake quoting handling in oss * Fix format ident failing test * Fix copybara failing workflow * Move renderable tasks to SA along with task cache dependency * Use typeops to fmt identifier * Remove sdf_frontend deps from unit_test: FormatAs and TypeConversion - Replace format_fqn to use TypeOps::format_ident per component instead of FullyQualifiedName::format_as (FormatAs trait) - Add TypeOps::can_cast_literal_to_type with default Ok(true); override in sdf-adapter TypeOpsImpl for Snowflake and DuckDB decimal checks (TypeConversion trait) - Fix pre-existing fmt and clippy warnings across dbt-tasks-sa Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> * enable tls-native-roots and transport for tonic --------- Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> Co-authored-by: Felipe Oliveira Carvalho <felipekde@gmail.com> GitOrigin-RevId: 098cd972f2ded6efb748bdee2a01ac2a31c13248

Record structured clone cache reasons (#10520) * Record structured clone cache reasons * Simplify clone reuse node status * Rename clone cache telemetry reason GitOrigin-RevId: 5124ccb9497b395e5f29cb87c88ca12dda98228a

Add error code names to log telemetry (#10519) * Add error code names to log telemetry * Address log telemetry review comments GitOrigin-RevId: 8ddccad0da2ae27b04a1c79fa6a91e5b83c1d6c7

Exclude connection waits from node duration (#10421) * Exclude connection waits from node duration * Remove TUI progress unit tests * Rename node wait metric to idle time * Move connection wait formatters * Exclude idle time from node evaluated logs * Update workspace hack formatting * fix hakari * formatting bug GitOrigin-RevId: 42d19949a0f4cc7ad6ccedce0712c6daf99053a1

Add trace events for adapter connection lifecycle (#10408) * add connection open/close events * chore: add connection telemetry changelog GitOrigin-RevId: 82a02c8b0e7a99019c3d4ee7beb40fa3db07bd14

Instrument adapter connection backpressure (#10293) * use intrumentation helpers * Instrument adapter connection backpressure * Add adapter connection wait changelog * Document adapter connection wait event * Refresh remaining telemetry goldens * add trace level metrics to connection backpressure span * goldies GitOrigin-RevId: ee389219dea03d036ac45dd241bdbdb523f50de7

feat(run-cache): port gRPC run-cache service integration from monorepo (#10184) * feat(run-cache): port gRPC run-cache service integration from monorepo Ports 17 merged PRs (#57–#303) from dbt-fivetran-collab/june-1-tmp-monorepo fs/ subtree into this repo, forked at v2.0.0-preview.175. **What's included** Run-cache gRPC service (Skip / Execute / Clone decisions): - New files: dbt-run-cache/{build.rs, proto/, src/metadata_cache.rs, src/proto.rs, src/request_builder.rs, src/service_client.rs, src/service_config.rs, src/view_traversal.rs} - New task files: dbt-tasks/src/{run_cache_lifecycle.rs, runnable/run_cache_{dev_clone,request,service}.rs} - Run-cache service dispatch wired into RunExecutionPath::Remote; after_success forwarded so ConfirmExecution fires and Skip/Clone work on subsequent runs - SAO (Redis) task cache replaced with TaskCacheNoop when gRPC service is active to prevent interference between the two caching layers Run-cache auto-deferral: - synthesize_defer_nodes synthesises prod-target defer nodes from the run-cache service config when no manifest-backed state is present - defer_common called immediately after synthesis so the node resolver is populated with deferred relations before precompile runs View-definition traversal (supporting Clone decisions): - fetch_view_definitions added to Snowflake, BigQuery, Databricks adapters - New view_definition.rs type; ViewDefinitionTraverser in dbt-run-cache Telemetry: - NodeCacheReason::Clone variant added to proto + generated code - task_spans.rs maps ReusedNoChanges("Cloned…") → Clone reason - formatters/node.rs emits "Cloned from cached relation" terminal message Other fixes ported: - DbtMaterialization::Incremental → RelationType::Table (fixes is_incremental() returning false after dev-clone) - run_cache_service flag gate added to sdf-features - RunTasksArgs.run_cache_service field; RunCacheLifecycle threaded through run_tasks_with_listener → context_factory → TaskRunnerCtxInner - TaskRunnerCtxInner (dbt-tasks-core) extended with run-cache fields (run_cache_metadata, service_requested/config/client, view_traverser, execute, run_cache_deferred_fqns) * fix(dbt-main): allow dbt-run-cache as sdf-frontend wrapper in deny.toml The run-cache merge introduced a new dbt-run-cache crate that depends on sdf-frontend. dbt-main transitively pulls it in, which triggered the unmatched-wrapper / banned errors in the cargo-deny consistency check. * fix(dbt-index): remove duplicate test_parquet_grain_declared_priority_chain The function was ported from the monorepo in the previous commit but had already been added independently to main via b9c7e0990a (dbt-index: Deferral Index #10117). Both definitions were identical; keeping the fs-repo version per source-of-truth policy. * refactor(dbt-tasks-core): move run-cache fields out of TaskRunnerCtxInner The public dbt-tasks-core crate was depending on the private dbt-run-cache crate, which broke the Copybara public export: cargo metadata in the copybara-transformed tree failed with "dependency.dbt-run-cache was not found in workspace.dependencies" because the workspace entry is stripped on export. Move the six run-cache-related fields off TaskRunnerCtxInner and onto ExtendedCtxImpl (already private, already a designated holder for "information not coupled to the tasks runner"). Access sites in crates/dbt-tasks/ go through a new TaskRunnerCtx::ext() accessor. Drops the dbt-run-cache dep from dbt-tasks-core/Cargo.toml. * Mr. Clipster * fix(run-cache): remove AUTODEBUG prints leftover from monorepo port The eprintln! in RefFunction was leaking into stdout and breaking goldie snapshot tests; the two tracing::warn! calls in RunCacheProfileResolver were equally unintended. * fix(run-cache): require explicit opt-in for gRPC service The gRPC Run Cache service was initializing whenever the legacy SAO task cache was active (`run_cache_mode.write_cache()`), conflating two unrelated features. State-aware tests using `--task-cache-url redis://...` silently triggered service init, which then failed in CI (no RUN_CACHE_OAUTH_CLIENT_SECRET) and leaked warnings into goldie outputs. Drop `|| arg.run_cache_mode.write_cache()` from both gates so the service is a no-op unless the user explicitly opts in via `--run-cache-service` or `RUN_CACHE_*` env vars: - run_cache_lifecycle.rs: initialize_run_cache_service - compilation.rs: run_cache_auto_defer_requested Also bundles PR-feedback cleanups: - context_factory.rs + task_runner.rs: consolidate four run-cache fields (requested / metadata / config / client) into a single `run_cache_lifecycle: RunCacheLifecycle` on the factory, destructured at ExtendedCtxImpl construction. - runnable/mod.rs: rename `record_sao_skip` → `record_cache_skip`; final arg renamed `actual_hash` → `source` since it also tags run-cache-service skips. Adds comment on SAO/service mutual exclusion. - runnable/run_cache_service.rs: refactor `prepare_write_only_execution_record` to an `else if` chain instead of early returns. - metadata/view_definition.rs: doc comment no longer references dbt-run-cache (traversal now lives in callers). - schemas/common.rs: explain why `DbtMaterialization::Incremental → RelationType::Table` matters for dev-cloned incrementals (is_incremental() checks relation.type). * fix(run-cache): run defer pipeline whenever defer_nodes is present The run-cache auto-defer flow synthesizes profile-target defer_nodes on augmented_resolved_state before precompile, but precompile gated all defer machinery (defer_common, defer_sa_upstreams, set_defer_context_on_resolver, persistence) on `arg.defer && defer_nodes.is_some()`. Since auto-defer runs without `--defer` (arg.defer == false), the pipeline never fired for it, which broke 8 Snowflake integration tests: test_dev_run_clones_prod_and_appends test_dev_run_clones_when_prod_table_has_no_cache_metadata test_dev_run_defers_when_upstream_exists test_dev_run_with_clone_incremental_in_dev_always test_dev_run_defer_favor_state test_incremental_multiple_runs_no_clone_across_dev_envs test_dbt_show_favor_state_defers_to_prod test_deferred_dependencies_use_lenient_matching The monorepo's port (314adec) papered over this with a one-shot defer_common call before precompile and `BTreeSet::new()` for run_cache_deferred_fqns — but that skipped defer_sa_upstreams and set_defer_context_on_resolver entirely, and lost defer_common's deferred_unique_ids return value. Widen precompile to gate on defer_nodes.is_some() regardless of source: - dbt-precompile/src/lib.rs: - Take any pre-populated augmented_resolved_state.defer_nodes (run-cache auto-defer source) before manifest loading. Manifest wins when both are present. - Drop `arg.defer &&` from the five gate sites; defer pipeline now fires whenever defer_nodes is set, regardless of source. - Unconditionally persist defer_nodes back to augmented_resolved_state. - Return (StaticAnalysisBucketsImpl, HashMap<CanonicalFqn, String>) so callers can thread deferred_unique_ids onward. - dbt-main/src/compilation.rs: - Drop the inline defer_common call from the auto-defer block; just synthesize and store on augmented_resolved_state.defer_nodes. Precompile now runs the full defer pipeline against them. - Destructure precompile's return value, pass deferred_unique_ids.keys() to run_tasks_with_listener (replacing the BTreeSet::new() placeholder that broke lenient dependency matching). Bundled PR-feedback refactor (orthogonal to the bug fix): - metadata_adapter.rs + metadata/mod.rs: rename run_cache_relation_metadata{,_inner} → relations_exist{,_inner}, return BTreeMap<String, bool> directly and drop the RunCacheRelationMetadata wrapper struct. Only `exists` was ever consumed. - runnable/run_cache_dev_clone.rs: caller renames to match. * fix(seed): restore Snowflake column-change hint on with-task-result path The monorepo port (314adec209) factored RunTask::run_task's "default execution" branch into execute_remote_node_with_after_success and dropped the post-call wrap to seed::maybe_resolve_remote_seed_column_hint. The resolver still ran on the no-task-result path (execute_remote_node_no_result), but Snowflake seeds always carry a task result and so lost the user-facing diagnostic: Seed 'my_seed' has column schema changes that require a full refresh. Existing columns: [...] New CSV columns: [...] Re-run with `--full-refresh` to drop and recreate the table. Restore the call inside execute_remote_node_with_after_success so the with-task-result path enriches the FsError before cache finalization. The resolver early-returns for non-seed nodes. Repros / verifies: commands_seed::seed_column_change_without_full_refresh_snowflake. * refactor(run-cache): address PR #10184 review Move run-cache-specific code out of dbt-main per Felipe's review: - Comment 1 + 3: call site and the full RunCacheProfileResolver family (~440 lines + 6 helpers) move from dbt-main into a new dbt-precompile::run_cache_defer module. precompile gains a jinja_env parameter and invokes the synthesizer at the top of defer_pre_classify_static_analysis, replacing the prior implicit augmented_resolved_state.defer_nodes pre-population hand-off. Manifest-vs-synth precedence and warn-and-continue degradation semantics preserved. dbt-main drops 7 imports plus the dbt-profile + dbt-run-cache crate deps. - Comment 2: RunCacheLifecycle::initialize moves out of compilation.rs into run_tasks_with_listener, where arg + execute_mode were already in scope. compilation.rs drops the RunCacheLifecycle import. * refactor(run-cache): unify CacheDecision into RunCacheServiceDecision Collapse the SAO-specific CacheDecision enum into RunCacheServiceDecision so both caching paths produce the same decision shape. Skip carries an optional sao_stored_hash, Execute an optional sao_guard, and a Disabled variant covers the cache-off case. `check_cache` returns the unified type directly; node_hash and finalize_sao_guard replace the removed CacheDecision methods. The Remote dispatch is now a single match on all four variants and execute_remote_node_with_after_success no longer recomputes the decision. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * more cleanup * refactor(run-cache): proto stopgap and emit_* sweep Address mishamsk's review on #10184: - Drop NODE_CACHE_REASON_CLONE from the public proto; carry the cache decision text via a new optional NodeCacheDetail.message instead. Formatter prefers the message when set, falls back to enum-derived default. Keeps "Cloned from cached relation" terminal text without committing to a Snowflake-named or Skipped-categorized variant; field is marked stopgap pending the planned NodeOutcome::Success(Clone). - Replace bare tracing::{debug,warn,info,error}! macros across the run-cache surface with dbt_common::tracing::emit::* APIs. Drop the meaningless `target =` structured-field usage. Warns route through the new RunCacheServiceWarn ErrorCode (1410). Internal-only diagnostics moved to trace; "service disabled by configuration" stays at debug as a user-facing operational signal. * more cleanup * fix(run-cache): restore run_stats insert on Clone-success path The merge resolution in 828c34cd3d combined the if/else shape from 4d10576f13 with the Err-warn block from 903073cb1f and ended up using `return Ok(status)` for the Clone-success arm. That early-returned from `RunTask::run_task`, bypassing `run_stats.insert`, so the cloned node never appeared in run_results.json. Six integration tests started failing with `Node 'X' not found in run_results.json`. Capture the Clone success into a sentinel and short-circuit the model body via `if let Some(status) = clone_status`, so the cloned status flows back into the outer `result` and post-processing (run_stats, report_completed, telemetry) still fires. * proto gen * build break fix --------- Co-authored-by: Alexander Bogdanowicz <alexkbog@gmail.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> GitOrigin-RevId: de7b2b699377af3d56a19ee159c2b2581b277fc1

chore(telemetry): add some new catalogs.yml info into RunModel (#9888) * chore(telemetry): add some new catalogs.yml info into run model information * chore(agents): Add instructions for working with Vortex. * chore(fusion-telemetry): sync relevant proto files GitOrigin-RevId: 6fa12d263dad6e4ddd35ea9ec1c1de49a202cdb0

feat: Add NodeStatus::SucceededWithWarning (#9311) * feat: Add new nodeSucceedWithWarn execution status * feat: propogate warn-on-success through run_results.json and CLI summary * test: update gitignore * chore: add change log entry * chore: updated opt-out list registry * refactor: simplify proto * test: update golden files + fix for failing tests * refactor: address code review: wrap NodeWarningOutcome in message * doc: proto comments * refactor: minor refactor helper function + fix: update golden files * chore: regen proto * code review: - Add NodeSubOutcome enum to replace TestOutcome as general sub-outcome discriminator - Add TelemetryNodeWarnOutcome middleware which automatically stamps WithWarnings - Clean up for helper surface and compact call-sites * chore: update change log to be more user friendly * refactor: remove find_and_record_span_status_from_attrs callsites now that middleware is in place * Address code review: Clean up SucceededWithWarning implementation * address code review: use data_provider API instead of find_and_update_span + update golden files and comments * fix: update tests + stamp warn authoritatively * chore: update golden files GitOrigin-RevId: fccc2ac36a9fea10354b2be9ca20e44f39b8d8f0

feat(telemetry): add group field to NodeProcessed telemetry events (#8716) * feat(telemetry): add group field to NodeProcessed telemetry events Add `Group` (name + owner with email/slack) to the `NodeProcessed` proto and wire it through to the otel parquet output. Proto / generated code: - Add `Owner` and `Group` messages to the node proto - Add `group` field (tag 44) to `NodeProcessed` - Regenerate serde impls and binary descriptors Telemetry serialization: - Add `group: Option<Group>` to `ArrowAttributes` - Add nested Arrow schema for `Group` and `Owner` (including email as `List<LargeUtf8>` and slack as `LargeUtf8`) to `create_arrow_schema()` - Populate `group` in `NodeProcessed::to_arrow_record()` and round-trip it back in `from_arrow_record()` Schema / parser: - Add `group_ref: Option<Arc<DbtGroup>>` to `DbtTestAttr` - Override `get_node_processed_event()` for both `DbtModel` and `DbtTest` to populate `Group` from the attached `DbtGroup` (name, owner email from `StringOrArrayOfStrings`, slack from `__common_attr__.meta`) - Add `infer_and_apply_test_groups()` in `resolve_data_tests.rs` that copies each test's attached model's `group_ref` onto the test after resolution; call it from the resolver after `infer_and_apply_primary_keys` * test: add unit tests for group telemetry wiring Add tests for two functions introduced in the group-notification feature: - `infer_and_apply_test_groups` (4 tests): verifies group_ref is copied from a model to its attached tests, and that missing/dangling references are handled gracefully. - `group_from_dbt_group` (3 tests): verifies DbtGroup → telemetry Group proto conversion, including email as array vs. single string, slack from meta, and missing optional fields. Also fix exhaustive DbtTestAttr/DbtModelAttr struct literals in primary_key_inference.rs and dbt-lsp/column_handler.rs that were missing the new `group_ref` field. * updated registry * Pivot to only outputting group name in otel * feat(telemetry): emit Groups log event before node execution Promote the Groups proto message to a first-class telemetry log event and emit it in task_runner.rs before node execution, reporting all groups defined in a project (name + owner with email/slack). - Add groups to protogen.toml (static_full_name, pbjson_build, faker) and regenerate to produce serde impls and StaticName - Implement ProtoTelemetryEvent and ArrowSerializableTelemetryEvent for Groups (log event, EXPORT_ALL, JSON payload serialization, owner info redacted in clone_without_sensitive_data) - Register Groups in the public telemetry registry; move it off the opt-out list (Group and Owner remain as helper types) - Inline DbtGroup → proto Group conversion in task_runner.rs (avoids a dbt-telemetry → dbt-schemas circular dependency) - Remove stale test_group_from_dbt_group_* tests that referenced the removed group_from_dbt_group function * refactor(telemetry): replace Owner.slack with Group.meta in groups proto The slack field on Owner was populated from DbtGroup.__common_attr__.meta, which is a group-level attribute. Replace it with a map<string, string> meta field on Group to capture the full meta map and support future values beyond slack. * Add golden test for Groups event in otel * Update goldent test to include a test * update golden test to include a model and test without a group to ensure that they do not include the group attribute * Remove group_ref and just use group * no sensitive data * Use Cow for group in arrow * rename groups.proto to follow conventions * rename email to emails * refactor proto group creation to function * refactor: inline test group assignment into resolve_data_tests Instead of creating tests with `group: None` and running a separate `infer_and_apply_test_groups` pass afterward, pass the resolved models map into `resolve_data_tests` and set the group at construction time. Removes `infer_and_apply_test_groups` and its unit tests. * Clean up setting group on NodeProcessed * Update golden tests * move group.proto from dbt-telemetry to dbt-telemetry-private Group events should be in our private protos. Move group.proto to dbt-telemetry-private and update the package from v1.public.events.fusion.group to v1.events.fusion.group to match the private proto convention. - Proto file moved to crates/dbt-telemetry-private/include/... - Generated files (rs + serde) and schema impl moved to dbt-telemetry-private - Groups registered in PRIVATE_TELEMETRY_EVENT_REGISTRY (removed from public) - dbt-tasks updated to import group types from dbt-telemetry-private - Golden test updated to reflect new event_type name - protogen.toml updated to codegen under v1.events.fusion.group * Fix Groups import to be cleaner * Addressing PR feedback * Use CLI Hook to emit Groups event, instead of sending from task runner * Add did_resolve_project to CliExtensionHooks This function supports emitting the Groups event via the hooks framework. The existing hooks are emitted after execution, which is too late for the Groups event. GitOrigin-RevId: 0c077dfdeabc93c526cb714392e746ba4fb25fca

feat(dbt-index): add anonymous usage telemetry via Vortex (#9220) * feat(dbt-index): add anonymous usage telemetry via Vortex Adds lightweight telemetry to dbt-index for tracking activation, engagement, and agent usage patterns (PRD-292). Events are sent to the Vortex analytics pipeline as protobuf over HTTP. Three events: - DbtIndexInvocation (start/end per command): tracks installs, WAP, time-to-value, and command performance - DbtIndexResourceCounts (per ingest): project size segmentation - DbtIndexToolCall (per MCP tool call in serve mode): agent adoption, tool reliability, and latency Privacy: only pseudonymous user_id (UUID cookie), MD5-hashed project_id, command names, and aggregate counts. No SQL, file paths, error messages, or tool arguments are collected. Consent: three-layer opt-out via --no-send-anonymous-usage-stats flag, DO_NOT_TRACK=1, or DBT_SEND_ANONYMOUS_USAGE_STATS=false. Implementation is self-contained (~400 lines in telemetry.rs) with no dependency on dbt-common, vortex-client, or proto-rust. Uses a background thread with mpsc channel for non-blocking sends and 200ms shutdown timeout. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat(dbt-index): add dbt Cloud context to telemetry events Add account_id, account_host, cloud_project_id, account_identifier, project_name, adapter_type, and dbt_version fields to DbtIndexInvocation. Cloud context is resolved from ~/.dbt/dbt_cloud.yml using the active project (context.active-project), not project name matching. account_identifier is fetched from the dbt Cloud admin API and cached to ~/.dbt/.dbt_index_cloud_cache.json with a 24h TTL. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat(dbt-index): add missing resource counts + emit on every invocation Add exposures, metrics, groups, and semantic_models to DbtIndexResourceCounts (proto tags 11-14). Non-ingest commands now also emit resource counts by querying the loaded DuckDB index, so every invocation has project size context for segmentation. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * refactor(dbt-index): simplify telemetry integration - Extract `with_context` helper to eliminate 10 repetitive match arms - Consolidate 5 SQL queries into single UNION ALL for resource counting - Try YAML parser first for dbt_cloud.yml (actual format) before JSON Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> GitOrigin-RevId: 5de1cef344282c2d1e58edf543519ca2736a3935

Enhance onboarding with static analysis mode and messaging updates (#8334) * Basic skeleton for baseline steps * Skeleton for new compile baseline step * Start building out baseline steps * Enhance onboarding experience by updating success messages and adding next steps for users. Improved messaging for baseline and strict compile success, including links to dbt VS Code extension features and resources. Refactored utility function usage for better consistency across steps. * Add static analysis mode updates for onboarding steps Implemented a new utility function to set the static analysis mode in dbt_project.yml based on the onboarding outcome. Updated the CompileBaselineSuccessStep and SuccessStep to call this function, ensuring the static analysis mode is set to "baseline" and "strict" respectively. This enhances the onboarding experience by automatically configuring project settings. * Refactor static analysis mode handling in onboarding steps Updated the onboarding steps to utilize a new utility function for setting the static analysis mode in dbt_project.yml. The CompileNoSaFailStep and CompileStrictFailStep now call this function to configure the static analysis mode based on the step outcome. Additionally, improved the success message in the SuccessStep to link to the correct documentation for strict mode configuration. * Add changelog entry * Refactor static analysis handling in dbt_project.yml Updated the utility function to ensure `+static_analysis: <kind>` is set for all supported node types in dbt_project.yml, including models, seeds, snapshots, and more. This change enhances the onboarding process by maintaining existing values while ensuring the correct structure is present for each node type. Improved code readability and organization by introducing a constant for node types and refactoring the logic for inserting static analysis settings. * Enhance success message display in CompileBaselineStep Updated the CompileBaselineStep to conditionally display success messages and additional guidance when the project compiles successfully on Fusion. This change improves user experience by providing relevant information about optional next steps in the onboarding process. * Remove unused CompileBaselineFail * Refactor node index handling in ensure_static_analysis_for_node_type Updated the logic for determining the node index to simplify the handling of empty content. The function now conditionally adds a new line only when the content is not empty, improving code clarity and maintaining the intended structure for node keys. * Run cargo fmt * Fix clippy failures * Adds test file * Update proto * Add tada * Refactor onboarding screen enum values in proto and generated files Updated the OnboardingScreen enum to rename CompileStrict and CompileBaseline to Compile and CompileSaBaseline, respectively. This change improves clarity and consistency across the proto and generated Rust files. Additionally, updated corresponding serialization and deserialization logic to reflect these changes. * Refactor onboarding steps and telemetry for static analysis modes Renamed and reorganized onboarding steps to support new static analysis modes: baseline, strict, and off. Updated telemetry tracking functions to reflect these changes, ensuring accurate tracking of user actions during the onboarding process. Introduced new steps for handling compilation in different modes and updated screen navigation accordingly. This refactor enhances clarity and consistency in the onboarding flow. * Refactor dbt_project utilities to use dbt_yaml for parsing Updated the dbt_project.rs file to utilize the dbt_yaml library for parsing dbt_project.yml, enhancing project name extraction and structure validation. This change improves the reliability of reading the YAML file and preserves formatting and comments during writes. Additionally, refactored related functions to streamline the static analysis mode handling and added tests to ensure comment preservation in various scenarios. * Refactor onboarding steps to utilize shared next steps utility Replaced inline next steps content in CompileSaBaselineSuccessStep, CompileSaStrictFailStep, and SuccessStep with calls to new utility functions in next_steps.rs. This change centralizes the next steps content, improving maintainability and consistency across onboarding steps. Additionally, added next_steps module to encapsulate related functionality. * Refactor onboarding steps to utilize compile_utils for static analysis mode handling Replaced direct calls to dbt_project for setting static analysis modes in CompileSaBaselineSuccessStep, CompileSaOffFailStep, CompileSaStrictFailStep, and SuccessStep with a new compile_utils module. This change centralizes the logic for applying static analysis modes and writing next steps content, enhancing maintainability and consistency across onboarding steps. * Update success step message to clarify project config updates for strict mode Modified the success step message to indicate that project configs will be updated automatically to enable Fusion to run with strict mode, while also providing a link for manual updates. This change enhances user understanding of the onboarding process. * Better link * Update proto GitOrigin-RevId: b497d0a36e1befc55ff035e8fc43a2e2039b1168

[DX] Log --store-failures query to console if eligible (#8429) * [1/2]: pass node to tracing::formatters::format_test_failure, emit query. need store_test_failures meta. * [2/2]: expose store_failures in proto to tell consumer they can look up fqn in node processed; logger emits if store_failures indicated * style / only set this if we have something to pass along * changie * [update snapshot]: test_regression_tests_internal_analytics_build_with_recording * add dbt-cli replay test w/store-failures golden output GitOrigin-RevId: 87996f2644630895ae27fcf5cc2700bc418c88c2

Migrate compiled code event to tracing (#7962) * update agents telemetry instructions * migrate rendered progress to CompiledCode telemetry * refine compiled code compatibility and layer behavior * align compiled code json compat test with shared helper pattern * refine compiled code formatting and json compat command wiring * use known arrow fields for compiled code events * simplify json compat tests * goldies * changelog GitOrigin-RevId: c3f3367a326b6f584e6b9a72a9ee2fd87704e7de

Add tracing spans for on-start-run/on-end-run (#7782) * feat: Instrument operations on-run-start and on-run-end into tracing - Add tracing spans for on-run-start and on-run-end operations in task runner - Define proto schema for hook telemetry events - Add hook formatter for TUI and JSON compat layers - Update phase execution tracking with hook outcome status - Update golden test files with new telemetry output * use .instrument * use the unique_id as hook qualifier * use ExecutionPhase enum * change hook success to Completed * make HookProcessJsonPayload * use record_span_status-from_attrs * impl with_context * add ProgressId::Hook * add json_compat test * update golden files * reset golden files * update some goldens * update golden files * telemetry: align hook/phase proto docs and add hook index * telemetry: add HookProcessed::start_on_run helper * tracing: include hook index in json compat payloads * tracing: normalize hook formatting and phase progress context * telemetry: align hook span naming context and payload enums * tracing: split outcome metric keys for node and hook spans * chore: apply fmt fixes and update snowflake hook golden * restore spuriously changed proo descriptor * clean up after codex * fix name/package name confusion. re-hide start lines * glodie's 1 * remote goldie's part 1 * goldie's part 2 --------- Co-authored-by: Mike Perlov <5206955+mishamsk@users.noreply.github.com> GitOrigin-RevId: eb5e444d3ef72ec537d618b2184c6f939ed080f1

Split old & new protogen and migrate new into telemetry crates (#7768) * Move fusion protogen outputs to telemetry crates * Generate fusion protos in telemetry crates * Remove private wrapper now that telemetry owns protos * Move StaticName into dbt-telemetry * Reuse telemetry traits in private crate * Remove legacy proto test utils * Relocate fusion protos into telemetry crates * Remove proto-rust macro dependency * Remove telemetry v1 re-exports * Scope test attributes to telemetry protos * Document sync_protos steps * Limit telemetry protos to telemetry include paths * Add onboarding screen enum for telemetry * Adjust telemetry registry filter * Rename telemetry proto module * split old & new proto code generation * fix clippy and mark generated files with git attrs * Fix private telemetry proto search paths to include public protos Allow private telemetry protos to import public protos, matching the behavior of the existing proto-rust-private codegen configuration. * Remove unused CloudInvocation::new method from proto-rust-private The CloudInvocation struct is now constructed using struct literal syntax throughout the codebase, making this helper method unused. * changelog * Add #[inline] to blanket impl methods to fix debug build perf regression After moving protobuf-generated event structs from proto-rust to dbt-telemetry, generic code is now instantiated in the same crate as the blanket impl. This causes cross-crate function calls in debug builds (opt-level=0) where inlining is disabled. Adding #[inline] makes the function bodies available for call-site instantiation, avoiding the 3-7x performance overhead from repeated cross-crate calls on the telemetry hot path. * fix after rebase GitOrigin-RevId: 151f1ae0f933515711c19199e6a972035d31d984

Migrate parsing messages to tracing & introduce a dedicated event for it (#7694) * wip * update goldies * clippy * clippy 2 GitOrigin-RevId: e0e881bc7cd4fe2f3f5e00edf0889a7aee77ff59

Move markdown downgrade to tracing middleware (#7660) * Add structured log location fields * Move markdown downgrade to middleware * Expose optional line and column accessors * Add changelog entries * Apply check --fix formatting * Add docs md warning coverage * Downgrade docs md warnings in LSP * Update docs md warning parse test * Expand docs md warning fixtures * Restore docs md warning template files * spurious change * Apply check --fix formatting GitOrigin-RevId: f09f8094b67ed63c47af7d32d2338c51c2e95a82

Fix missing go package on generic event proto (#7612) * fix missing go package on generic event proto * update descriptors GitOrigin-RevId: 491cac34edc724e85061806795a5aade724bf2ef

State modified telemetry (#7588) * Structure the previous state content check to detect comparison failures for logging * Add debug level logging for state:modified differences * Update changelog * Fix clippy errors * Convert logging macro to a function * Create a state:modified telemetry event * Addressing reviewer comments - Converting node_type to node_type_or_category. - Inlining the emitting function for better line number tracking. GitOrigin-RevId: 87f587ec1e9f41660a561f5421ba7e197604f1dd

Instrument/migrate source freshness to tracing (#7567) * wip wip wip wip properly handle source_name * record freshness span errors on early exit handle early returns in freshness update freshness function name * clippy * update freshness progress rendering * do not use new formatting for sao freshness * changelog GitOrigin-RevId: e163e8d1425c164df1636b0b56810b2d9a6faf9c

Migrate remaining progress bar macros to tracing (#7275) * migrate pre_compile functions from progress macro * move "downloading relations" to debug level use "Downloading schemas" for progress spinner add severitynumber vs tracing level/filter ordering handle verbosity levels in tui_layer properly * use debug level for lineage NodeEvaluated to match other NodeEvaluated events * update goldie's * migrate loader with_progress to reacing remove with_progress macro * update goldie's GitOrigin-RevId: 290a159fc47815ffd641dfac957097a0fa783986

Migrate hidden commands to tracing (#7267) * make format sync, no value in async * remove outdated constants usage in progress bars * new progress events & instrument formatter fix * changelog * migrate linter to tracing * migrate sampler to tracing * migrate clean command to tracing and clean up constants * remove progress bar from lineage, too fast to see anyway GitOrigin-RevId: 1ff6adf20864ccf5ca2b2b817755a9b0b3821e46

Migrate show_result_with_title macro to tracing (#7229) * migrate show_result_with_title macro to tracing re-gen protogen * changelog GitOrigin-RevId: d18176b200bad0e6300f0a29c535bf8ea9f7ec1e

Improve source freshness and source_status messages (#7155) * Some source_status selector messages * Fix warning message for selection criteria when no depth is specified for '+' * changie * Issue sources.json write message * Update some tests * Update source_freshness test and loaded_at_query test GitOrigin-RevId: c6aeade0d9377b631ee4144fa45e2fb64162b05f

Migrate package install to tracing (#6948) * WIP: rebase changes make package name optional fix tui for package installs instrument other package types use singular in packages installed when only 1 package has been installed improve progress bar for deps improve code in install+packages * trim commit hash newline * add package add event & console output clippy clippy * changelog * goldie's local * remote goldie's remaining remote goldie's goldie's goldie's * fix add package parsing & ensure no passwords leak use platform independent test clippy GitOrigin-RevId: 4f657dda5e5ca1fbad6d9f52f75cc9e4168d0fc7

Add rows_affected to node evaluated & node processed (#7129) * add rows_affected to node evaluated & node processed * changelog GitOrigin-RevId: 3cc8be9a9992ee18f076a749cf72a93f297fe22a

Add run summary metrics to invocation event (#6962) * add run summary metrics to invocation span * changelog * clippy GitOrigin-RevId: a031f0610bf42410085a802b0027f104a3e3a4b0

Migrate debug command (#6964) * prioritize status reporter over tracing events in emission api's * migrate debug progress to tracing * changelog * restore progress spinner for debug GitOrigin-RevId: 5edf040363bad5b7825ded7a680e4dcaebb30f79

Add support for --parent-span-id (#6840) * add support for --parent-span-id * No breaking proto changes * plumb parent id through * fix parent span id invariant checks add tests * changelog * clippy --------- Co-authored-by: Mike Perlov <5206955+mishamsk@users.noreply.github.com> GitOrigin-RevId: 95db0d69bbf30a46597fddc3dedbd740d01fa7d4

Migrate node completed to tracing (#6780) * wip * span manager and stuff * more stuff * wip * add more fields to node events * remove unused humantime from dbt-common * re-create node formatters in tracing * tui rendering for node processed * add sao_flag and detailed sao data * ues span key instead of attributes in span manager * wip * wip * add node processed duration and use for tui formatting * add new data provider apis * add A001 compat event * remove log version and versio from irrelevat compat logs * json comapt tests * fix forgotten filters in evets add test group mior data provider stuff * split each compat message into separate method * improve test helpers for json compat * add MainReportArgs compat log * add ArtifactWritten compat log * split tui methods * node evaluated formatters * add test group handling ad formatting * fix span manager more progress split closures into functions remove excessive bounds trying to fix deadlock fix deadlock add spa manager docs * wip fix node formatting * improve formatting, fix tailing test skips * many fixes to formatting * fix phatom seed from hydration phase * remove some show_completed callsites & adjust compat log goldie tests * clippy, and extract shared ts formatters * add missing events & strucutre file log * add missing formatters to file & tui layers * add phase couters * unit test for compat node events & bug fixes * fix test pass outcome * add unique_id's to regular logs when available * fix handling of nodes with not all phases and formatting fix padding * add no-op to summary line * chagelog * fix packageupdate parquet serializatio * show_completed no more * do not count source in summary node counts ever * set up tracing in lsp tests, necessary for span manager * avoid calculating groupped flag on each call * fix test defined at location to be relative to in_dir * simplify span manager * simplify task spa screation further * fix spurious MarkSkippedChild add in_selection to NodeProcessed for filtering * remove now unnecessary Reused event * do not show progress for non-selected nodes * handle new timestamp normalisation in tests * clippy * fix test using outdated json logs * goldie's goldie GitOrigin-RevId: 0a6b95d248d57a536f9a3b61f6593476e9f46aa9

Migrate show command to tracing (#6740) * stop passing around immutable db refs * make run_adhoc & do_query side-effect free, almost; pull table printing to callsites * restore naming for repl based queries * migrate show node to tracing event * chagelog * remove `show_result!` macro by migrating remaining usage sites * clippy * fixes * restore support for zero row show results from remote GitOrigin-RevId: 6840705d1559b22f8edfd4590c2ae277231c7429

Migrate list command to tracing (#6696) * add show-dirty to claude.md * migrate listp output to structured event * update goldie's * use dedicated enum for cli and proto * small bugs * changelog GitOrigin-RevId: f97cca952f994544acc820f7f70262e5be72567a

Migrate jinja log and print to tracing (#6550) * migrate user logs to tracing add proper support of debug user logs add new user log otel event fix otel file verbosity level (use file log verbosity) * update otel goldie's * changelog GitOrigin-RevId: 27a2c4beb3f0cf52b35df8502c95bfd812642f2a

Enable agentic autofix telemetry (#6372) * Enable agentic autofix telemetry * Sync proto buffers * Missed some commented out code * Update proto bin files * Update proto GitOrigin-RevId: 9168b7c785690011cf8ecd2472a6e7e248ae6fdf

Migrate errors & warnings to tracing (#6145) * wip wip * allow passing event structs into tracing without into call * new emit event functions allow empty message remove emit event macros migrate to functions from macros for tracing api's * wip wip wip wip wip wip wip wip wip wip wip fix tests fix error/warn formatting clippy * unify read-only and mutable data provider allow safe usage of per-invocation storage by consumers make data layer immune to possible corruption of stored data by middlewares * re-enable test summaries on tui * add package_name field to log message event * move legacy status reporter code into functions from macro * remove all legacy warn and error macros nit * clippy * fix error code * add new middleware to layers * unify unit test fail table between local and remote * write unit test summary to dbt.log * semi fix for dbt.log interleaving * fix invocation metrics improve tests for metrics & use stricter ordering of atomics update telemetry test goldie * remove show_progress_exit add debug assertions for metrics move invocation total metrics update to middleware * fix double reporting of local unit test exec errors * fix uninstrumented tasks causing missing autofix counter * instrument xdbc * fix lp cli root span * remove redundnat onboadring telemetry function * more idiomatic rust please! * avoid unnecessary &mut refs to data provider in invocation formatter * clippy * goldie's GitOrigin-RevId: 1ad243ef09d6bd917b6023a1f289a2ce202d1942

Fix fusion missing output counts in summary and metrics infra upgrade (#5977) * remove old unit test only summary from task runner * serialize timestamps in compat logs in expected format * wip scc api upgrade data provider tests pre-allocate metric storage to avoid panics due to non-atomic checks remove non-atomic read + init from data provider remove start very wip capture errors update stuff on invocation span wip wip wip clippy refactor visitor to capture more skip detailes wip finish nit clippy * changelog * use legacy counters for errors/warnings/autofixes until we fully migrate to the new ones * fix terminal width * enable summary for compile * do not show 0 count success in summary * nits use u for micros clippy * fix success counter gate * Revert "remove old unit test only summary from task runner" This reverts commit edefb2c1f5c4cd28150b0cda9fd810d8e24ffb09. * fix after udf rebase, use processed in summary * goldie's * goldie's * goldies GitOrigin-RevId: ef5da15c47f164a175707b559f389bde4450186d

Add all changes from dbt-compare branch (#6192) This commit contains all the changes from the dbt-compare branch, properly applied using the git reset technique: - Created new branch from main - Reset hard to dbt-compare to get all changes - Reset soft to main to stage all differences as additions This creates a clean merge-ready branch with all dbt-compare functionality. GitOrigin-RevId: f0fca1cf2d8b65d38e41b9b40b89ddb7ea693fb6

Implement UDF nodes (#5761) * Implement DbtFunction resource type * update json_schema GitOrigin-RevId: 5c21f1baeee77001f1b8f94a361f5914229fec28

Add Query Id telemetry event and migrate query log to use tracing (#5950) * rename export flags to output_flags * new output flags * rename display_name, fix export flag rename after rebase * more rebase fixes add efficient newline writer methods and is_terminal to sharedwritier trate * pretty writer layer custom filtering wrapper for tracing fixes in telemetry tests after method name change * add query proto and more arrow tests * wip new query log * fixes * make query span debug level * actually write query log * allow missing nullable columns in telemetry arrow * changelog * improve adapter type getting. revert to storing as string in event proto GitOrigin-RevId: af94b8caa0fcecac2fb9082e273b7b7fc920d6cb

Resurrect json log tests (#5881) * sync protos * wip * it is alive! * fix clippy fix deps tests update goldies * remove anonymous stats flag from test args, move to project yml * forgot flags * debuggind ci * more ci debugging * random try * ok. let's get serious and catch panic! * use stable schema in tests * Revert "ok. let's get serious and catch panic!" This reverts commit 3c17d29e81a37f3e225b347c61f73a427a532bf5. * Revert "random try" This reverts commit 0049e31e0a8649ab8ebaad407b63d0c804ed55d6. * Revert "more ci debugging" This reverts commit 7973d14d2c3fd94fca612156e8c64a686dfbbcdf. * Revert "debuggind ci" This reverts commit 7b0adf43f2837388da208e75ca96766014275b78. GitOrigin-RevId: bce861a857e5254040107f08b5e882ffe209de2a

[TELEMETRY] Add "user_id" to onboarding telemetry (#5641) * Add user_id to onboarding telemetry/vortex events * Sync proto * Use correct constant GitOrigin-RevId: 02265dea42f28bfa3c9ca530167894c9e5669b15

Telemetry refactor to support private events & Proto defined event data structs (#5715) * remove unusued pbjson serde gen. add skip_serialize_none pull protogen.toml path out of implementation add glob support to protogen better messages move type attributes to protogen config remove deserialize derive from vortex events add message & enum attrs support to protogen * add new tracing config creation api not relying on io_args telemetry context avoid rare flaky failures in tracing tests update metadata crates fix json payload inconsistency, simplify phase/node span_name enusre stable sort in protogen structure tracing tests and add jsonl filtering tests fix metadata fix copybara structure module for manual impl for generated proto files add otlp testing infra hide arrow schema from public api's make a private api to deserialize all event types from arrow including private * switch to macro derive for serde stuff in proto add new method derive for protos wip NodeEvaluated wip update goldie wip * proper skipped reporting * fix evaluation detail serialization and update metadata sample * add COPY fs/sa/crates/proto-rust-macros fs/sa/crates/proto-rust-macros for metadata GitOrigin-RevId: 439a1d347344453817dd736ceaa0346931a7384f

Generate code for mantle_query.proto (#5288) * Sync the protos with the latest version * Generate code for mantle_query.proto * proto-rust-private: Wire the modules to make mantle_query.rs buildable * Remove .public. files from the private prot crate * protogen: Remove all the .public. files from the private crate so we don't accidentally rely on them GitOrigin-RevId: 7d2278512005bd2a0ba298a82092ab064ba5feb4

xtask: Generate code for selected .proto files from proto-rust-private (#5252) * proto-rust-private: Remove files that don't belong in private/ * Add scripts/sync_protos.sh * proto-rust: Latest changes to {vortex_options,fusion,vortex}.proto * proto-rust-private: Sync all the files * proto-rust: Add all the new protos * Correct the protoc-generated paths in .gitattributes * Add protogen.toml file to limit the amount of code generated by * proto-rust: Remove v1.events.fusion.rs (it's v1.public.events.fusion.rs now) * proto-rust-private: Add a lib.rs and include it in Cargo.toml * proto-rust: Delete gen/mod.rs that doesn't get generated anymore * dbt-common: Define the dyn-trait DescreteEventEmitter * dbt-cli: Inject the DiscreteEventEmitter from main() * xtask: protogen the private .proto files as well * xtask: Implement the protogen.toml handling for real * REMOVE all the code that we don't need because of protogen.toml GitOrigin-RevId: b9c9a85f740db526c74630ec971f8aeb1de584ab

[TELEMETRY] Add telemetry events to track the CLI onboarding flow (#5186) * Add telemetry support to onboarding process - Introduced invocation ID to OnboardingConfig for tracking events. - Implemented telemetry functions to track onboarding steps and user actions. - Updated onboarding context to include project ID, user ID, and invocation ID. - Enhanced onboarding flow with telemetry tracking for various steps including welcome, parse, compile, and success. - Removed deprecated onboarding function for cleaner API. * Refactor telemetry imports and enhance onboarding step tracking - Reorganized telemetry imports for consistency across files. - Improved readability of telemetry tracking function calls in onboarding steps. - Ensured all necessary telemetry components are included in relevant modules. * Enhance cloud invocation event structure with default values - Added default values to the cloud invocation event message structure for improved consistency and reliability. - Ensured that the enrichment field remains optional while maintaining the integrity of the message format. * Include cargo.toml changes that I didn't commit earlier * Small fixes * Styling and remove change that shouldn't be there * Sync binders and proto update * Remove unused values GitOrigin-RevId: da94cbb8d06bc19896dfab792be80ca2f1aa4dd8

Add resource_type to RunModel event GitOrigin-RevId: 610fc34f83b6d1f391406c4290032383deb3bca6

Contracted log output(Proto) and test for IDE (#3940) GitOrigin-RevId: 058db0fa09f7f9dbda65339277e50a3eb9240635

Project import generated by Copybara. GitOrigin-RevId: ad7e3349e907abb85fe1d035dc9503e7f2c06ff3

Project import generated by Copybara. GitOrigin-RevId: 6ddefbbaef81945641d51d072a92873c0b099253

2026-05-28

Project import generated by Copybara. GitOrigin-RevId: aab6d7ae0f773afb550d2405a81497a91bcae75a

2025-05-20

move protos into external dependency (#11608)

This commit does not contain any .proto files.

Fix #11506: Add freshness config to sources (#11628) * Fix #11506: Add freshness config to exposures * Add freshness to some config * Update Fixes-20250515-145857.yaml * Add PropertyMovedToConfigDeprecation * make core_proto_types * Add log check * Use +freshness instead of freshness * Fix test_events.py

2025-05-16

Refinements and test

First draft of SourceOverrideDeprecation warning.

2025-05-14

deprecation warning for --models, --model, -m

deprecate --output/-o usage in dbt source freshness (#11621)

deprecate --output/-o usage in dbt source freshness

2025-05-12

move protos into external dependency

This commit does not contain any .proto files.

2025-04-25

Initial slate of deprecations for 1.10 (#11544) * Begin basic jsonschema validations of dbt_project.yml (#11505) * Add jsonschema for validation project file * Add utility for helping to load jsonschema resources Currently things are a bit hard coded. We should probably alter this to be a bit more flexible. * Begin validating the the `dbt_project.yml` via jsonschema * Begin emitting deprecation warnings for generic jsonschema violations in dbt_project.yml * Move from `DbtInternalError` to `DbtRuntimeError` to avoid circular imports * Add tests for basic jsonschema validation of `dbt_project.yml` * Add changie doc * Add seralization test for new deprecation events * Alter the project jsonschema to not require things that are optional * Add datafiles to package egg * Update inclusion of project jsonschema in setup.py to get files correctly Using the glob spec returns a list of found files. Our previous spec was raising the error `error: can't copy 'dbt/resources/input_schemas/project/*.json': doesn't exist or not a regular file` * Try another approach of adding jsonschema to egg * Add input_schemas dir to MANIFEST.in spec * Drop jsonschema inclusion spec from setup.py * Begin using importlib.resources.files for loading project jsonschema This doesn't currently work with editable installs :'( * Use relative paths for loading jsonchemas instead of importlib Using "importlib" is the blessed way to do this sort of thing. However, that is failing for us on editable installs. This commit switches us to using relative paths. Technically doing this has edge cases, however this is also what we do for the `start_project` used in `dbt init`. So we're going to do the same, for now. We should revisit this soon. * Drop requirment of `__additional_properties__` specified by project jsonschema * Drop requirement for `pre-hook` and `post-hook` specified by project jsonschema * Reset `active_deprecations` global at the end of tests using `project` fixture * Begin validation the jsonschema of YAML resource files (#11516) * Add jsonschema for resources * Begin jsonschema validating YAML resource files in dbt projects * Drop `tests` and `data_tests` as required properties of `Columns` and `Models` for resources jsonschema * Drop `__additional_properties__` as required for `_Metrics` in resources jsonschema * Drop `post_hook` and `pre_hook` requirement for `__SnapshotsConfig` in resources jsonschema * Update `_error_path_to_string` to handle empty paths * Create + use custom Draft7Validator to ignore datetime and date classes * Break `TestRetry` functional test class into multiple test classes There was some overflow global state from one test to another which was causing some of the tests to break. * Refactor duplicate instances of `jsonschema_validate` to single definition * Begin testing jsonschema validation of resource YAMLs * Add changie doc * Add Deprecation Warnings for Unexpected Jinja Blocks (#11514) * Add deprecation warnings on unexpected jinja blocks. * Add changelog entry. * Add test event. * Regen proto types. * Fix event test. * Add `UnexpectedJinjaBlockDeprecationSummary` and add file context to `UnexpectedJinjaBlockDeprecation` (#11517) * Add summary event for UnexpectedJinjaBlockDeprecation * Begin including file information in UnexpectedJinjaBlockDeprecation event * Add UnexpectedJinjaBlockDeprecationSummary to test_events.py * Deprecate Custom Top-Level Keys (#11518) * Add specific deprecation for custom top level keys. * Add changelog entry * Add test events * Add Check for Duplicate YAML Keys (#11510) * Add functionality to check for duplicate yaml keys, working around PyYAML limitation. * Fix up some ancient typing issues. * Ignore typing issue, for now. * Correct unit tests of `checked_load` * Add event and deprecation types for duplicate yaml keys * Begin validating `dbt_project.yml` for duplicate key violations * Begin checking for duplicate key violations in schema files * Add test to check duplicate keys are checked in schema files * Refactor checked_yaml failure handling to reduce duplicate code * Move `checked_load` utilities to separate file to avoid circular imports * Handle yaml `start_mark` correctly for top level key errors * Update changelog * Fix test. --------- Co-authored-by: Quigley Malcolm <quigley.malcolm@dbtlabs.com> * Fix issue with YAML anchors in new CheckedLoader class. * Deprecate having custom keys in config blocks (#11522) * Add deprecation event for custom keys found in configs * Begin checking schema files for custom keys found in configs * Test new CustomConfigInConfigDeprecation event * Add changie doc * Add custom config key deprecation events to event serialization test * Provide message to ValidationError in `SelectorConfig.from_path` This typing error is unrelated to the changes in this PR. However, it was failing CI, so I figured it'd be simple to just fix it. * Add some extra guards around the ValidationFailure `path` and `instance` * [TIDY-FRIST] Use new `deprecation_tag` (#11524) * Tidy First: Update deprecation events to use the new `deprecation_tag` Note did this for a majority of deprecations, but not _all_ deprecations. That is because not all deprecations were following the pattern. As some people do string parsing of our logs with regex, altering the deprecations that weren't doing what `deprecation_tag` does to use `deprecation_tag` would be a _breaking change_ for those events, thus we did not alter those events * Bump minimum dbt-common to `1.22.0` * Fix tests * Begin emitting deprecation events for custom properties found in objects (#11526) * Fix CustomKeyInConfigDeprecationSummary * Add deprecation type for custom properties in YAML objects * Begin emitting deprecation events for custom properties found in objects * Add changie doc * Add `loaded_at_query` property to `_Sources` definition in jsonschema This was breaking the test tests/unit/parser/test_parser.py::SchemaParserSourceTest::test_parse_source_custom_freshness_at_source * Move validating jsonschema of schema files earlier in the process Previously we were validating the jsonschema of schema files in `SchemaParser.parse_file`. However, the file is originally loaded in `yaml_from_file` (which happens before `SchemaParser.parse_file`), and `yaml_from_file` _modifies_ the loaded dictionary to add some additional properties. These additional properties violate the jsonschema unfortunately, and thus we needed to start validating the schema against the jsonschema before any such modifications. * Skip parser tests for `model.freshness` Model freshness never got fully implemented, won't be implemented nor documented for 1.10. As such we're gonna consider the `model.freshness` property an "unknown additional property". This is actually good as some people have "accidentally" defined "freshness" on their models (likely due to copy/paste of a source), and that property isn't doing anything. * One single DeprecationsSummary event to rule them all (#11540) * Begin emitting singular deprecations summary, instead of summary per deprecation type * Remove concept of deprecation specific summary events in deprecations module * Drop deprecation summary events that have been added to `feature-branch--11335-deprecations` but not `main` These are save to drop with no notice because they only ever existed on a feature branch, never main. * Correct code numbers for new events on feature-branch that haven't made it to main yet * Kill `PackageRedirectDeprecationSummary` event, and retire its event code * add changie doc * Update jsonschemas to versions 0.0.110 (#11541) * Update jsonschems to 0.0.110 * Don't allow additional properties in configs * Don't allow additional top level properties on objects * Allow for 'loaded_at_query' on Sources and Tables * Don't allow additional top level properties in schema files --------- Co-authored-by: Peter Webb <peter.webb@dbtlabs.com>

Begin emitting deprecation events for custom properties found in objects (#11526) * Fix CustomKeyInConfigDeprecationSummary * Add deprecation type for custom properties in YAML objects * Begin emitting deprecation events for custom properties found in objects * Add changie doc * Add `loaded_at_query` property to `_Sources` definition in jsonschema This was breaking the test tests/unit/parser/test_parser.py::SchemaParserSourceTest::test_parse_source_custom_freshness_at_source * Move validating jsonschema of schema files earlier in the process Previously we were validating the jsonschema of schema files in `SchemaParser.parse_file`. However, the file is originally loaded in `yaml_from_file` (which happens before `SchemaParser.parse_file`), and `yaml_from_file` _modifies_ the loaded dictionary to add some additional properties. These additional properties violate the jsonschema unfortunately, and thus we needed to start validating the schema against the jsonschema before any such modifications. * Skip parser tests for `model.freshness` Model freshness never got fully implemented, won't be implemented nor documented for 1.10. As such we're gonna consider the `model.freshness` property an "unknown additional property". This is actually good as some people have "accidentally" defined "freshness" on their models (likely due to copy/paste of a source), and that property isn't doing anything.

Deprecate having custom keys in config blocks (#11522) * Add deprecation event for custom keys found in configs * Begin checking schema files for custom keys found in configs * Test new CustomConfigInConfigDeprecation event * Add changie doc * Add custom config key deprecation events to event serialization test * Provide message to ValidationError in `SelectorConfig.from_path` This typing error is unrelated to the changes in this PR. However, it was failing CI, so I figured it'd be simple to just fix it. * Add some extra guards around the ValidationFailure `path` and `instance`

One single DeprecationsSummary event to rule them all (#11540) * Begin emitting singular deprecations summary, instead of summary per deprecation type * Remove concept of deprecation specific summary events in deprecations module * Drop deprecation summary events that have been added to `feature-branch--11335-deprecations` but not `main` These are save to drop with no notice because they only ever existed on a feature branch, never main. * Correct code numbers for new events on feature-branch that haven't made it to main yet * Kill `PackageRedirectDeprecationSummary` event, and retire its event code * add changie doc

2025-04-24

Kill `PackageRedirectDeprecationSummary` event, and retire its event code

Correct code numbers for new events on feature-branch that haven't made it to main yet

Drop deprecation summary events that have been added to `feature-branch--11335-deprecations` but not `main` These are save to drop with no notice because they only ever existed on a feature branch, never main.

Begin emitting singular deprecations summary, instead of summary per deprecation type

2025-04-22

Begin emitting deprecation events for custom properties found in objects (#11526) * Fix CustomKeyInConfigDeprecationSummary * Add deprecation type for custom properties in YAML objects * Begin emitting deprecation events for custom properties found in objects * Add changie doc * Add `loaded_at_query` property to `_Sources` definition in jsonschema This was breaking the test tests/unit/parser/test_parser.py::SchemaParserSourceTest::test_parse_source_custom_freshness_at_source * Move validating jsonschema of schema files earlier in the process Previously we were validating the jsonschema of schema files in `SchemaParser.parse_file`. However, the file is originally loaded in `yaml_from_file` (which happens before `SchemaParser.parse_file`), and `yaml_from_file` _modifies_ the loaded dictionary to add some additional properties. These additional properties violate the jsonschema unfortunately, and thus we needed to start validating the schema against the jsonschema before any such modifications. * Skip parser tests for `model.freshness` Model freshness never got fully implemented, won't be implemented nor documented for 1.10. As such we're gonna consider the `model.freshness` property an "unknown additional property". This is actually good as some people have "accidentally" defined "freshness" on their models (likely due to copy/paste of a source), and that property isn't doing anything.

2025-04-21

Add deprecation type for custom properties in YAML objects

2025-04-18

Add deprecation type for custom properties in YAML objects

2025-04-17

Deprecate having custom keys in config blocks (#11522) * Add deprecation event for custom keys found in configs * Begin checking schema files for custom keys found in configs * Test new CustomConfigInConfigDeprecation event * Add changie doc * Add custom config key deprecation events to event serialization test * Provide message to ValidationError in `SelectorConfig.from_path` This typing error is unrelated to the changes in this PR. However, it was failing CI, so I figured it'd be simple to just fix it. * Add some extra guards around the ValidationFailure `path` and `instance`

Add Check for Duplicate YAML Keys (#11510) * Add functionality to check for duplicate yaml keys, working around PyYAML limitation. * Fix up some ancient typing issues. * Ignore typing issue, for now. * Correct unit tests of `checked_load` * Add event and deprecation types for duplicate yaml keys * Begin validating `dbt_project.yml` for duplicate key violations * Begin checking for duplicate key violations in schema files * Add test to check duplicate keys are checked in schema files * Refactor checked_yaml failure handling to reduce duplicate code * Move `checked_load` utilities to separate file to avoid circular imports * Handle yaml `start_mark` correctly for top level key errors * Update changelog * Fix test. --------- Co-authored-by: Quigley Malcolm <quigley.malcolm@dbtlabs.com>

Deprecate Custom Top-Level Keys (#11518) * Add specific deprecation for custom top level keys. * Add changelog entry * Add test events

Add `UnexpectedJinjaBlockDeprecationSummary` and add file context to `UnexpectedJinjaBlockDeprecation` (#11517) * Add summary event for UnexpectedJinjaBlockDeprecation * Begin including file information in UnexpectedJinjaBlockDeprecation event * Add UnexpectedJinjaBlockDeprecationSummary to test_events.py

2025-04-16

Add Deprecation Warnings for Unexpected Jinja Blocks (#11514) * Add deprecation warnings on unexpected jinja blocks. * Add changelog entry. * Add test event. * Regen proto types. * Fix event test.

Begin basic jsonschema validations of dbt_project.yml (#11505) * Add jsonschema for validation project file * Add utility for helping to load jsonschema resources Currently things are a bit hard coded. We should probably alter this to be a bit more flexible. * Begin validating the the `dbt_project.yml` via jsonschema * Begin emitting deprecation warnings for generic jsonschema violations in dbt_project.yml * Move from `DbtInternalError` to `DbtRuntimeError` to avoid circular imports * Add tests for basic jsonschema validation of `dbt_project.yml` * Add changie doc * Add seralization test for new deprecation events * Alter the project jsonschema to not require things that are optional * Add datafiles to package egg * Update inclusion of project jsonschema in setup.py to get files correctly Using the glob spec returns a list of found files. Our previous spec was raising the error `error: can't copy 'dbt/resources/input_schemas/project/*.json': doesn't exist or not a regular file` * Try another approach of adding jsonschema to egg * Add input_schemas dir to MANIFEST.in spec * Drop jsonschema inclusion spec from setup.py * Begin using importlib.resources.files for loading project jsonschema This doesn't currently work with editable installs :'( * Use relative paths for loading jsonchemas instead of importlib Using "importlib" is the blessed way to do this sort of thing. However, that is failing for us on editable installs. This commit switches us to using relative paths. Technically doing this has edge cases, however this is also what we do for the `start_project` used in `dbt init`. So we're going to do the same, for now. We should revisit this soon. * Drop requirment of `__additional_properties__` specified by project jsonschema * Drop requirement for `pre-hook` and `post-hook` specified by project jsonschema * Reset `active_deprecations` global at the end of tests using `project` fixture

2025-04-05

General Deprecation Warning Improvements (#11466)

2025-04-01

Move resource names with spaces deprecation to using new deprecations manager

2025-03-25

Support artifact upload (#11419) * wip * reorganize * changie * retry * nits * nits * improve retry, adjust error, adjust host name * adjust logic * pr_feedback * Update .changes/unreleased/Features-20250323-151625.yaml Co-authored-by: Emily Rockman <emily.rockman@dbtlabs.com> --------- Co-authored-by: Emily Rockman <emily.rockman@dbtlabs.com>

2025-03-20

Add node_checksum to node_info on structured logs (#11368) * update node_info to include node checksum * changelog entry * Discard changes to dev-requirements.txt --------- Co-authored-by: Chenyu Li <chenyulee777@gmail.com> Co-authored-by: Quigley Malcolm <quigley.malcolm@dbtlabs.com>

2025-03-18

Macro Annotations and Inference (#11389) * Default macro argument information from original definitions. * Add argument type and count warnings behind behavior flag. * Add changelog entry. * Make flag test more robust. * Use a unique event for macro annotation warnings, per review. * Add event to test list. * Regenerate core_types_pb2 using protoc 5.28.3 --------- Co-authored-by: Quigley Malcolm <quigley.malcolm@dbtlabs.com>

2025-01-21

Misc fixes for group info in logging (#11218)

2025-01-08

Create `LogNodeResult` event (#11195) * Create LogNodeResult event * add msg directly during object creation

2024-12-16

Warn if `concurrent_batches` config is set to `True`, but the available adapter doesn't support it (#11145) (#11154) * Begin producing warning when attempting to force concurrent batches without adapter support Batches of microbatch models can be executed sequentially or concurrently. We try to figure out which to do intelligently. As part of that, we implemented an override, the model config `concurrent_batches`, to allow the user to bypass _some_ of our automatic detection. However, a user _cannot_ for batches to run concurrently if the adapter doesn't support concurrent batches (declaring support is opt in). Thus, if an adapter _doesn't_ support running batches concurrently, and a user tries to force concurrent execution via `concurrent_batches`, then we need to warn the user that that isn't happening. * Add custom event type for warning about invalid `concurrent_batches` config * Fire `InvalidConcurrentBatchesConfig` warning via `warn_or_error` so it can be silenced (cherry picked from commit 6c61cb7f7adbdce8edec35a887d6c766a401e403) Co-authored-by: Quigley Malcolm <QMalcolm@users.noreply.github.com>

Warn if `concurrent_batches` config is set to `True`, but the available adapter doesn't support it (#11145) * Begin producing warning when attempting to force concurrent batches without adapter support Batches of microbatch models can be executed sequentially or concurrently. We try to figure out which to do intelligently. As part of that, we implemented an override, the model config `concurrent_batches`, to allow the user to bypass _some_ of our automatic detection. However, a user _cannot_ for batches to run concurrently if the adapter doesn't support concurrent batches (declaring support is opt in). Thus, if an adapter _doesn't_ support running batches concurrently, and a user tries to force concurrent execution via `concurrent_batches`, then we need to warn the user that that isn't happening. * Add custom event type for warning about invalid `concurrent_batches` config * Fire `InvalidConcurrentBatchesConfig` warning via `warn_or_error` so it can be silenced

2024-12-09

Microbatch first last batch serial (#11072) (#11107) * microbatch: split out first and last batch to run in serial * only run pre_hook on first batch, post_hook on last batch * refactor: internalize parallel to RunTask._submit_batch * Add optional `force_sequential` to `_submit_batch` to allow for skipping parallelism check * Force last batch to run sequentially * Force first batch to run sequentially * Remove batch_idx check in `should_run_in_parallel` `should_run_in_parallel` shouldn't, and no longer needs to, take into consideration where in batch exists in a larger context. The first and last batch for a microbatch model are now forced to run sequentially by `handle_microbatch_model` * Begin skipping batches if first batch fails * Write custom `on_skip` for `MicrobatchModelRunner` to better handle when batches are skipped This was necessary specifically because the default on skip set the `X of Y` part of the skipped log using the `node_index` and the `num_nodes`. If there was 2 nodes and we are on the 4th batch of the second node, we'd get a message like `SKIPPED 4 of 2...` which didn't make much sense. We're likely in a future commit going to add a custom event for logging the start, result, and skipping of batches for better readability of the logs. * Add microbatch pre-hook, post-hook, and sequential first/last batch tests * Fix/Add tests around first batch failure vs latter batch failure * Correct MicrobatchModelRunner.on_skip to handle skipping the entire node Previously `MicrobatchModelRunner.on_skip` only handled when a _batch_ of the model was being skipped. However, that method is also used when the entire microbatch model is being skipped due to an upstream node error. Because we previously _weren't_ handling this second case, it'd cause an unhandled runtime exception. Thus, we now need to check whether we're running a batch or not, and there is no batch, then use the super's on_skip method. * Correct conditional logic for setting pre- and post-hooks for batches Previously we were doing an if+elif for setting pre- and post-hooks for batches, where in the `if` matched if the batch wasn't the first batch, and the `elif` matched if the batch wasn't the last batch. The issue with this is that if the `if` was hit, the `elif` _wouldn't_ be hit. This caused the first batch to appropriately not run the `post-hook` but then every hook after would run the `post-hook`. * Add two new event types `LogStartBatch` and `LogBatchResult` * Update MicrobatchModelRunner to use new batch specific log events * Fix event testing * Update microbatch integration tests to catch batch specific event types --------- Co-authored-by: Quigley Malcolm <quigley.malcolm@dbtlabs.com> (cherry picked from commit 03fdb4c1578d74e25615a4f46fb572ecee0685e0) Co-authored-by: Michelle Ark <MichelleArk@users.noreply.github.com>

2024-12-07

Microbatch first last batch serial (#11072) * microbatch: split out first and last batch to run in serial * only run pre_hook on first batch, post_hook on last batch * refactor: internalize parallel to RunTask._submit_batch * Add optional `force_sequential` to `_submit_batch` to allow for skipping parallelism check * Force last batch to run sequentially * Force first batch to run sequentially * Remove batch_idx check in `should_run_in_parallel` `should_run_in_parallel` shouldn't, and no longer needs to, take into consideration where in batch exists in a larger context. The first and last batch for a microbatch model are now forced to run sequentially by `handle_microbatch_model` * Begin skipping batches if first batch fails * Write custom `on_skip` for `MicrobatchModelRunner` to better handle when batches are skipped This was necessary specifically because the default on skip set the `X of Y` part of the skipped log using the `node_index` and the `num_nodes`. If there was 2 nodes and we are on the 4th batch of the second node, we'd get a message like `SKIPPED 4 of 2...` which didn't make much sense. We're likely in a future commit going to add a custom event for logging the start, result, and skipping of batches for better readability of the logs. * Add microbatch pre-hook, post-hook, and sequential first/last batch tests * Fix/Add tests around first batch failure vs latter batch failure * Correct MicrobatchModelRunner.on_skip to handle skipping the entire node Previously `MicrobatchModelRunner.on_skip` only handled when a _batch_ of the model was being skipped. However, that method is also used when the entire microbatch model is being skipped due to an upstream node error. Because we previously _weren't_ handling this second case, it'd cause an unhandled runtime exception. Thus, we now need to check whether we're running a batch or not, and there is no batch, then use the super's on_skip method. * Correct conditional logic for setting pre- and post-hooks for batches Previously we were doing an if+elif for setting pre- and post-hooks for batches, where in the `if` matched if the batch wasn't the first batch, and the `elif` matched if the batch wasn't the last batch. The issue with this is that if the `if` was hit, the `elif` _wouldn't_ be hit. This caused the first batch to appropriately not run the `post-hook` but then every hook after would run the `post-hook`. * Add two new event types `LogStartBatch` and `LogBatchResult` * Update MicrobatchModelRunner to use new batch specific log events * Fix event testing * Update microbatch integration tests to catch batch specific event types --------- Co-authored-by: Quigley Malcolm <quigley.malcolm@dbtlabs.com>

2024-11-21

Microbatch parallelism (#10958)

2024-11-19

Parseable JSON and text output in quiet mode for `dbt show` and `dbt compile` (#9958) * Allow `dbt show` and `dbt compile` to output JSON without extra logs * Add `quiet` attribute for ShowNode and CompiledNode messages * Output of protoc compiler * Utilize the `quiet` attribute for ShowNode and CompiledNode * Reuse the `dbt list` approach when the `--quiet` flag is used * Use PrintEvent to get to stdout even if the logger is set to ERROR * Functional tests for quiet compile * Functional tests for quiet show * Fire event same way regardless if LOG_FORMAT is json or not * Switch back to firing ShowNode and CompiledNode events * Make `--inline-direct` to be quiet-compatible * Temporarily change to dev branch for dbt-common * Remove extraneous newline * Functional test for `--quiet` for `--inline-direct` flag * Update changelog entry * Update `core_types_pb2.py` * Restore the original branch in `dev-requirements.txt` --------- Co-authored-by: Kshitij Aranke <kshitij.aranke@dbtlabs.com>

2024-11-11

Replace environment variable with a project flag to gate microbatch functionality (#10799) * first pass: replace os env with project flag * Fix `TestMicrobatchMultipleRetries` to not use `os.env` * Turn off microbatch project flag for `TestMicrobatchCustomUserStrategyDefault` as it was prior to a9df50f * Update `BaseMicrobatchTest` to turn on microbatch via project flags * Add changie doc * Fix functional tests after merging in main * Add function to that determines whether the new microbatch functionality should be used The new microbatch functionality is, unfortunately, potentially dangerous. That is it adds a new materalization strategy `microbatch` which an end user could have defined as a custom strategy previously. Additionally we added config keys to nodes, and as `config` is just a Dict[str, Any], it could contain anything, thus meaning people could already be using the configs we're adding for different purposes. Thus we need some intellegent gating. Specifically something that adheres to the following: cms = Custom Microbatch Strategy abms = Adapter Builtin Microbatch Strategy bf = Behavior flag umb = Use Microbatch Batching t/f/e = True/False/Error | cms | abms | bf | umb | | t | t | t | t | | f | t | t | t | | t | f | t | t | | f | f | t | e | | t | t | f | f | | f | t | f | t | | t | f | f | f | | f | f | f | e | (The above table assumes that there is a microbatch model present in the project) In order to achieve this we need to check that either the microbatch behavior flag is set to true OR microbatch materializaion being used is the _root_ microbatch materialization (i.e. not custom). The function we added in this commit, `use_microbatch_batches`, does just that. * Gate microbatch functionality by `use_microbatch_batches` manifest function * Rename microbatch behavior flag to `require_batched_execution_for_custom_microbatch_strategy` * Extract logic of `find_macro_by_name` to `find_macro_candiate_by_name` In 0349968c615444de05360509ddeaf6d75d41d826 I had done this for the function `find_materialization_macro_by_name`, but that wasn't the right function to do it to, and will be reverted shortly. `find_materialization_macro_by_name` is used for finding the general materialization macro, whereas `find_macro_by_name` is more general. For the work we're doing, we need to find the microbatch macro, which is not a materialization macro. * Use `find_macro_candidate_by_name` to find the microbatch macro * Fix microbatch macro locality check to search for `core` locality instead of `root` Previously were were checking for a locality of `root`. However, a locality of `root` means it was provided by a `package`. We wnt to check for locality of `core` which basically means `builtin via dbt-core/adapters`. There is another locality `imported` which I beleive means it comes from another package. * Move the evaluation of `use_microbatch_batches` to the last position in boolean checks The method `use_microbatch_batches` is always invoked to evaluate an `if` statement. In most instances, it is part of a logic chain (i.e. there are multiple things being evaluated in the `if` statement). In `if` statements where there are multiple things being evaulated, `use_microbatch_batches` should come _last_ (or as late as possible). This is because it is likely the most costly thing to evaluate in the logic chain, and thus any shortcuts cuts via other evaluations in the if statement failing (and thus avoiding invoking `use_microbatch_batches) is desirable. * Drop behavior flag setting for BaseMicrobatchTest tests * Rename 'env_var' to 'project_flag' in test_microbatch.py * Update microbatch tests to assert when we are/aren't running with batches * Update `test_resolve_event_time_filter` to use `use_microbatch_batches` * Fire deprecation warning for custom microbatch macros * Add microbatch deprecation events to test_events.py --------- Co-authored-by: Quigley Malcolm <quigley.malcolm@dbtlabs.com>

2024-11-05

Behavior change cumulative type param (#10909) * Behavior change for mf timespinse without yaml config * Flipping behavior flag causes parse error * Added more tests * Appending just one error

2024-10-31

Behavior change for mf timespine without yaml configuration (#10857)