Proto commits in dagster-io/dagster

These 64 commits are when the Protocol Buffers files have changed:

2026-05-20

Move dagster-cloud package to dagster-oss/python_modules (#20797) ## Summary Relocates the `dagster-cloud` user-facing agent package from `dagster-cloud/python_modules/dagster-cloud/` to `dagster-oss/python_modules/dagster-cloud/`, alongside the other core Dagster packages (`dagster`, `dagster-graphql`, `dagster-cloud-cli`, etc.). The package itself is unchanged — only its location. ## What changed - **Physical move** of the package tree via `git mv`, preserving history. - **Workspace config** (root `pyproject.toml`): updated `[tool.uv.sources]`, pyright/ty `extraPaths` and `executionEnvironments`, ruff `per-file-ignores`, and removed it from `[tool.uv.workspace].members`. - **Buildkite pipeline** (`.buildkite/dagster_internal_buildkite/`): updated `KUBERNETES_TOX_MODULES`, `get_tox_env_suffixes`, `get_tox_splits`, `get_tox_steps`, and dogfood path detection to point at the new location. - **Release automation** (`automation_internal/release/dagster_module_publisher.py`): `dagster-cloud` now created with `is_cloud=False, internal_oss_repo_path` and removed from `expected_cloud_modules_subdirectories`. - **Shadow CLI** (`automation_internal/cloud/shadow/cli.py`): `dagster_cloud_dir` now resolves under `dagster-oss/python_modules/`. - **Docker image build defs** (`automation_internal/image/defs/{agent,user_code,serverless,examples,release,dagster_cloud_pex}.py`): `copy_directories` entries point at the new path. - **Service Dockerfiles + lockfiles** (`services/{dagster-cloud-agent-pre-license,dagster-cloud-examples,dagster-cloud-serverless-{agent,base,k8s-agent},dagster-cloud-user-code,release-pipeline}/`): updated each service's `pyproject.toml`/`uv.lock` to reference the new editable path; dropped now-unused `COPY build_cache/dagster-cloud/python_modules` lines in Dockerfiles whose image defs no longer stage anything at that path (kept it for serverless-agent and serverless-k8s-agent, which still stage sibling packages there). - **Root `uv.lock`**: editable path updated in all 12 locations. - **Shell scripts** (6 files), integration test + module `tox.ini` files (~14 files), and the `python_modules/{toys,insights-consumption-test}` + `dagster-dg-cli` Dockerfile templates: relative/absolute path references updated. ## Test plan - [ ] CI green on this branch (this PR exists primarily to validate that — the previous build's 311 cascading `broken` jobs were all downstream of the Cloud Docker image builds failing on stale `build_cache/dagster-cloud/python_modules` references) - [ ] `just ruff`, `just pyright`, `just ty` clean for the moved package - [ ] `tox -e py312-default` from `dagster-oss/python_modules/dagster-cloud/` - [ ] A dependent package (e.g. `dagster-cloud-backend`) still resolves its dagster-cloud dep - [ ] `just host-cloud` smoke test Closes #20663 ## Implementation Plan <details> <summary><strong>Implementation Plan</strong> (Plan #20797)</summary> This change relocates the `dagster-cloud` user-facing agent package from `dagster-cloud/python_modules/dagster-cloud/` to `dagster-oss/python_modules/dagster-cloud/`, aligning it with other core Dagster packages like `dagster`, `dagster-graphql`, and `dagster-cloud-cli`. This consolidation improves the repository structure by colocating all user-facing Python packages under `dagster-oss/`. ## Files Changed <details> <summary>Files Changed</summary> ### Moved (1 directory tree) - `dagster-cloud/python_modules/dagster-cloud/` → `dagster-oss/python_modules/dagster-cloud/` - User-facing agent package with all source, tests, and configuration ### Modified (50+ files) - `pyproject.toml` - Updated workspace members, uv sources, pyright paths, and ruff config - `.buildkite/dagster_internal_buildkite/pipeline.py` - Updated tox module path and test configurations - `.buildkite/dagster_internal_buildkite/steps/dogfood/__init__.py` - Updated file change detection paths - `python_modules/automation-internal/automation_internal/release/dagster_module_publisher.py` - Updated module creation and validation logic - `python_modules/automation-internal/automation_internal/cloud/shadow/cli.py` - Updated dagster_cloud_dir resolution - `python_modules/automation-internal/automation_internal/image/defs/*.py` - Updated package paths in all Docker image builders - Integration test `tox.ini` files (5 files) - Updated relative dependency paths - Service shell scripts (4 files) - Updated pip installation paths - Module `tox.ini` files (9 files) - Updated relative dependency paths - Dockerfiles (3 files) - Updated COPY and pip install commands </details> ## Key Changes - Physical move of package directory tree with git history preservation - Updated uv workspace configuration to reference new path via editable path dependency - Modified all CI/CD paths in Buildkite pipeline configuration and dogfood test detection - Updated release automation to resolve package from new OSS location with `is_cloud=False` - Fixed all relative and absolute path references across 50+ configuration files and shell scripts - Removed dagster-cloud from workspace members list while keeping path-based dependency resolution - Updated ruff linter configuration with new path for test file ignores </details> ## Implementation Plan <details> <summary><strong>Implementation Plan</strong> (Issue #20663)</summary> # Plan: Move `dagster-cloud` package to `dagster-oss/python_modules` ## Context The `dagster-cloud` Python package (the user-facing agent package published to PyPI) currently lives at `dagster-cloud/python_modules/dagster-cloud/` within the internal repo. The goal is to move it to `dagster-oss/python_modules/dagster-cloud/` so it sits alongside other core packages like `dagster`, `dagster-graphql`, etc. The package path is hardcoded in 50+ locations across the repo: workspace config, type checker paths, Buildkite CI, Docker image builds, release automation, shell scripts, tox.ini files, and Dockerfiles. --- ## Phase 1: Physical Move **Action:** Move (git mv) `dagster-cloud/python_modules/dagster-cloud/` to `dagster-oss/python_modules/dagster-cloud/`. This includes `dagster_cloud/`, `dagster_cloud_tests/`, `pyproject.toml`, `tox.ini`, etc. --- ## Phase 2: Root `pyproject.toml` **File:** `pyproject.toml` | Section | Change | |---------|--------| | `[tool.uv.workspace].members` (L426) | Remove `"dagster-cloud/python_modules/dagster-cloud"` | | `[tool.uv.sources]` (L369) | `dagster-cloud = { workspace = true }` -> `dagster-cloud = { path = "dagster-oss/python_modules/dagster-cloud", editable = true }` | | `[tool.pyright].extraPaths` (L120) | `"dagster-cloud/python_modules/dagster-cloud"` -> `"dagster-oss/python_modules/dagster-cloud"` | | `[tool.pyright].executionEnvironments` (L199) | `root = "dagster-cloud/python_modules/dagster-cloud"` -> `root = "dagster-oss/python_modules/dagster-cloud"` | | `[tool.ty.environment].extra-paths` (L212) | `"dagster-cloud/python_modules/dagster-cloud"` -> `"dagster-oss/python_modules/dagster-cloud"` | | `[tool.ruff.lint.per-file-ignores]` (L343) | `"dagster-cloud/python_modules/dagster-cloud/..."` -> `"dagster-oss/python_modules/dagster-cloud/..."` | --- ## Phase 3: Release Pipeline ### `python_modules/automation-internal/automation_internal/release/dagster_module_publisher.py` 1. **L102-104** -- Change dagster-cloud module creation: ```python # Before DagsterModule.create_dagster_module("dagster-cloud", internal_repo_path, is_cloud=True, is_internal=True) # After DagsterModule.create_dagster_module("dagster-cloud", internal_oss_repo_path, is_cloud=False, is_internal=True) ``` With `is_cloud=False` and `git_root=internal_oss_repo_path`, `_get_module_path` resolves to `{internal_repo}/dagster-oss/python_modules/dagster-cloud` -- the correct new location. 2. **L213-223** -- Remove `"dagster-cloud"` from `expected_cloud_modules_subdirectories`. 3. **`check_cloud_module_publishing()` (L273-291)** -- No change needed; `dagster-cloud` is still in `self.cloud_modules` so the check still finds it. 4. **`get_cloud_module_directories()` (L506-518)** -- No change needed; scanning `dagster-cloud/python_modules/` just won't find dagster-cloud there anymore, which is correct since it was removed from the expected list. ### `dagster_module.py` -- No changes needed `_get_module_path` handles the new case automatically via the existing `is_library=False` fallback. --- ## Phase 4: Buildkite CI ### `.buildkite/dagster_internal_buildkite/pipeline.py` | Line | Change | |------|--------| | L78 (KUBERNETES_TOX_MODULES) | `"dagster-cloud/python_modules/dagster-cloud"` -> `"dagster-oss/python_modules/dagster-cloud"` | | L85 (get_tox_env_suffixes) | `== "dagster-cloud/python_modules/dagster-cloud"` -> `== "dagster-oss/python_modules/dagster-cloud"` | | L94 (get_tox_splits) | `== "dagster-cloud/python_modules/dagster-cloud"` -> `== "dagster-oss/python_modules/dagster-cloud"` | | L310-329 (get_tox_steps) | The `glob.glob("dagster-cloud/python_modules/*")` loop (L321) will no longer find `dagster-cloud`. Add an explicit entry for `"dagster-oss/python_modules/dagster-cloud"` to the tox_build_steps, using the same pattern (env_suffixes, splits, timeout, queue selection). | ### `.buildkite/dagster_internal_buildkite/steps/dogfood/__init__.py` Update `files_changed(...)` paths: - `"dagster-cloud/python_modules/dagster-cloud/dagster_cloud/workspace/ecs/"` -> `"dagster-oss/python_modules/dagster-cloud/dagster_cloud/workspace/ecs/"` - `"dagster-cloud/python_modules/dagster-cloud/dagster_cloud_tests/workspace_tests/ecs_tests/"` -> `"dagster-oss/python_modules/dagster-cloud/dagster_cloud_tests/workspace_tests/ecs_tests/"` --- ## Phase 5: Docker Image Definitions All in `python_modules/automation-internal/automation_internal/image/defs/`: | File | Change | |------|--------| | `agent.py` L35 | `"dagster-cloud/python_modules/dagster-cloud"` -> `"dagster-oss/python_modules/dagster-cloud"` | | `agent.py` L163 | Same change (serverless agent context) | | `serverless.py` | Same change | | `release.py` | Same change | | `examples.py` | Same change | | `dagster_cloud_pex.py` | Same change | Note: References to `dagster-cloud/python_modules/dagster-cloud-serverless-agent` (agent.py L164) stay unchanged -- that's a different package. --- ## Phase 6: Shadow CLI **File:** `python_modules/automation-internal/automation_internal/cloud/shadow/cli.py` The `dagster_cloud_dir` construction (~L365) currently resolves via `cloud_modules_dir = os.path.join(internal_dir, "dagster-cloud/python_modules")` then `os.path.join(cloud_modules_dir, "dagster-cloud/dagster_cloud")`. Change the dagster_cloud_dir line to: ```python os.path.join(internal_dir, "dagster-oss/python_modules/dagster-cloud/dagster_cloud") ``` The `cloud_modules_dir` variable is still used for `dagster-cloud-dagit` and `dagster-cloud-backend`, so it stays. --- ## Phase 7: Shell Scripts | File | Change (old path -> new path) | |------|------| | `services/dagster-cloud-pex/local_build.sh` | `dagster-cloud/python_modules/dagster-cloud` -> `dagster-oss/python_modules/dagster-cloud` | | `scripts/buildkite/deploy_user_cloud_ecs.sh` | Same | | `scripts/buildkite/release/launch_release_pipeline.sh` | Same | | `scripts/buildkite/release/sync_release_pipeline.sh` | Same | | `dagster-cloud/sandbox/user_cloud/from_source/docker_pex/build.sh` | Same | | `dagster-cloud/sandbox/host_cloud_docker/build.sh` | Same | --- ## Phase 8: Dockerfiles | File | Change | |------|--------| | `python_modules/toys/Dockerfile` | `-e './internal/dagster-cloud/python_modules/dagster-cloud[insights]'` -> `-e './internal/dagster-oss/python_modules/dagster-cloud[insights]'` | | `python_modules/insights-consumption-test/Dockerfile` | Same | | `dagster-oss/.../dagster-dg-cli/.../deploy_uv_editable_Dockerfile.jinja` | `COPY --from=internal dagster-cloud/python_modules/dagster-cloud ...` -> `COPY --from=internal dagster-oss/python_modules/dagster-cloud ...` | The host cloud Dockerfile (`services/dagster-cloud-host/Dockerfile`) already copies `dagster-oss/` (and the new location is within it), so the package will be included. The bulk `COPY dagster-cloud/ dagster-cloud/` line just won't include the dagster-cloud package anymore. Verify the uv lockfile install step resolves correctly after lockfile regen. --- ## Phase 9: tox.ini Files (Relative Path Updates) All `../../dagster-cloud/python_modules/dagster-cloud` -> `../../dagster-oss/python_modules/dagster-cloud` (and similar for deeper paths): **Internal python_modules (depth 2):** - `python_modules/product-operations/tox.ini` - `python_modules/web-performance-telemetry/tox.ini` - `python_modules/toys/tox.ini` - `python_modules/dagster-yaml-demo/tox.ini` - `python_modules/purina-sales/tox.ini` - `python_modules/dogfood/tox.ini` - `python_modules/dagster-cloud-examples/tox.ini` **Infrastructure (depth 4: `../../../../`):** - `infra/k8s/agent/agent-schema/tox.ini` - `infra/k8s/multitenant_host/host-schema/tox.ini` **Integration tests (depth 4: `../../../../`):** - `dagster-cloud/integration_tests/test_suites/local_dagster_cloud/tox.ini` - `dagster-cloud/integration_tests/test_suites/prod_canary_tests/tox.ini` - `dagster-cloud/integration_tests/test_suites/kubernetes_dagster_cloud/tox.ini` - `dagster-cloud/integration_tests/test_suites/dogfood_serverless_tests/tox.ini` - `dagster-cloud/integration_tests/test_suites/multitenant_canary_tests/tox.ini` **Dev requirements:** - `python_modules/purina/dev_requirements.txt` --- ## Phase 10: Cloud Package tox.ini Files Other packages under `dagster-cloud/python_modules/` that depend on `dagster-cloud` (backend, dagit, worker, ursula, insights, etc.) likely have tox.ini entries referencing `../dagster-cloud`. After the move, this relative path changes -- the package is no longer a sibling. These need to become absolute or differently-relative paths. For a package at `dagster-cloud/python_modules/dagster-cloud-backend/tox.ini`, the old relative path `../dagster-cloud` becomes `../../../dagster-oss/python_modules/dagster-cloud`. Check all tox.ini files under `dagster-cloud/python_modules/*/tox.ini` for references. --- ## Phase 11: Lock File and Sync ```bash just rebuild_uv_lock_file # Regenerate uv.lock just sync # Sync environment ``` --- ## Phase 12: Verification 1. `just ruff` -- formatting/lint passes 2. `just pyright` -- type checking resolves the new path 3. Run dagster-cloud tox suite from new location 4. Run a dependent package's tox suite (e.g., dagster-cloud-backend) to verify cross-package deps resolve 5. Verify `just host-cloud` still starts (smoke test) --- ## Open Questions 1. **OSS install script** (`dagster-oss/scripts/install_dev_python_modules.py`): Should `dagster-cloud` be added to this list? This script is for OSS-only development. Since dagster-cloud won't be in the public mirror, it likely should NOT be added. 2. **dagster-cloud's own tox.ini**: After moving, its relative paths to other dagster-cloud packages (backend, dagit) and to dagster-oss libraries change. The tox.ini at `dagster-cloud/python_modules/dagster-cloud/tox.ini` (which will move to `dagster-oss/python_modules/dagster-cloud/tox.ini`) needs inspection -- relative paths like `../../dagster-cloud-backend` would no longer work from the new location. </details> --- Closes #20663 To checkout this PR in a fresh worktree and environment locally, run: ``` source "$(erk pr checkout 20797 --script)" && erk pr sync --dangerous ``` </details> --- To replicate this PR locally, run: ``` erk pr teleport 20797 ``` Internal-RevId: 1265bc25a0f1bcf82ec8b27a07070e5d27fb5626

2025-09-03

partial reloading grpc

2025-08-26

partial reloading grpc

2025-05-01

grpc stubs for new methods > Insert changelog entry or delete this section. starting to flesh out contents method > Insert changelog entry or delete this section.

2025-04-30

grpc stubs for new methods > Insert changelog entry or delete this section. starting to flesh out contents method > Insert changelog entry or delete this section.

grpc stubs for new methods > Insert changelog entry or delete this section. starting to flesh out contents method > Insert changelog entry or delete this section.

2025-04-29

grpc stubs for new methods > Insert changelog entry or delete this section. starting to flesh out contents method > Insert changelog entry or delete this section.

2025-04-28

grpc stubs for new methods > Insert changelog entry or delete this section. starting to flesh out contents method > Insert changelog entry or delete this section.

grpc stubs for new methods > Insert changelog entry or delete this section. starting to flesh out contents method > Insert changelog entry or delete this section.

2025-03-20

checkpoint: instant reloads w/o a process reload > Insert changelog entry or delete this section. progress on hot reloading > Insert changelog entry or delete this section. yikes > Insert changelog entry or delete this section.

checkpoint: instant reloads w/o a process reload > Insert changelog entry or delete this section. progress on hot reloading > Insert changelog entry or delete this section. yikes > Insert changelog entry or delete this section.

2025-03-05

checkpoint: instant reloads w/o a process reload > Insert changelog entry or delete this section. progress on hot reloading > Insert changelog entry or delete this section. yikes > Insert changelog entry or delete this section.

2025-03-04

progress on hot reloading > Insert changelog entry or delete this section.

2025-02-03

new grpc proto file name for dagster (#27498) ## Summary & Motivation This is called 'api.proto' which is a very generic name - if another grpc project is imported with that name, they can't coexist. Rename ours to 'dagster_api.proto' instead (this is not a breaking change from grpc's perspective). ## How I Tested These Changes BK, sanity check that the change isn't breaking by launchign a grpc server in master and connecting to it with dagster dev with this change applied, and vice versa ## Changelog Fixed an issue where dagster could not be imported alongside some other libraries using gRPC with an 'api.proto' file.

The documentation is generated from this commit.

2024-06-14

grpc

2024-05-15

vaguely working

2023-12-29

Cleanup grpc server metrics (#18945) Cleans up grpc server metrics - Renames *_workers to be in terms of requests - Renames counts to be in terms of requests - serialized_server_health_metadata -> serialized_server_utilization_metrics (more in line with other PRs) - decorates Ping grpc api function

2023-12-28

Unify naming under "utilization metrics"

[RFC] Add metrics retrieval to grpc server (#18721) Introduces a potential API for metrics retrieval from the GRPC server. - We retrieve metrics about the currently running requests of each type for a grpc server. This is implemented through a `retrieve_metrics` decorator. The decorator manages lifecycle and stores metadata about requests in a global threadsafe-access dictionary. - Methods decorated with `retrieve_metrics` are added to a registrar, which allows us to enforce that all new methods have the decorator. - The Ping API has been repurposed to snapshot these metrics. - Adds a test demonstrating metrics retrieval of a running sensor. - Adds a test which could be used to enforce that all GRPC methods are instrumented with this decorator. I'll leave the actual enforcement to a future PR. Using a decorator here keeps the implementation contained / doesn't muddy up the actual methods, while still allowing us to intercept arguments / retrieve more interesting information than just "a request of this type is running". Viewing slices of grpc server utilization over time like this can be incredibly powerful. It could power a view, for example, of the number of running sensors over the total number of workers, and which sensors are actually being run in each of those slices. There's definitely some potential spiciness with the global dictionary + lock shenanigans, but I think it stays pretty constrained. We're going to have to have some sort of global data structure to consolidate information regardless, I think. **Changes from round 1 of comments** - Record max_workers from the threadpool - Make metrics retrieval opt-in - Only record count per request type

2023-11-30

Implement a sync version of the schedule and sensor grpc call (#18261) ## Summary & Motivation We've never needed this to be an async call and we seem to be hitting upstream grpc issues with streaming API calls where they sporadically return early without raising an error. Make a sync version of the call and use that. For back-compat, fall back to the async version if an UnimplementedError is returned. This adds a marginal bit of latency in the case where you update your webserver/daemon/dagster cloud agent before you upgrade your code, but shouldn't increase the throughput on the boxes since the first call returns more or less immediately with an UnimplementedError without doing any actual work. If that's a concern we could try to add some caching of the fact that a particular server doesn't have the method implemented, but my sense is that that is more trouble than it is worth. ## How I Tested These Changes BK, test back-compat case manually by running a sensor locally on a new version of dagster-webserver but an old version of `dagster api grpc'

2023-06-01

Add a `dagster code-server start` CLI entrypoint with the ability to reload code without restarting the process (#14275) ## Summary & Motivation A new entry point that adds a layer of indirection between the entrypoint process, and a subprocess that actually runs user code. The main benefit this gives is the ability to reload code without restarting the process (or re-deploying the k8s pod / task / etc.) - resolving https://github.com/dagster-io/dagster/issues/12581. The main entry point implements the same gRPC api, but then redirects requests to a subprocess that it spins up. The parent process uses a GrpcServerRegistry under the hood, which handles things like cleaning up subprocesses after they die and are no longer running any executions. Unfortunately since gRPC servers still don't always have access to the instance, had to make the instance argument to GrpcServerRegistry optional (and add a few more config toggles since they get passed through). Also added a new graphql test suite which should exercise the actual underlying gRPC api pretty comprehensively, and added a test that verifies that reloading works the way you would expect. This is a large PR, I could potentially split it into two - one that adds the entrypoint, the other that implements the reload logic in dagster-graphql.

2023-05-23

Add a `dagster code-server start` CLI entrypoint with the ability to reload code without restarting the process (#14275) ## Summary & Motivation A new entry point that adds a layer of indirection between the entrypoint process, and a subprocess that actually runs user code. The main benefit this gives is the ability to reload code without restarting the process (or re-deploying the k8s pod / task / etc.) - resolving https://github.com/dagster-io/dagster/issues/12581. The main entry point implements the same gRPC api, but then redirects requests to a subprocess that it spins up. The parent process uses a GrpcServerRegistry under the hood, which handles things like cleaning up subprocesses after they die and are no longer running any executions. Unfortunately since gRPC servers still don't always have access to the instance, had to make the instance argument to GrpcServerRegistry optional (and add a few more config toggles since they get passed through). Also added a new graphql test suite which should exercise the actual underlying gRPC api pretty comprehensively, and added a test that verifies that reloading works the way you would expect. This is a large PR, I could potentially split it into two - one that adds the entrypoint, the other that implements the reload logic in dagster-graphql.

2023-05-19

Add a grpc proxy server

2023-05-09

Add a grpc proxy server

2023-04-04

[resource verification][2b/n] Implement GRPC call for ConfigVerifiable checks using tick abstraction

2023-04-03

Move to generic user-code-execution GRPC endpoint

[resource verification][2/n] Generate verification jobs, GRPC call for ConfigVerifiable checks

2022-11-29

Get current runs on grpc server (#10525) ### Summary & Motivation Expose the self._executions map over grpc ### How I Tested These Changes unit tests

2022-09-27

[managed stacks][wip] Prototype managed stack API

2022-09-15

[grpc] deferred snapshot support (#9630)

2022-09-14

[grpc] deferred snapshot support (#9630)

2022-09-06

[prototype] defer snapshots [INTERNAL_BRANCH=al/08-31-_prototype_defer_snapshots]

2022-07-23

Mark dagster.{daemon,generate,grpc,loggers} private (#8884)

2022-03-14

grpc > _grpc

2021-12-20

grpc call for dynamic partitions

2021-08-23

create grpc call to send back remote notebook content (#4515) - create grpc call to send back remote notebook content - include_notebook_route defaults to false - disable the notebook route by default for security

2021-03-29

change sensor/schedule/backfill grpc calls to use streaming query to avoid limit on message size Summary: also, seems like there's an opportunity to consolidate some of the streaming boilerplate, but leaving that to a separate diff Test Plan: bk Reviewers: dgibson, sashank Reviewed By: dgibson Differential Revision: https://dagster.phacility.com/D7127

2020-11-23

[sensors-2] sensor def Summary: Sets up a sensor definition, inheriting from the abstract job type. Currently all jobs are defined by: pipeline_name solid_selection mode At evaluation time, sensors return a should_execute boolean, a run_config dict, and a tags dict. An alternative implementation has a sensor defined by a single job params function, which returns a list of JobParams (containing a run config dict and a tags dict). Will send out in a separate diff for comparison. Test Plan: bk Reviewers: schrockn, dgibson, sandyryza Reviewed By: dgibson Subscribers: sandyryza Differential Revision: https://dagster.phacility.com/D5019

[sensors-1] redefine schedules as jobs Summary: Rips out the old job definitions, builds schedules on top of jobs. Test Plan: bk Reviewers: dgibson, schrockn, alangenfeld, johann Reviewed By: dgibson Differential Revision: https://dagster.phacility.com/D5011

2020-11-21

Revert "[sensors-1] redefine schedules as jobs" This reverts commit 7e4c53bd2498024cd39fd1a02d5179e2f318c291.

[sensors-1] redefine schedules as jobs Summary: Rips out the old job definitions, builds schedules on top of jobs. Test Plan: bk Reviewers: dgibson, schrockn, alangenfeld, johann Reviewed By: dgibson Differential Revision: https://dagster.phacility.com/D5011

2020-11-13

(dagit-reload-1/n) Add GetServerId gRPC service method Summary: This diff adds a `GetServerId` that exposes a unique UUID associated with each server. This is used to help track gRPC servers updating. Test Plan: unit Reviewers: alangenfeld, dgibson, prha, max Reviewed By: dgibson Differential Revision: https://dagster.phacility.com/D5105

grpc ExecuteRun api call fixes Summary: - Missing test coverage on the 'launch after cleaning up server case' - wait for the process to clean up before leaving the ExecuteRun API call Test Plan: BK + Azure Reviewers: alangenfeld, max, prha, johann Reviewed By: alangenfeld Differential Revision: https://dagster.phacility.com/D5119

2020-10-22

RFC: rename executable => job Test Plan: bk Reviewers: schrockn, alangenfeld, dgibson Reviewed By: schrockn, dgibson Differential Revision: https://dagster.phacility.com/D4837

2020-10-15

Increase gRPC message size limit for ExternalRepository API call Summary: Apparently prezi has a large enough pipeline that they are running into this. Bump the default for this specific API call since that's the one that can have large respnoses (eventually we could make it configurable in the instance too) Test Plan: Load dagit on normal pipeslines, no noticable slowdown despite increasing the message limit BK for large pipeline Reviewers: sashank, max, alangenfeld Reviewed By: sashank Differential Revision: https://dagster.phacility.com/D4750

2020-09-18

TriggeredExecutionDefinition => ExecutableDefinition Summary: Mostly just a rename... also removed `should_execute`, will rely on sensors/schedules for instigation policy Depends on D4458. Test Plan: bk Reviewers: alangenfeld, schrockn, catherinewu Reviewed By: alangenfeld Differential Revision: https://dagster.phacility.com/D4467

2020-08-27

[trigger-2] host representation for triggers Summary: Add host representation, out of process execution of trigger params TODO: Combine some of the boilerplate for schedules, triggers, and partition params Test Plan: bk Reviewers: alangenfeld, sashank, schrockn Reviewed By: sashank Subscribers: catherinewu Differential Revision: https://dagster.phacility.com/D4284

2020-08-18

[partition-backfill-1] backfill data IPC/GRPC implementations Summary: Batch up the calls to fetch partition tags / config to minimize RPC calls to the user process when doing backfills Test Plan: bk Reviewers: dgibson, alangenfeld Reviewed By: alangenfeld Differential Revision: https://dagster.phacility.com/D4169

2020-08-11

nicer error on failure to load repository in external process Summary: Addresses https://github.com/dagster-io/dagster/issues/2772 Example error produced from `dagster pipeline execute`: ``` Traceback (most recent call last): File "/Users/sryza/.pyenv/versions/dagster-3.6.8/bin/dagster", line 11, in <module> load_entry_point('dagster', 'console_scripts', 'dagster')() File "/Users/sryza/dagster/python_modules/dagster/dagster/cli/__init__.py", line 38, in main cli(obj={}) # pylint:disable=E1123 File "/Users/sryza/.pyenv/versions/3.6.8/envs/dagster-3.6.8/lib/python3.6/site-packages/click/core.py", line 764, in __call__ return self.main(*args, **kwargs) File "/Users/sryza/.pyenv/versions/3.6.8/envs/dagster-3.6.8/lib/python3.6/site-packages/click/core.py", line 717, in main rv = self.invoke(ctx) File "/Users/sryza/.pyenv/versions/3.6.8/envs/dagster-3.6.8/lib/python3.6/site-packages/click/core.py", line 1137, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/Users/sryza/.pyenv/versions/3.6.8/envs/dagster-3.6.8/lib/python3.6/site-packages/click/core.py", line 1137, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/Users/sryza/.pyenv/versions/3.6.8/envs/dagster-3.6.8/lib/python3.6/site-packages/click/core.py", line 956, in invoke return ctx.invoke(self.callback, **ctx.params) File "/Users/sryza/.pyenv/versions/3.6.8/envs/dagster-3.6.8/lib/python3.6/site-packages/click/core.py", line 555, in invoke return callback(*args, **kwargs) File "/Users/sryza/dagster/python_modules/dagster/dagster/cli/pipeline.py", line 262, in pipeline_execute_command return _logged_pipeline_execute_command(config, preset, mode, DagsterInstance.get(), kwargs) File "/Users/sryza/dagster/python_modules/dagster/dagster/core/telemetry.py", line 89, in wrap result = f(*args, **kwargs) File "/Users/sryza/dagster/python_modules/dagster/dagster/cli/pipeline.py", line 290, in _logged_pipeline_execute_command result = execute_execute_command(env, kwargs, mode, tags) File "/Users/sryza/dagster/python_modules/dagster/dagster/cli/pipeline.py", line 297, in execute_execute_command external_pipeline = get_external_pipeline_from_kwargs(cli_args, instance) File "/Users/sryza/dagster/python_modules/dagster/dagster/cli/workspace/cli_target.py", line 411, in get_external_pipeline_from_kwargs external_repo = get_external_repository_from_kwargs(kwargs, instance) File "/Users/sryza/dagster/python_modules/dagster/dagster/cli/workspace/cli_target.py", line 374, in get_external_repository_from_kwargs repo_location = get_repository_location_from_kwargs(kwargs, instance) File "/Users/sryza/dagster/python_modules/dagster/dagster/cli/workspace/cli_target.py", line 342, in get_repository_location_from_kwargs workspace = get_workspace_from_kwargs(kwargs, instance) File "/Users/sryza/dagster/python_modules/dagster/dagster/cli/workspace/cli_target.py", line 198, in get_workspace_from_kwargs return workspace_from_load_target(created_workspace_load_target(kwargs), instance) File "/Users/sryza/dagster/python_modules/dagster/dagster/cli/workspace/cli_target.py", line 168, in workspace_from_load_target user_process_api=python_user_process_api, File "/Users/sryza/dagster/python_modules/dagster/dagster/cli/workspace/load.py", line 253, in location_handle_from_python_file attribute=attribute, File "/Users/sryza/dagster/python_modules/dagster/dagster/api/list_repositories.py", line 25, in sync_list_repositories raise DagsterSubprocessError(result.to_string(), subprocess_error_infos=[result]) dagster.core.errors.DagsterSubprocessError: (ImportError) - ImportError: attempted relative import with no known parent package Stack Trace: File "/Users/sryza/dagster/python_modules/dagster/dagster/cli/api.py", line 143, in list_repositories_command python_file, module_name, working_directory, attribute File "/Users/sryza/dagster/python_modules/dagster/dagster/grpc/utils.py", line 20, in get_loadable_targets else loadable_targets_from_python_file(python_file, working_directory) File "/Users/sryza/dagster/python_modules/dagster/dagster/cli/workspace/autodiscovery.py", line 11, in loadable_targets_from_python_file loaded_module = load_python_file(python_file, working_directory) File "/Users/sryza/dagster/python_modules/dagster/dagster/core/code_pointer.py", line 88, in load_python_file return import_module_from_path(module_name, python_file) File "/Users/sryza/dagster/python_modules/dagster/dagster/seven/__init__.py", line 110, in import_module_from_path spec.loader.exec_module(module) File "<frozen importlib._bootstrap_external>", line 678, in exec_module File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed File "examples/legacy_examples/dagster_examples/simple_pyspark/pipelines.py", line 10, in <module> from .solids import ( ``` We could do some user_code_error_boundary stuff to make this even nicer, but I think this gives us parity with what we had a couple releases ago. Test Plan: added tests, ran "dagster pipeline execute" in both grpc and cli mode Reviewers: alangenfeld, dgibson, max Reviewed By: max Differential Revision: https://dagster.phacility.com/D4087

2020-08-06

Heartbeat for GRPC server Summary: This lets managed servers shut themselves down when the client processes disappear Test Plan: Unit Reviewers: dgibson Reviewed By: dgibson Differential Revision: https://dagster.phacility.com/D4099

2020-08-05

k8s user code deployments Test Plan: bk / manual. Reviewers: alangenfeld, nate Reviewed By: nate Subscribers: nate Differential Revision: https://dagster.phacility.com/D4041

2020-08-04

explain what devs must do after editing api.proto Test Plan: none Reviewers: prha, max Reviewed By: prha Differential Revision: https://dagster.phacility.com/D4095

2020-07-29

Persistent GRPC run launcher Summary: Introduces StartRun Test Plan: unit Reviewers: alangenfeld, dgibson, johann, prha, sashank Reviewed By: alangenfeld, dgibson Differential Revision: https://dagster.phacility.com/D4003

2020-07-25

Support soft run termination over GRPC Test Plan: Unit Reviewers: sashank, prha, alangenfeld, dgibson, johann Reviewed By: alangenfeld Differential Revision: https://dagster.phacility.com/D3954

2020-07-21

Support soft server termination over GRPC Test Plan: Unit Reviewers: sashank, prha, alangenfeld, johann, dgibson Reviewed By: dgibson Differential Revision: https://dagster.phacility.com/D3946

2020-07-20

Update GRPC server to store target Test Plan: unit Reviewers: dgibson, alangenfeld Reviewed By: dgibson Subscribers: alangenfeld Differential Revision: https://dagster.phacility.com/D3941

2020-07-08

Streaming API call over GRPC Summary: This moves execute_run to use GRPC. A follow-on diff will implement the (fairly complex) soft termination scheme. Test Plan: Unit Reviewers: alangenfeld, schrockn Reviewed By: alangenfeld, schrockn Differential Revision: https://dagster.phacility.com/D3662

2020-07-06

Snapshot schedule execution over GRPC Test Plan: Unit Reviewers: alangenfeld, schrockn Reviewed By: alangenfeld Differential Revision: https://dagster.phacility.com/D3720

External repositories over GRPC Test Plan: Unit Reviewers: alangenfeld, schrockn Reviewed By: alangenfeld Differential Revision: https://dagster.phacility.com/D3719

External pipeline snapshot over GRPC Test Plan: Unit Reviewers: alangenfeld, schrockn Reviewed By: alangenfeld Differential Revision: https://dagster.phacility.com/D3718

Implement snapshot partition over GRPC Test Plan: Unit Reviewers: alangenfeld, schrockn Reviewed By: alangenfeld Differential Revision: https://dagster.phacility.com/D3711

Implement list_repositories over GRPC Summary: This also switches the ephemeral client to use a module-based scheme for starting the server, so that we can stub in different python interpreters more easily. Test Plan: Unit Reviewers: alangenfeld, schrockn Reviewed By: alangenfeld Differential Revision: https://dagster.phacility.com/D3679

Unary API call over GRPC Summary: This demonstrates moving a unary API call onto the new GRPC system. Test Plan: Unit Reviewers: alangenfeld, schrockn Reviewed By: alangenfeld Differential Revision: https://dagster.phacility.com/D3656

2020-06-26

Scaffolding for GRPC Summary: This introduces a CLI command, server and client classes and helpers, and generation machinery from protobuf. Test Plan: Unit Reviewers: alangenfeld, schrockn Reviewed By: alangenfeld Differential Revision: https://dagster.phacility.com/D3610