Proto commits in dagster-io/dagster

These 61 commits are when the Protocol Buffers files have changed:

2025-05-01

grpc stubs for new methods > Insert changelog entry or delete this section. starting to flesh out contents method > Insert changelog entry or delete this section.

2025-04-30

grpc stubs for new methods > Insert changelog entry or delete this section. starting to flesh out contents method > Insert changelog entry or delete this section.

grpc stubs for new methods > Insert changelog entry or delete this section. starting to flesh out contents method > Insert changelog entry or delete this section.

2025-04-29

grpc stubs for new methods > Insert changelog entry or delete this section. starting to flesh out contents method > Insert changelog entry or delete this section.

2025-04-28

grpc stubs for new methods > Insert changelog entry or delete this section. starting to flesh out contents method > Insert changelog entry or delete this section.

grpc stubs for new methods > Insert changelog entry or delete this section. starting to flesh out contents method > Insert changelog entry or delete this section.

2025-03-20

checkpoint: instant reloads w/o a process reload > Insert changelog entry or delete this section. progress on hot reloading > Insert changelog entry or delete this section. yikes > Insert changelog entry or delete this section.

checkpoint: instant reloads w/o a process reload > Insert changelog entry or delete this section. progress on hot reloading > Insert changelog entry or delete this section. yikes > Insert changelog entry or delete this section.

2025-03-05

checkpoint: instant reloads w/o a process reload > Insert changelog entry or delete this section. progress on hot reloading > Insert changelog entry or delete this section. yikes > Insert changelog entry or delete this section.

2025-03-04

progress on hot reloading > Insert changelog entry or delete this section.

2025-02-03

new grpc proto file name for dagster (#27498) ## Summary & Motivation This is called 'api.proto' which is a very generic name - if another grpc project is imported with that name, they can't coexist. Rename ours to 'dagster_api.proto' instead (this is not a breaking change from grpc's perspective). ## How I Tested These Changes BK, sanity check that the change isn't breaking by launchign a grpc server in master and connecting to it with dagster dev with this change applied, and vice versa ## Changelog Fixed an issue where dagster could not be imported alongside some other libraries using gRPC with an 'api.proto' file.

The documentation is generated from this commit.

2024-06-14

grpc

2024-05-15

vaguely working

2023-12-29

Cleanup grpc server metrics (#18945) Cleans up grpc server metrics - Renames *_workers to be in terms of requests - Renames counts to be in terms of requests - serialized_server_health_metadata -> serialized_server_utilization_metrics (more in line with other PRs) - decorates Ping grpc api function

2023-12-28

Unify naming under "utilization metrics"

[RFC] Add metrics retrieval to grpc server (#18721) Introduces a potential API for metrics retrieval from the GRPC server. - We retrieve metrics about the currently running requests of each type for a grpc server. This is implemented through a `retrieve_metrics` decorator. The decorator manages lifecycle and stores metadata about requests in a global threadsafe-access dictionary. - Methods decorated with `retrieve_metrics` are added to a registrar, which allows us to enforce that all new methods have the decorator. - The Ping API has been repurposed to snapshot these metrics. - Adds a test demonstrating metrics retrieval of a running sensor. - Adds a test which could be used to enforce that all GRPC methods are instrumented with this decorator. I'll leave the actual enforcement to a future PR. Using a decorator here keeps the implementation contained / doesn't muddy up the actual methods, while still allowing us to intercept arguments / retrieve more interesting information than just "a request of this type is running". Viewing slices of grpc server utilization over time like this can be incredibly powerful. It could power a view, for example, of the number of running sensors over the total number of workers, and which sensors are actually being run in each of those slices. There's definitely some potential spiciness with the global dictionary + lock shenanigans, but I think it stays pretty constrained. We're going to have to have some sort of global data structure to consolidate information regardless, I think. **Changes from round 1 of comments** - Record max_workers from the threadpool - Make metrics retrieval opt-in - Only record count per request type

2023-11-30

Implement a sync version of the schedule and sensor grpc call (#18261) ## Summary & Motivation We've never needed this to be an async call and we seem to be hitting upstream grpc issues with streaming API calls where they sporadically return early without raising an error. Make a sync version of the call and use that. For back-compat, fall back to the async version if an UnimplementedError is returned. This adds a marginal bit of latency in the case where you update your webserver/daemon/dagster cloud agent before you upgrade your code, but shouldn't increase the throughput on the boxes since the first call returns more or less immediately with an UnimplementedError without doing any actual work. If that's a concern we could try to add some caching of the fact that a particular server doesn't have the method implemented, but my sense is that that is more trouble than it is worth. ## How I Tested These Changes BK, test back-compat case manually by running a sensor locally on a new version of dagster-webserver but an old version of `dagster api grpc'

2023-06-01

Add a `dagster code-server start` CLI entrypoint with the ability to reload code without restarting the process (#14275) ## Summary & Motivation A new entry point that adds a layer of indirection between the entrypoint process, and a subprocess that actually runs user code. The main benefit this gives is the ability to reload code without restarting the process (or re-deploying the k8s pod / task / etc.) - resolving https://github.com/dagster-io/dagster/issues/12581. The main entry point implements the same gRPC api, but then redirects requests to a subprocess that it spins up. The parent process uses a GrpcServerRegistry under the hood, which handles things like cleaning up subprocesses after they die and are no longer running any executions. Unfortunately since gRPC servers still don't always have access to the instance, had to make the instance argument to GrpcServerRegistry optional (and add a few more config toggles since they get passed through). Also added a new graphql test suite which should exercise the actual underlying gRPC api pretty comprehensively, and added a test that verifies that reloading works the way you would expect. This is a large PR, I could potentially split it into two - one that adds the entrypoint, the other that implements the reload logic in dagster-graphql.

2023-05-23

Add a `dagster code-server start` CLI entrypoint with the ability to reload code without restarting the process (#14275) ## Summary & Motivation A new entry point that adds a layer of indirection between the entrypoint process, and a subprocess that actually runs user code. The main benefit this gives is the ability to reload code without restarting the process (or re-deploying the k8s pod / task / etc.) - resolving https://github.com/dagster-io/dagster/issues/12581. The main entry point implements the same gRPC api, but then redirects requests to a subprocess that it spins up. The parent process uses a GrpcServerRegistry under the hood, which handles things like cleaning up subprocesses after they die and are no longer running any executions. Unfortunately since gRPC servers still don't always have access to the instance, had to make the instance argument to GrpcServerRegistry optional (and add a few more config toggles since they get passed through). Also added a new graphql test suite which should exercise the actual underlying gRPC api pretty comprehensively, and added a test that verifies that reloading works the way you would expect. This is a large PR, I could potentially split it into two - one that adds the entrypoint, the other that implements the reload logic in dagster-graphql.

2023-05-19

Add a grpc proxy server

2023-05-09

Add a grpc proxy server

2023-04-04

[resource verification][2b/n] Implement GRPC call for ConfigVerifiable checks using tick abstraction

2023-04-03

Move to generic user-code-execution GRPC endpoint

[resource verification][2/n] Generate verification jobs, GRPC call for ConfigVerifiable checks

2022-11-29

Get current runs on grpc server (#10525) ### Summary & Motivation Expose the self._executions map over grpc ### How I Tested These Changes unit tests

2022-09-27

[managed stacks][wip] Prototype managed stack API

2022-09-15

[grpc] deferred snapshot support (#9630)

2022-09-14

[grpc] deferred snapshot support (#9630)

2022-09-06

[prototype] defer snapshots [INTERNAL_BRANCH=al/08-31-_prototype_defer_snapshots]

2022-07-23

Mark dagster.{daemon,generate,grpc,loggers} private (#8884)

2022-03-14

grpc > _grpc

2021-12-20

grpc call for dynamic partitions

2021-08-23

create grpc call to send back remote notebook content (#4515) - create grpc call to send back remote notebook content - include_notebook_route defaults to false - disable the notebook route by default for security

2021-03-29

change sensor/schedule/backfill grpc calls to use streaming query to avoid limit on message size Summary: also, seems like there's an opportunity to consolidate some of the streaming boilerplate, but leaving that to a separate diff Test Plan: bk Reviewers: dgibson, sashank Reviewed By: dgibson Differential Revision: https://dagster.phacility.com/D7127

2020-11-23

[sensors-2] sensor def Summary: Sets up a sensor definition, inheriting from the abstract job type. Currently all jobs are defined by: pipeline_name solid_selection mode At evaluation time, sensors return a should_execute boolean, a run_config dict, and a tags dict. An alternative implementation has a sensor defined by a single job params function, which returns a list of JobParams (containing a run config dict and a tags dict). Will send out in a separate diff for comparison. Test Plan: bk Reviewers: schrockn, dgibson, sandyryza Reviewed By: dgibson Subscribers: sandyryza Differential Revision: https://dagster.phacility.com/D5019

[sensors-1] redefine schedules as jobs Summary: Rips out the old job definitions, builds schedules on top of jobs. Test Plan: bk Reviewers: dgibson, schrockn, alangenfeld, johann Reviewed By: dgibson Differential Revision: https://dagster.phacility.com/D5011

2020-11-21

Revert "[sensors-1] redefine schedules as jobs" This reverts commit 7e4c53bd2498024cd39fd1a02d5179e2f318c291.

[sensors-1] redefine schedules as jobs Summary: Rips out the old job definitions, builds schedules on top of jobs. Test Plan: bk Reviewers: dgibson, schrockn, alangenfeld, johann Reviewed By: dgibson Differential Revision: https://dagster.phacility.com/D5011

2020-11-13

(dagit-reload-1/n) Add GetServerId gRPC service method Summary: This diff adds a `GetServerId` that exposes a unique UUID associated with each server. This is used to help track gRPC servers updating. Test Plan: unit Reviewers: alangenfeld, dgibson, prha, max Reviewed By: dgibson Differential Revision: https://dagster.phacility.com/D5105

grpc ExecuteRun api call fixes Summary: - Missing test coverage on the 'launch after cleaning up server case' - wait for the process to clean up before leaving the ExecuteRun API call Test Plan: BK + Azure Reviewers: alangenfeld, max, prha, johann Reviewed By: alangenfeld Differential Revision: https://dagster.phacility.com/D5119

2020-10-22

RFC: rename executable => job Test Plan: bk Reviewers: schrockn, alangenfeld, dgibson Reviewed By: schrockn, dgibson Differential Revision: https://dagster.phacility.com/D4837

2020-10-15

Increase gRPC message size limit for ExternalRepository API call Summary: Apparently prezi has a large enough pipeline that they are running into this. Bump the default for this specific API call since that's the one that can have large respnoses (eventually we could make it configurable in the instance too) Test Plan: Load dagit on normal pipeslines, no noticable slowdown despite increasing the message limit BK for large pipeline Reviewers: sashank, max, alangenfeld Reviewed By: sashank Differential Revision: https://dagster.phacility.com/D4750

2020-09-18

TriggeredExecutionDefinition => ExecutableDefinition Summary: Mostly just a rename... also removed `should_execute`, will rely on sensors/schedules for instigation policy Depends on D4458. Test Plan: bk Reviewers: alangenfeld, schrockn, catherinewu Reviewed By: alangenfeld Differential Revision: https://dagster.phacility.com/D4467

2020-08-27

[trigger-2] host representation for triggers Summary: Add host representation, out of process execution of trigger params TODO: Combine some of the boilerplate for schedules, triggers, and partition params Test Plan: bk Reviewers: alangenfeld, sashank, schrockn Reviewed By: sashank Subscribers: catherinewu Differential Revision: https://dagster.phacility.com/D4284

2020-08-18

[partition-backfill-1] backfill data IPC/GRPC implementations Summary: Batch up the calls to fetch partition tags / config to minimize RPC calls to the user process when doing backfills Test Plan: bk Reviewers: dgibson, alangenfeld Reviewed By: alangenfeld Differential Revision: https://dagster.phacility.com/D4169

2020-08-11

nicer error on failure to load repository in external process Summary: Addresses https://github.com/dagster-io/dagster/issues/2772 Example error produced from `dagster pipeline execute`: ``` Traceback (most recent call last): File "/Users/sryza/.pyenv/versions/dagster-3.6.8/bin/dagster", line 11, in <module> load_entry_point('dagster', 'console_scripts', 'dagster')() File "/Users/sryza/dagster/python_modules/dagster/dagster/cli/__init__.py", line 38, in main cli(obj={}) # pylint:disable=E1123 File "/Users/sryza/.pyenv/versions/3.6.8/envs/dagster-3.6.8/lib/python3.6/site-packages/click/core.py", line 764, in __call__ return self.main(*args, **kwargs) File "/Users/sryza/.pyenv/versions/3.6.8/envs/dagster-3.6.8/lib/python3.6/site-packages/click/core.py", line 717, in main rv = self.invoke(ctx) File "/Users/sryza/.pyenv/versions/3.6.8/envs/dagster-3.6.8/lib/python3.6/site-packages/click/core.py", line 1137, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/Users/sryza/.pyenv/versions/3.6.8/envs/dagster-3.6.8/lib/python3.6/site-packages/click/core.py", line 1137, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/Users/sryza/.pyenv/versions/3.6.8/envs/dagster-3.6.8/lib/python3.6/site-packages/click/core.py", line 956, in invoke return ctx.invoke(self.callback, **ctx.params) File "/Users/sryza/.pyenv/versions/3.6.8/envs/dagster-3.6.8/lib/python3.6/site-packages/click/core.py", line 555, in invoke return callback(*args, **kwargs) File "/Users/sryza/dagster/python_modules/dagster/dagster/cli/pipeline.py", line 262, in pipeline_execute_command return _logged_pipeline_execute_command(config, preset, mode, DagsterInstance.get(), kwargs) File "/Users/sryza/dagster/python_modules/dagster/dagster/core/telemetry.py", line 89, in wrap result = f(*args, **kwargs) File "/Users/sryza/dagster/python_modules/dagster/dagster/cli/pipeline.py", line 290, in _logged_pipeline_execute_command result = execute_execute_command(env, kwargs, mode, tags) File "/Users/sryza/dagster/python_modules/dagster/dagster/cli/pipeline.py", line 297, in execute_execute_command external_pipeline = get_external_pipeline_from_kwargs(cli_args, instance) File "/Users/sryza/dagster/python_modules/dagster/dagster/cli/workspace/cli_target.py", line 411, in get_external_pipeline_from_kwargs external_repo = get_external_repository_from_kwargs(kwargs, instance) File "/Users/sryza/dagster/python_modules/dagster/dagster/cli/workspace/cli_target.py", line 374, in get_external_repository_from_kwargs repo_location = get_repository_location_from_kwargs(kwargs, instance) File "/Users/sryza/dagster/python_modules/dagster/dagster/cli/workspace/cli_target.py", line 342, in get_repository_location_from_kwargs workspace = get_workspace_from_kwargs(kwargs, instance) File "/Users/sryza/dagster/python_modules/dagster/dagster/cli/workspace/cli_target.py", line 198, in get_workspace_from_kwargs return workspace_from_load_target(created_workspace_load_target(kwargs), instance) File "/Users/sryza/dagster/python_modules/dagster/dagster/cli/workspace/cli_target.py", line 168, in workspace_from_load_target user_process_api=python_user_process_api, File "/Users/sryza/dagster/python_modules/dagster/dagster/cli/workspace/load.py", line 253, in location_handle_from_python_file attribute=attribute, File "/Users/sryza/dagster/python_modules/dagster/dagster/api/list_repositories.py", line 25, in sync_list_repositories raise DagsterSubprocessError(result.to_string(), subprocess_error_infos=[result]) dagster.core.errors.DagsterSubprocessError: (ImportError) - ImportError: attempted relative import with no known parent package Stack Trace: File "/Users/sryza/dagster/python_modules/dagster/dagster/cli/api.py", line 143, in list_repositories_command python_file, module_name, working_directory, attribute File "/Users/sryza/dagster/python_modules/dagster/dagster/grpc/utils.py", line 20, in get_loadable_targets else loadable_targets_from_python_file(python_file, working_directory) File "/Users/sryza/dagster/python_modules/dagster/dagster/cli/workspace/autodiscovery.py", line 11, in loadable_targets_from_python_file loaded_module = load_python_file(python_file, working_directory) File "/Users/sryza/dagster/python_modules/dagster/dagster/core/code_pointer.py", line 88, in load_python_file return import_module_from_path(module_name, python_file) File "/Users/sryza/dagster/python_modules/dagster/dagster/seven/__init__.py", line 110, in import_module_from_path spec.loader.exec_module(module) File "<frozen importlib._bootstrap_external>", line 678, in exec_module File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed File "examples/legacy_examples/dagster_examples/simple_pyspark/pipelines.py", line 10, in <module> from .solids import ( ``` We could do some user_code_error_boundary stuff to make this even nicer, but I think this gives us parity with what we had a couple releases ago. Test Plan: added tests, ran "dagster pipeline execute" in both grpc and cli mode Reviewers: alangenfeld, dgibson, max Reviewed By: max Differential Revision: https://dagster.phacility.com/D4087

2020-08-06

Heartbeat for GRPC server Summary: This lets managed servers shut themselves down when the client processes disappear Test Plan: Unit Reviewers: dgibson Reviewed By: dgibson Differential Revision: https://dagster.phacility.com/D4099

2020-08-05

k8s user code deployments Test Plan: bk / manual. Reviewers: alangenfeld, nate Reviewed By: nate Subscribers: nate Differential Revision: https://dagster.phacility.com/D4041

2020-08-04

explain what devs must do after editing api.proto Test Plan: none Reviewers: prha, max Reviewed By: prha Differential Revision: https://dagster.phacility.com/D4095

2020-07-29

Persistent GRPC run launcher Summary: Introduces StartRun Test Plan: unit Reviewers: alangenfeld, dgibson, johann, prha, sashank Reviewed By: alangenfeld, dgibson Differential Revision: https://dagster.phacility.com/D4003

2020-07-25

Support soft run termination over GRPC Test Plan: Unit Reviewers: sashank, prha, alangenfeld, dgibson, johann Reviewed By: alangenfeld Differential Revision: https://dagster.phacility.com/D3954

2020-07-21

Support soft server termination over GRPC Test Plan: Unit Reviewers: sashank, prha, alangenfeld, johann, dgibson Reviewed By: dgibson Differential Revision: https://dagster.phacility.com/D3946

2020-07-20

Update GRPC server to store target Test Plan: unit Reviewers: dgibson, alangenfeld Reviewed By: dgibson Subscribers: alangenfeld Differential Revision: https://dagster.phacility.com/D3941

2020-07-08

Streaming API call over GRPC Summary: This moves execute_run to use GRPC. A follow-on diff will implement the (fairly complex) soft termination scheme. Test Plan: Unit Reviewers: alangenfeld, schrockn Reviewed By: alangenfeld, schrockn Differential Revision: https://dagster.phacility.com/D3662

2020-07-06

Snapshot schedule execution over GRPC Test Plan: Unit Reviewers: alangenfeld, schrockn Reviewed By: alangenfeld Differential Revision: https://dagster.phacility.com/D3720

External repositories over GRPC Test Plan: Unit Reviewers: alangenfeld, schrockn Reviewed By: alangenfeld Differential Revision: https://dagster.phacility.com/D3719

External pipeline snapshot over GRPC Test Plan: Unit Reviewers: alangenfeld, schrockn Reviewed By: alangenfeld Differential Revision: https://dagster.phacility.com/D3718

Implement snapshot partition over GRPC Test Plan: Unit Reviewers: alangenfeld, schrockn Reviewed By: alangenfeld Differential Revision: https://dagster.phacility.com/D3711

Implement list_repositories over GRPC Summary: This also switches the ephemeral client to use a module-based scheme for starting the server, so that we can stub in different python interpreters more easily. Test Plan: Unit Reviewers: alangenfeld, schrockn Reviewed By: alangenfeld Differential Revision: https://dagster.phacility.com/D3679

Unary API call over GRPC Summary: This demonstrates moving a unary API call onto the new GRPC system. Test Plan: Unit Reviewers: alangenfeld, schrockn Reviewed By: alangenfeld Differential Revision: https://dagster.phacility.com/D3656

2020-06-26

Scaffolding for GRPC Summary: This introduces a CLI command, server and client classes and helpers, and generation machinery from protobuf. Test Plan: Unit Reviewers: alangenfeld, schrockn Reviewed By: alangenfeld Differential Revision: https://dagster.phacility.com/D3610