Proto commits in stateful/runme

These 93 commits are when the Protocol Buffers files have changed:

2025-05-07

Restructure internal, pkg, and neither (aka unnested) (#792) Instead of making vague stability guarantees via `pkg` let's unnest anything that's not truly considered `internal`. --------- Signed-off-by: Sebastian (Tiedtke) Huckleberry <sebastiantiedtke@gmail.com>

The documentation is generated from this commit.

2025-05-05

Unnest anything that's not truly internal Signed-off-by: Sebastian (Tiedtke) Huckleberry <sebastiantiedtke@gmail.com>

The documentation is generated from this commit.

Unnest anything that's not truly internal Signed-off-by: Sebastian (Tiedtke) Huckleberry <sebastiantiedtke@gmail.com>

The documentation is generated from this commit.

2025-04-03

Remove ExecuteOneShot since it has no clients, no need for deprecation (#776) This API became obsolete when we figured out how to do full-duplex streaming in the browser. Signed-off-by: Sebastian (Tiedtke) Huckleberry <sebastiantiedtke@gmail.com>

2025-03-18

Move from stateful to runmedev org as part of CNCF sandbox admission (#762) Move the new package namespace (`runmedev/runme`) explicitly instead of relying on GitHub's redirect from the old `stateful/runme`. --------- Signed-off-by: Sebastian (Tiedtke) Huckleberry <sebastiantiedtke@gmail.com>

2025-03-16

Server Streaming Execute Request (#764) Add a server streaming (half duplex) execute method. This is intended to support execution from the browser. --------- Signed-off-by: Jeremy Lewi <jeremy@lewi.us> Signed-off-by: Sebastian (Tiedtke) Huckleberry <sebastiantiedtke@gmail.com> Co-authored-by: Sebastian (Tiedtke) Huckleberry <sebastiantiedtke@gmail.com>

2025-03-13

Introduce Env Variable Retention strategies (toggle between shell- and docs-mode) (#761) In a nutshell, this PR introduces different strategies to retain environment variables. This option will allow to make available a user setting that sets it's value according to the use case. The problem is that Runme's user base has different use cases that roughly break down into these two: ### 1. "Docs Mode" Existing runbooks/notebooks/markdown with placeholder values, e.g., export OPEN_API_KEY=[insert super secret key]. Once a value is set/confirmed by a user prompt, don't ask/evaluate again (unless overwritten promptEnv requires it in cell/document config). Essentially, the **first** value resolved for a variable is retained, and subsequent reassignments are ignored. ### 2. "Shell Mode" Take exported environment variables as is, aka "shell script" style. e.g. export MY_VAR=foo followed by export MY_VAR=bar, the last ran/evaluated value wins. When reassigned, subsequently the **last** value assigned is retained. ### Protobuf Changes This PR introduces `Retention` with values `RetentionFirstRun` and `RetentionLastRun`. If `RetentionUnspecified` it defaults to `RetentionFirstRun` which is up for reconsideration in a future major release. Related to https://github.com/stateful/runme/issues/731. Related to https://github.com/stateful/runme/issues/653#issuecomment-2652222056. --------- Signed-off-by: Sebastian (Tiedtke) Huckleberry <sebastiantiedtke@gmail.com>

2025-03-06

Unlock DirEnv Support (#757) This PR introduces support for `.envrc` files at the root of a project. The root is usually the git repo's root. Runme requires `direnv` to be installed locally. **Important: Runme's direnv implementation does not support abritrary directories and changes between them since it's not a freewheeling terminal.** PR includes: - [x] runnerv1 supports `direnv` - [x] runnerv1 tests pass in CI/CD All of runnerv2 will be handled in a separate PR: - runnerv2 supports `direnv` - runnerv1 tests pass in CI/CD --------- Signed-off-by: Sebastian (Tiedtke) Huckleberry <sebastiantiedtke@gmail.com>

2025-02-13

Fix imports globally (#741)

2025-02-06

First class dagger shell (#736) In order to natively support Dagger's shell Runme needs to take the entire notebook as opposed to an atomic cell into account. This PR introduces the APIs necessary to transform a notebook into a Dagger Shell script using some basic AST transformation. Integration with extension in https://github.com/stateful/vscode-runme/pull/1947. PS: The protos are not published yet and need local-gen to work. The new service will turn a deserialized version of the following markdown into a runnable Dagger shell script. ## Simple example notebook Unamed cell: ```sh {"closeTerminalOnSuccess":"false","interactive":"true","openTerminalOnError":"true","terminalRows":"23"} git github.com/stateful/runme | head | tree | file examples/README.md ``` And the same as a named cell: ```sh {"name":"README"} # Named in README in cell attributes git github.com/stateful/runme | head | tree | file examples/README.md ``` ## Resulting Dagger Shell Script Running the unamed cell: ```sh DAGGER_01JKE40B6A0F5NBESE49X122RD() { git github.com/stateful/runme \ | head \ | tree \ | file examples/README.md } DAGGER_01JKE40B6A0F5NBESE49X122RD ``` Running the named cell: ```sh DAGGER_01JKE4K781F87XRK9SQZ9T1GQD() { git github.com/stateful/runme \ | head \ | tree \ | file examples/README.md } README() { git github.com/stateful/runme | head | tree | file examples/README.md } README ```

2024-12-02

Implement "runme beta session" (#683) It implements a new command that allows starting a session and collecting all exported environment variables throughout its lifetime. ## Testing As a prerequisite, add the following to your shell's startup script (`.bashrc` or `.zshrc`): ``` # runme source <(runme beta session setup) ``` Then, you can start a new session in the terminal: ``` runme beta session ``` The session is a regular shell of your choice, inferred from `$SHELL`. You can use it like a normal shell, but any exported environment variables will be returned upon exiting the session via `exit`. Currently, the collected variables are printed out, but in the future, they will be routed to a proper store. --------- Co-authored-by: Sebastian (Tiedtke) Huckleberry <sebastiantiedtke@gmail.com>

2024-10-21

Integrate Owl Store into Runner v2 (#686) Refactor integration based on lessons learned in v1.

2024-10-04

internal/config: use JSON Schema (#678) This change switches the `runme.yaml` schema definition from protobuf to JSON Schema. This results in fewer conversions and more natural representation. It also adds support for parsing and merging multiple configuration files. Default config was moved to [a file](internal/config/runme.default.yaml). Go's struct representing the config is generated from the schema. When you change `internal/config/config.schema.json`, remember to call `make schema/generate`. --------- Co-authored-by: Sebastian Tiedtke <sebastiantiedtke@gmail.com>

2024-09-16

Use superset of tags and categories plus deprecation notices

Substitute category (deprecate) with tag (#667) Users have reported that `tags` are self-explanatory, whereas `categories` are not. 1. In the stable CLI interface, let's move off `category` without interrupting folks. 2. In `beta`, there will only be a `tag`.

2024-09-13

Only resolve programs that are in shell (#668) The CLI is broken when any non-shell tasks/cells are being run. The problem is that everything's treated as Shell and goes through the program resolver (env var extraction/prompting etc).

2024-09-04

Merge remote-tracking branch 'upstream/main' into jlewi/protocleanup

2024-09-03

Init reporter (#655) We've decided to forgo using the reporter to normalize the JSON payload and, instead, at a later stage, refactor it to report payloads to the remote backend. For now, disable the API (keep the types in place) so it's not accidentally relied on. --------- Co-authored-by: Sebastian Tiedtke <sebastiantiedtke@gmail.com>

2024-09-02

Remove AI Service Protocol Definition * Foyle is now publishing its protos to the buf registry and the client libraries pulled from the registry are what's used by the vscode extension. * The AI service defined in the runme repository is no longer needed. * Related to jlewi/foyle#173

2024-08-26

runner: rename runner.v2alpha1 to runner.v2 (#651) Simply rename `runner.v2alpha1` to `runner.v2`. --------- Co-authored-by: Sebastian Tiedtke <sebastiantiedtke@gmail.com>

2024-08-15

Temp provide autoconf defaults (#650) Allow beta sub-commands to run without repo config file

2024-07-31

Owl more validation (#583) This is in a good state for runner v1. Let's merge and work from smaller feature branches rather than continually rebasing this branch. - Introduce complex specs (working title; impeding renaming) - Introduce tag and dburl validation - Propagate execution info via context Unfinished.

2024-07-29

command: fix signalling to command process group (#637) This change also fixes a regression that broken running interactive commands, like runme in runme, run from the CLI in https://github.com/stateful/runme/pull/637/commits/da2c38c928edec6858b1833bcfb3e2db638d2090. It also disables FIFO which should mitigate https://github.com/stateful/runme/issues/635 but a proper fix is needed. Relates to https://github.com/stateful/runme/issues/636

2024-07-01

Allow terminalRows to be set at document-level (#622) Allow overwriting the default `terminalRows` setting for the entire document in the frontmatter.

2024-05-24

GRPC changes to enable a gradual migration from Runner v1 to v2 (#589) Intentionally leaving the service stubs unimplemented to allow changing them later. Having the stubs in place will however make it easier to have the extension temporarily support both Runner v1 & v2.

2024-05-23

Remove kernel from runme.yaml and add explicit docker support (#582) This change removes complicated kernels and introduces an explicit support for Docker: ```yaml docker: enabled: true image: runme-runtime:latest build: context: ./experimental/docker dockerfile: Dockerfile ``` Running code blocks in Docker is still not fully implemented.

2024-05-15

Define AI GRPC service to support using AI to generate cells (#573) We'd like to use AI to generate cells. In particular, we'd like to support using [Foyle](foyle.io) to add AI capabilities. The initial plan is to use a gRPC service to allow RunMe to communicate with Foyle (or potentially other AI services). This proto needs to define that gRPC service.

2024-05-14

Move `api` & `gen` from internal to pkg (#575) This refactor will allow linking against Runme's proto API definitions.

2024-05-10

Fix shebang++ default interpreter "cat" when language is unknown (#568) Initial implementation was wrong and didn't stick out due to lack of integration testing. --------- Co-authored-by: Adam Babik <a.babik@designfortress.com>

2024-05-06

Implement TerminalCommand and Terminal command mode (#563) This change implements a new mode called `TERMINAL`. In this mode, the server, right after starting the script, writes initial commands that are responsible for collecting environment variables, which will be added or deleted to the current session. The `TERMINAL` mode is supported only in the `v2alpha1` runner. Fixes https://github.com/stateful/runme/issues/551

2024-04-22

Unlock shebang++ (#560) 1. Allow cells to be store in ENV as long as their name conforms to an opinionated layout (we can give visual feedback in the notebook UI). 2. If a language is not executable and no interpreter/program is set, run `cat` to stash the input cell in the output. This will enable use cases where e.g. you have a SQL query you're live-editing in one cell and running it with `$ bq query` (BigQuery CLI) in another cell. Enables better interplay between cells than just running them back-to-back (`$__`).

2024-04-19

Implement command execution in Docker (#548) This PR introduces a `DockerCommand` which allows to execute code blocks in Docker. It's equivalent to `NativeCommand` and `VirtualCommand`. Co-authored-by: Sebastian Tiedtke <sebastiantiedtke@gmail.com>

2024-04-12

Refactor config into kernel, project, etc

Retro fit config to separate core and repo

2024-04-09

Add mime type in ExecuteResponse in runner service (#549) Mime type is sent in the first `ExecuteReponse` that contains any stdout; it is sent only once. Closes https://github.com/stateful/runme/issues/545 Co-authored-by: Sebastian Tiedtke <sebastiantiedtke@gmail.com>

2024-04-06

Fix comment

2024-03-28

Implement beta commands (#518) This PR implements a bunch of `runme beta` commands

2024-03-14

Include information about sensitive keys in Program Resolver (#531) Connect the dots between Owl Store and Program Resolver with a low amount of coupling. Likely needs a refactor but works for now 👌.

2024-03-13

Validate require vs optional and return validation errors (#527) Introduce the first error state required vs optional. This showcases the concept before going into the breadth of error states.

2024-03-11

Simple proposed Env Store API (#522) View model in protos that streams information into "Env Store" view. --------- Co-authored-by: Cristian Cepeda <43882+pastuxso@users.noreply.github.com>

2024-02-27

Return Script for now

Implement configuration in runme.yaml (#505) This PR introduces an implementation of file-based configuration in a form of `runme.yaml`.

514 bug go install command installs older version v182 instead of latest one (#517) Fix `go install github.com/stateful/runme@latest` to install actual latest

2024-02-26

Refactor ProgramResolver (#513) Follow up for https://github.com/stateful/runme/pull/512. --------- Co-authored-by: Sebastian Tiedtke <sebastiantiedtke@gmail.com>

2024-02-23

Evolve ResolveVars into ResolveProgram (#512) "Program" as in "Cell Shell Program".

Fix v2alpha1 service, needs better tests

Surface renaming

Renaming

Document protos

2024-02-21

Rename props in API

Respect var resolution mode

Respect resolution mode

wip

Fully delegate prompt resolution

2024-02-02

Backport ResolveEnv() to runnerv1

Add ResolveEnv() to runnerv2alpha1

Handle sessions and store_last_stdout in runnerv2service

Run commands natively in service when interactive=false

Implement new command run-locally. The main advantage is that it supports executing interactive commands, including runme-in-runme, and pipes (|). On top of that, there is the new package internal/command. It gives a better interface to create commands, as well as unifies the idea of a program config that now is defined in a proto file.

Implement ResolveEnv() in runnerv2service

Add UpdateSession() to runnerv2alpha1

Rename ResolveEnv to ResolveVars in runner.proto

Fix go_package in runnerv2alpha1 It also contains typo fixes and additional tests.

2023-12-28

Add session metadata to serialization + frontmatter (#458) * Baseline * Respect runme session frontmatter for deserialization * Fix tests * Add test cases * Typo

2023-12-08

Implement Project service (#438) * Add internal/project package It implements similar functionality to pkg/projects, but hides implementation details and is intended to be a building block for a gRPC service. * internal/project: implement Project service * Remove dependency on Project from Session * Canonical Project service examples with runme * Fix unit tests * Fix make test * Add info about generated name to LoadEventFoundTask * Fix unit tests after internal/project/testdata changes * Fix ldflags in .vscode config files * Use default options --------- Co-authored-by: Sebastian Tiedtke <sebastiantiedtke@gmail.com>

2023-12-01

First stab at serializing output (#443) * No code-gen * Check pointers * Minor changes to account for wrapped values * Use wrapper for pid * Break out text/image output items * Add serializer options * Handle outputs according to options * Flatten * Reconcile code-gen * Basic tests * Add note

2023-11-28

Includes notebook lifecycle id in protos (#440)

2023-11-20

Fix typo (#434) * Fix typo * Fix tests --------- Co-authored-by: Sebastian Tiedtke <sebastiantiedtke@gmail.com>

2023-11-13

Lifecycle identity refactor (#428) * Baseline refactor, tests still failing * Don't overwrite with v0.0 if set * More refactoring, tests still failing * Wrong mapping * Move to serialization phase * Preserve existing cell ID semantics * Make linter happy * Fmt * None until v2 * Deserialize options only

2023-11-10

Fix delete existing IDs on unspecified IDT (#423)

2023-11-07

Pre-release identity (#421) Co-authored-by: Sebastian Tiedtke <sebastiantiedtke@gmail.com>

2023-10-23

Add skipPrompts frontmatter option (#404) * Add skip_env frontmatter option * Add examples * Change to skipPrompts and fix yaml parser

2023-09-04

chore: blocks logic

2023-07-20

Support custom cell executions, "temp file" mode (#302) * support custom execution models * typescript with ts-node * fix file extension * support remote and local runners * support custom program in cell annotations * fix tests * fix lint * test command.go * test service, integrations * split shebang * error on invalid program * re-gen protos * update test script * fix tests * javascript example in docs * fix EOF * fallback to python3 * switch from `program` to `interpreter`

2023-07-06

don't prompt if already in session (#319) * don't prompt if already in session * support .env loading * fix lint

2023-05-30

expose cwd in frontmatter (#278)

2023-05-25

support custom shells (#266) * frontmatter in codeblock * use shell from frontmatter * expose frontmatter in parser * fix tests * add test * fix lint * unneeded function

2023-05-22

Project-based model (#243) * single file project * progress * basic run, ls support * cd to correct directory * better errors * prompt user if many tasks selected * repo location bug * add regex filter to list * use project in tui command * load environment variables in local runner * load envs in runner * fix lint * fix tests * fix lint * fix staticcheck * fix style * append block * project is opt-in * find git repo regardless * allow disabling gitignore * use project in tui runner * indicate filename in tui * fix paths, newlines on windows test * filter unnamed cells by default in projects * set project mode with `RUNME_PROJECT` * remove # of commands * sort in ascending nested order * sort tui * fix runme print command * remove commented code * protos * fix tests * fix staticcheck

2023-05-19

internal/runner: add option to store last stdout in `$__` (#250) * add store last stdout * make lint

2023-04-18

parser: add text range to code blocks (#218) * proto defs * proto changes * add text range to parser service * fix range calculation * fix eof

2023-04-11

Support background processes over CLI w/ gRPC runner (#194) * support background processes * lint errors * test background execution * add disable background cli flag * disable background to runner flags * disable background support by default

2023-03-13

internal/runner: add "most recent" session strategy (#184) * session list * new session list * better comment * session_test boilerplate * session strategy proto defs * implement for service * fix lock hanging * test at service level * test sessionlist * add session strategy option to run command * fix eof

2023-03-06

internal/runner: pty sizing (#168) * implement pty sizing * document `Winsize` * fix eof * dont run winsize tests in ci * better env

2023-02-17

internal/runner: add ExecuteRequest.stop to runner.proto

2023-02-16

internal/kernel: remove kernel and move shell session to runner

2023-02-13

internal/runner: add gRPC service

internal/runner: clean up

2023-01-20

kernel: use expect to detect prompt

internal/kernel: improve unit tests

kernel: implement missing methods in kernelServiceServer

internal/api: s/lang_id/language_id

2023-01-11

examples: add demo with gRPC client (#97)

2023-01-09

Add gRPC API and shell command (#93)