Proto commits in maelstrom-software/maelstrom

These 85 commits are when the Protocol Buffers files have changed:

2025-04-10

Rename the `command` field to `program` in the CommandLayer.

The documentation is generated from this commit.

Rename program layers to command layers.

2025-03-26

Introduce new LayerSpec::Program and related types. Make sure we can parse them correctly. There is no template variable expansion for program layers.

2025-03-19

Change broker to use new github config values.

2025-03-18

Nix the MixedClusterCommunicationStrategy type. Now, the client will either go all-TCP or all-GitHub.

2025-03-17

Send ClusterCommunicationStrategy with client's start rpc.

2025-03-03

Remove state_dir from client. As I was updating the documentation, I realized that we no longer use the state_dir in the client. It's now only used in the test runner to store the database of test runs.

2025-02-19

Change Restart to Stop. This is now called implicitly when the Client is dropped. The idea behind doing this in the drop method is that we want to be able to immediately start another Client and know that the old one cleaned up.

2024-12-13

Change how the client protocol specifies root-overlay. The base JobSpec has a single field called root_overlay. This can be one of None, Tmp, or Local. The latter contains two fields to indicate what directory the overlayfs should use. Before this change, the RootOverlay type, which is what the root_overlay field is, was propagated all the way up to the client's ContainerSpec. The problem was that this didn't really make much sense from the point of view of how clients wanted to specify this field. From a client's point of view, they really just want to have an `enable_writable_file_system` flag. The Local value is really a more hidden thing that will hopefully, eventually, go away or at least be specified differently. To that end, this patch changes the client's JobSpec to have a separate field called capture_file_system_changes. This is on the JobSpec, not on the ContainerSpec. The idea is that this will only be used by maelstrom-pytest to generate the container used to run tests in. When command layers are working, this will go away completely. This patch also changes the root_overlay field on the client's ContainerSpec to be an enable_writable_file_system bool field. This is how the clients actually think about things, so it matches well.

2024-12-11

Make network and root_overlay optional in ContainerSpec, CollapsedJobSpec, and TestMetadataContainer. We want to be able to distinguish between values that aren't set and values that are set to the default value so that we can stack containers properly.

2024-12-06

Rename ImageSpec to ImageRef.

2024-12-05

protobuf: Make ArtifactTransferStrategy have correct variant naming

Rename ArtifactUploadStrategy to ArtifactTransferStrategy I want to re-use this type in the worker

Rename some enum variants and a "message" field in items.proto.

Add container parents to containers.

Use ImageSpec inside of ContainerParent::Image. This simplifies things.

Use an EnumSet of ImageUse in ContainerParent::Image and ImageSpec instead of three bools.

Replace ImageSpec with ContainerParent. This plumbs through a change where every container can have a parent that is either another container or an image.

Remove ContainerRef and instead always embed a ContainerSpec in a JobSpec. This is the first step of some changes. We're going to introduce a `parent` field to the ContainerSpec that will replace the `image` field. This will be either an image, a named container, or nothing.

Hook up github artifact upload in client-process to new start argument

2024-10-09

Remove clear_cached_layers RPC We are using "restart" instead

Add restart RPC to client-process and use in test-runner

Add a "clear_layer_cache" call the client can use It calls it when restarting in watch mode

2024-08-28

Add new priority field to JobSpec.

2024-08-22

Remove JobStateCounts from IntrospectResponse

2024-08-21

Stream JobStatus back from run_job in client

Rename cjid to client_job_id in JobCompletedStatus

Add protobuf types for the job status stuff

2024-08-18

Rename Layer to LayerSpec to match JobSpec and ContainerSpec

2024-08-14

Change JobSpec to be able to accept a named container

Add add_container client RPC (unimplemented)

Remove some now unused protobuf things

Name our protobuf enums variants correctly They follow C style name scoping rules, and are usually shouty-snake-case prost also is expecting this kind of naming

Forward client-process log messages to client via RPC

2024-08-13

Split out ContainerSpec from JobSpec

2024-08-12

Remove add_layer client API Instead just pass what was being accepted in add_layer directly in the JobSpec. I don't try to parallelize the layer creation at all

Add `enum_type` feature proto-buf conversion macros This is only for enums containing fields and aids in wrapping them up in structs

2024-08-07

Add SharedLibraryDependencies layer type It just returns an error if used right now

2024-07-18

Make working_directory, user, and group optional in the JobSpec. Fixes #348.

2024-07-09

Remove old, deprecated devices stuff. Devices are now specified as mounts. Fixes #321.

2024-06-13

Add --accept-invalid-remote-container-tls-certs configuration option This will let us connect to a service using a self-signed certificate which is useful for locally-run registries

2024-06-07

Set the pty window size properly.

2024-06-06

Move image tag parsing to inside maelstrom-container This removes the tag field from the various ImageSpecs, instead the image name will just include the tag and in a future change any of the other information. Doing this parsing inside maelstrom-container is very convenient since now the various front-ends to the client don't have to worry about parsing valid image names. This means that some validation is now happening later rather than earlier, but I think the convenience of it outweighs this concern. You have to try pretty hard to create an invalid image name.

Remove ptmx device type. It turns out that bind mounting this into a container doesn't really work. It seems like what you're supposed to do for Linux containers is to use the ptmx in /dev/pts. Our maelstrom-test.toml now has an example of this.

2024-06-05

Add allocate_tty field to JobSpecs. Currently this is an Option<AbstractUnixDomainAddress>, where AbstractUnixDomainAddress is an array of 6 bytes. I chose this 6-byte array because that's what we'll get back from the autobind feature of the linux kernel for abstract unix domain sockets.

2024-05-30

Do all mounts using the multi-step, new linux mount API.

Add a JobRootOverlay. This replaces the enable_writeable_file_system field in the JobSpec. However, at this point, both clients continue to parse the enable_writable_file_system flag and convert it to a JobRootOverlay.

Remove overlay mount type. We're instead just going to provide overlay mounts for the root directory.

2024-05-27

Add a new Overlay mount type. More to come.

Add `devices` mount type. This will replace the `devices` field of the `JobSpec`.

Add support for /dev/ptmx.

2024-05-25

Reorder some cases for JobMounts.

Add support for devpts and mqueue file system types.

2024-05-21

Re-order the JobSpec fields to be a little more logical.

2024-05-20

Add /dev/shm device to maelstrom

2024-05-17

Replace bind mount flags with "read_only" boolean field.

Remove "recursive" flag for bind mounts. There is no point having this flag. We basically always force it to be set now. Here's the comment from the code: We always pass recursive here because non-recursive bind mounts don't make a lot of sense in this context. The reason is that, when a new mount namespace is created in Linux, all pre-existing mounts become locked. These locked mounts can't be unmounted or have their mount flags adjusted. In this context, we're always in a new mount namespace, hence all mounts on the system are locked. When you try to call open_tree without AT_RECURSIVE on a tree that has mounts under it, you get an error. The reason is that allowing the binding of this subtree would have the effect of revealing what was underneath those locked mounts -- in effect, unmounting them. If we allowed the user to specify the recursive flag, it would just mean that their mount would fail if there happened to be any mount at that point or lower in the tree. That's probably not what they want.

2024-05-16

Replace BindMount::access with BindMount::flags. Flags are optional.

2024-05-15

Add in new JobMount::Bind variant. This will panic in the executor if it is ever encountered. We'll add in actual support in the next few patches.

Change how JobMount is represented to prepare for adding in a new variant. This changes the representation, but keeps the TOML and JSON parsing the same. It does change the protobuf representation.

2024-05-13

Change enable_loopback in maelstrom_client_base::spec::JobSpec to a new JobNetwork enum. This enum has three variant: disabled, enabled, and local. The local variant indicates that the job has to be run on the local runner, and it will be run in the enclosing network namespace.

2024-05-09

Remove the get_container_image client RPC

2024-05-08

Add estimated_duration field to client JobSpec

Change proto-buf JobSpec to support upcoming image changes The way the environment is evaluated is going to change so that evaluation happens on the other side of the RPC. The structures have been changed to support that, but it doesn't evaluate anything yet. Similarly the image field is ignored right now.

Change start field numbering to sequential We aren't worrying about backward compatibility here yet

2024-05-07

Remove add_artifact RPC, it is unused

Add introspect client RPC to replace two existing ones

2024-05-06

Rename AddJob RPC to RunJob This makes more sense since the RPC returns when the job finishes.

Remove wait_for_outstanding_jobs RPC This RPC is error-prone because it is easy to think it will wait for all your job RPCs to have completed, but it isn't able to do that. Furthermore, if you have threads waiting for jobs to complete, this call can't possibly wait for those to finish. It is better to encourage some kind of client-side synchronization instead.

2024-05-03

Plumb container_image_depot_dir out to clients.

Plumb in new StateDir.

2024-04-23

Time tests being run and keep in JobEffects see #81

2024-04-19

Pass the container_image_depot_cache_dir into the client process. We very quickly become root in our own namespace in the client process, so we can't use xdg to get the result when we need it. In theory, we could get the value before we go into our own namespace, but it seems cleaner to just have the start message contain in.

Plumb the local worker config parmeters into the protobuf.

Remove unused ProcessBrokerMsgSingleThreadedRequest.

Get first cut of standalone working. If you don't provide a broker to cargo-maelstrom or maelstrom-run, you'll enter standalone mode.

Remove streaming response for GetContainerImage RPC. We'll add in something like GetArtifactUploadProgress later.

Convert ClientDriver into client_main.

Remove integration tests and the RPC messages they depend on. We're going to re-do these tests in a different way.

2024-04-04

Rename proto::JobOutcomeCompleted to JobCompleted.

2024-03-06

Add fuse device to make new fuse tests pass on maelstrom

2024-02-27

Add follow_symlinks flag to PrefixOptions

2024-02-26

cargo-maelstrom: progress bars for artifact upload

Restore the progress bar for downloading containers

2024-02-23

Replace client RPCs with tonic This breaks progress bar stuff with container images again I also accidentally added in updates to the cargo-maelstrom progress indicator messages