Proto commits in hashicorp/nomad

These commits are when the Protocol Buffers files have changed: (only the last 100 relevant commits are shown)

2025-01-29

remove remote task execution code (#24909)

The documentation is generated from this commit.

2024-11-12

[wip] tears, only tears

2024-11-04

backport of commit 658c429d75e405b5a30880b6a8d90b80196cc2a5 (#24351) Co-authored-by: Michael Smithhisler <michael.smithhisler@hashicorp.com>

2024-11-01

Drivers: add work_dir config to exec/raw_exec/java drivers (#24249) --------- Co-authored-by: wurosh <uros.m.perisic@gmail.com> Co-authored-by: Michael Schurter <mschurter@hashicorp.com> Co-authored-by: Tim Gross <tgross@hashicorp.com>

2024-10-03

driver: remove LXC and ECS driver documentation. (#24107) Co-authored-by: Aimee Ukasick <aimee.ukasick@hashicorp.com>

2024-06-14

backport of commit 85430be6dd48574124567211cacf4367b58fb2fb (#23324) Co-authored-by: Piotr Kazmierczak <470696+pkazmierczak@users.noreply.github.com>

raw_exec: oom_score_adj support (#23308)

2024-05-07

drivers/raw_exec: enable setting cgroup override values (#20481) * drivers/raw_exec: enable setting cgroup override values This PR enables configuration of cgroup override values on the `raw_exec` task driver. WARNING: setting cgroup override values eliminates any gauruntee Nomad can make about resource availability for *any* task on the client node. For cgroup v2 systems, set a single unified cgroup path using `cgroup_v2_override`. The path may be either absolute or relative to the cgroup root. config { cgroup_v2_override = "custom.slice/app.scope" } or config { cgroup_v2_override = "/sys/fs/cgroup/custom.slice/app.scope" } For cgroup v1 systems, set a per-controller path for each controller using `cgroup_v1_override`. The path(s) may be either absolute or relative to the controller root. config { cgroup_v1_override = { "pids": "custom/app", "cpuset": "custom/app", } } or config { cgroup_v1_override = { "pids": "/sys/fs/cgroup/pids/custom/app", "cpuset": "/sys/fs/cgroup/cpuset/custom/app", } } * drivers/rawexec: ensure only one of v1/v2 cgroup override is set * drivers/raw_exec: executor should error if setting cgroup does not work * drivers/raw_exec: create cgroups in raw_exec tests * drivers/raw_exec: ensure we fail to start if custom cgroup set and non-root * move custom cgroup func into shared file --------- Co-authored-by: Michael Schurter <mschurter@hashicorp.com>

2024-04-17

backport of commit 120c3ca3c942b79df6004f3b95ecaf3cfd7ddd55 (#20423) Co-authored-by: Juana De La Cuesta <juanita.delacuestamorales@hashicorp.com>

2024-04-11

driver/exec: allow using custom cgroups This commit implements the cgroup_override task config which allows using a custom cgroup with the following behaviors: - cgroup1 and cgroup2 are supported. - No resource limits are enforced. - Absolute paths such as `/custom` will be rooted in the cgroup filesystem. - All cgroup1 controllers will be created if they do not exist prior to the task launching, but no limits will be set. - The custom cgroup is removed when the task terminates.

2024-03-20

cgroup overrides for exec wip cgroup1 unimplemented cgroup2 working port of 8f1cc097c7e94863d4c0e7197eb01d4982f32474

2024-03-13

exec2: setup RPC plumbing for dynamic workload users (#20129) And pass the dynamic users pool from the client into the hook.

exec2: add client support for unveil filesystem isolation mode (#20115) * exec2: add client support for unveil filesystem isolation mode This PR adds support for a new filesystem isolation mode, "Unveil". The mode introduces a "alloc_mounts" directory where tasks have user-owned directory structure which are bind mounts into the real alloc directory structure. This enables a task driver to use landlock (and maybe the real unveil on openbsd one day) to isolate a task to the task owned directory structure, providing sandboxing. * actually create alloc-mounts-dir directory * fix doc strings about alloc mount dir paths

2024-03-11

backport of commit 5f5b34db0ea6e761b5c988a06923014b0cc4c308 (#20114) Co-authored-by: carrychair <137268757+carrychair@users.noreply.github.com>

backport of commit 5f5b34db0ea6e761b5c988a06923014b0cc4c308 (#20113) Co-authored-by: carrychair <137268757+carrychair@users.noreply.github.com>

backport of commit 5f5b34db0ea6e761b5c988a06923014b0cc4c308 (#20112) Co-authored-by: carrychair <137268757+carrychair@users.noreply.github.com>

remove repetitive words (#20110) Signed-off-by: carrychair <linghuchong404@gmail.com>

2024-02-05

Add granular control of SELinux labels for host mounts (#19839) Add new configuration option on task's volume_mounts, to give a fine grained control over SELinux "z" label * Update website/content/docs/job-specification/volume_mount.mdx Co-authored-by: Luiz Aoqui <luiz@hashicorp.com> * fix: typo * func: make volume mount verification happen even on mounts with no volume --------- Co-authored-by: Luiz Aoqui <luiz@hashicorp.com> Co-authored-by: Tim Gross <tgross@hashicorp.com>

2024-01-31

cgroup override

2024-01-29

tmp

2024-01-11

drivers/raw_exec: remove plumbing for ineffective no_cgroups configuration (#19599) * drivers/raw_exec: remove plumbing for ineffective no_cgroups configuration * fix tests

2024-01-03

backport of commit d75293d2ab33532a2c3fe62c8d8ae26cfee28df3 (#19595) Co-authored-by: Marvin Chin <cheesemuffin95@gmail.com>

Add OOM detection for exec driver (#19563) * Add OomKilled field to executor proto format * Teach linux executor to detect and report OOMs * Teach exec driver to propagate OOMKill information * Fix data race * use tail /dev/zero to create oom condition * use new test framework * minor tweaks to executor test * add cl entry * remove type conversion --------- Co-authored-by: Marvin Chin <marvinchin@users.noreply.github.com> Co-authored-by: Seth Hoenig <shoenig@duck.com>

2023-12-13

Update LICENSE, add MPL workaround, add headers

2023-09-18

drivers: plumb hardware topology via grpc into drivers (#18504) * drivers: plumb hardware topology via grpc into drivers This PR swaps out the temporary use of detecting system hardware manually in each driver for using the Client's detected topology by plumbing the data over gRPC. This ensures that Client configuration is taken to account consistently in all references to system topology. * cr: use enum instead of bool for core grade * cr: fix test slit tables to be possible

2023-08-10

Update copyright file headers to BUSL-1.1

2023-07-10

Include parent job ID as a Docker container label (#17843) Fixes: #17751

2023-05-16

drivers: make internal `DisableLogCollection` capability public (#17196) The `DisableLogCollection` capability was introduced as an experimental interface for the Docker driver in 0.10.4. The interface has been stable and allowing third-party task drivers the same capability would be useful for those drivers that don't need the additional overhead of logmon. This PR only makes the capability public. It doesn't yet add it to the configuration options for the other internal drivers. Fixes: #14636 #15686

2023-05-04

backport of commit ee1b3875ed8bcc2cab8213233c745730aa842153

2023-04-10

[COMPLIANCE] Add Copyright and License Headers

2023-03-13

Backport of plugin: add missing fields to `TaskConfig` into release/1.5.x (#16466) This pull request was automerged via backport-assistant

Backport of plugin: add missing fields to `TaskConfig` into release/1.4.x (#16465) This pull request was automerged via backport-assistant

Backport of plugin: add missing fields to `TaskConfig` into release/1.3.x (#16464) This pull request was automerged via backport-assistant

plugin: add missing fields to `TaskConfig` (#16434)

2023-01-31

renamed stanza to block for consistency with other projects (#15941) (#15953) Co-authored-by: Piotr Kazmierczak <470696+pkazmierczak@users.noreply.github.com>

Backport of renamed "stanza" to "block" for consistency with other projects into release/1.3.x (#15952) Co-authored-by: Piotr Kazmierczak <470696+pkazmierczak@users.noreply.github.com>

Backport of renamed "stanza" to "block" for consistency with other projects into release/1.2.x (#15951) Co-authored-by: Piotr Kazmierczak <470696+pkazmierczak@users.noreply.github.com>

2023-01-30

renamed stanza to block for consistency with other projects (#15941)

2022-11-17

Backport of Add mount propagation to protobuf definition of mounts into release/1.4.x (#15302) This pull request was automerged via backport-assistant

Backport of Add mount propagation to protobuf definition of mounts into release/1.2.x (#15301) This pull request was automerged via backport-assistant

Backport of Add mount propagation to protobuf definition of mounts into release/1.3.x (#15300) This pull request was automerged via backport-assistant

Add mount propagation to protobuf definition of mounts (#15096) * Add mount propagation to protobuf definition of mounts * Fix formatting * Add mount propagation to the simple roundtrip test. * changelog: add entry for #15096 Co-authored-by: Luiz Aoqui <luiz@hashicorp.com>

2022-10-14

[wip] logging plugins

2021-09-20

allow configuration of Docker hostnames in bridge mode (#11173) Add a new hostname string parameter to the network block which allows operators to specify the hostname of the network namespace. Changing this causes a destructive update to the allocation and it is omitted if empty from API responses. This parameter also supports interpolation. In order to have a hostname passed as a configuration param when creating an allocation network, the CreateNetwork func of the DriverNetworkManager interface needs to be updated. In order to minimize the disruption of future changes, rather than add another string func arg, the function now accepts a request struct along with the allocID param. The struct has the hostname as a field. The in-tree implementations of DriverNetworkManager.CreateNetwork have been modified to account for the function signature change. In updating for the change, the enhancement of adding hostnames to network namespaces has also been added to the Docker driver, whilst the default Linux manager does not current implement it.

2021-09-16

allow configuration of Docker hostnames in bridge mode (#11173) Add a new hostname string parameter to the network block which allows operators to specify the hostname of the network namespace. Changing this causes a destructive update to the allocation and it is omitted if empty from API responses. This parameter also supports interpolation. In order to have a hostname passed as a configuration param when creating an allocation network, the CreateNetwork func of the DriverNetworkManager interface needs to be updated. In order to minimize the disruption of future changes, rather than add another string func arg, the function now accepts a request struct along with the allocID param. The struct has the hostname as a field. The in-tree implementations of DriverNetworkManager.CreateNetwork have been modified to account for the function signature change. In updating for the change, the enhancement of adding hostnames to network namespaces has also been added to the Docker driver, whilst the default Linux manager does not current implement it.

2021-07-14

build: no longer use vendor directory This PR removes the vendor directory from the Nomad repository. Contributers will no longer need to deal with our `make sync` step when working on Nomad, which was suprising when making changes to the api. It also causes huge diffs in PRs that nobody looks at.

2021-07-07

Add HTTP API v1 openapi spec

2021-06-24

wip: opentelemetry poc

2021-06-16

docker: generate /etc/hosts file for bridge network mode (#10766) When `network.mode = "bridge"`, we create a pause container in Docker with no networking so that we have a process to hold the network namespace we create in Nomad. The default `/etc/hosts` file of that pause container is then used for all the Docker tasks that share that network namespace. Some applications rely on this file being populated. This changeset generates a `/etc/hosts` file and bind-mounts it to the container when Nomad owns the network, so that the container's hostname has an IP in the file as expected. The hosts file will include the entries added by the Docker driver's `extra_hosts` field. In this changeset, only the Docker task driver will take advantage of this option, as the `exec`/`java` drivers currently copy the host's `/etc/hosts` file and this can't be changed without breaking backwards compatibility. But the fields are available in the task driver protobuf for community task drivers to use if they'd like.

2021-06-07

Update runc and protobuf dependency

2021-05-17

Merge pull request #10600 from hashicorp/f-exec-allow_caps drivers/exec+java: reduce default set of linux capabilities

drivers/exec: enable setting allow_caps on exec driver This PR enables setting allow_caps on the exec driver plugin configuration, as well as cap_add and cap_drop in exec task configuration. These options replicate the functionality already present in the docker task driver. Important: this change also reduces the default set of capabilities enabled by the exec driver to match the default set enabled by the docker driver. Until v1.0.5 the exec task driver would enable all capabilities supported by the operating system. v1.0.5 removed NET_RAW from that list of default capabilities, but left may others which could potentially also be leveraged by compromised tasks. Important: the "root" user is still special cased when used with the exec driver. Older versions of Nomad enabled enabled all capabilities supported by the operating system for tasks set with the root user. To maintain compatibility with existing clusters we continue supporting this "feature", however we maintain support for the legacy set of capabilities rather than enabling all capabilities now supported on modern operating systems.

drivers/exec: pass capabilities through executor RPC Add capabilities to the LaunchRequest proto so that the capabilities set actually gets plumbed all the way through to task launch.

2021-04-27

core: propagate remote task handles Add a new driver capability: RemoteTasks. When a task is run by a driver with RemoteTasks set, its TaskHandle will be propagated to the server in its allocation's TaskState. If the task is replaced due to a down node or draining, its TaskHandle will be propagated to its replacement allocation. This allows tasks to be scheduled in remote systems whose lifecycles are disconnected from the Nomad node's lifecycle. See https://github.com/hashicorp/nomad-driver-ecs for an example ECS remote task driver.

2021-04-16

plugins/drivers: fix deprecated fields

2021-04-15

plugins/driver: add cpuset_cpus back and mark cpuset_mems as reserved

executor: add support for cpuset cgroup

2021-04-13

executor: add support for cpuset cgroup

2021-04-12

deps: upgrade go-plugin dependency This PR brings go-plugin up to date from ~v1.0.1 to v1.4.0 (about 18 months) with a grab-bag of bux fixes.

2021-03-30

oversubscription: driver/exec to honor MemoryMaxMB

2021-03-26

pipe memoryMaxMB through grpc to drivers/executors

2021-02-05

drivers/exec+java: Add configuration to restore previous PID/IPC namespace behavior. This PR adds default_pid_mode and default_ipc_mode options to the exec and java task drivers. By default these will default to "private" mode, enabling PID and IPC isolation for tasks. Setting them to "host" mode disables isolation. Doing so is not recommended, but may be necessary to support legacy job configurations. Closes #9969

2020-11-25

use comment ignores (#9448) Use targetted ignore comments for the cases where we are bound by backward compatibility. I've left some file based linters, especially when the file is riddled with linter voilations (e.g. enum names), or if it's a property of the file (e.g. package and file names). I encountered an odd behavior related to RPC_REQUEST_RESPONSE_UNIQUE and RPC_REQUEST_STANDARD_NAME. Apparently, if they target a `stream` type, we must separate them into separate lines so that the ignore comment targets the type specifically.

2020-11-19

Switch event payload to oneof

Merge remote-tracking branch 'origin/master' into event/proto-service

copy paste some waypoint glint structs/interfaces/pkgs

2020-11-10

protos: Update .proto files not to use Go package name (#9301) Previously, it was required that you `go get github.com/hashicorp/nomad` to be able to build protos, as the protoc invocation added an include directive that pointed to `$GOPATH/src`, which is how dependent protos were discovered. As Nomad now uses Go modules, it won't necessarily be cloned to `$GOPATH`. (Additionally, if you _had_ go-gotten Nomad at some point, protoc compilation would have possibly used the _wrong_ protos, as those wouldn't necessarily be the most up-to-date ones.) This change modifies the proto files and the `protoc` invocation to handle discovering dependent protos via protoc plugin modifier statements that are specific to the protoc plugin being used. In this change, `make proto` was run to recompile the protos, which results in changes only to the gzipped `FileDescriptorProto`.

Flesh out TestSinkServer_Subscribe The test passes, but it presently emits a listener error when the test ends.

2020-11-09

exploratory work to add grpc endpoint for streaming events

add proto file for subscribing and streaming events

2020-08-24

consul/connect: update consul to match current

2020-08-23

vendor: consul v1.7.7 Signed-off-by: Yoan Blanc <yoan@dosimple.ch>

2020-08-11

docker: support group allocated ports and host_networks (#8623) * docker: support group allocated ports * docker: add new ports driver config to specify which group ports are mapped * docker: update port mapping docs

2020-06-18

Task DNS Options (#7661) Co-Authored-By: Tim Gross <tgross@hashicorp.com> Co-Authored-By: Seth Hoenig <shoenig@hashicorp.com>

2020-06-02

deps: Switch to Go modules for dependency management This PR switches the Nomad repository from using govendor to Go modules for managing dependencies. Aspects of the Nomad workflow remain pretty much the same. The usual Makefile targets should continue to work as they always did. The API submodule simply defers to the parent Nomad version on the repository, keeping the semantics of API versioning that currently exists.

2020-05-27

volumes: return better error messages for unsupported task drivers (#8030) When an allocation runs for a task driver that can't support volume mounts, the mounting will fail in a way that can be hard to understand. With host volumes this usually means failing silently, whereas with CSI the operator gets inscrutable internals exposed in the `nomad alloc status`. This changeset adds a MountConfig field to the task driver Capabilities response. We validate this when the `csi_hook` or `volume_hook` fires and return a user-friendly error. Note that we don't currently have a way to get driver capabilities up to the server, except through attributes. Validating this when the user initially submits the jobspec would be even better than what we're doing here (and could be useful for all our other capabilities), but that's out of scope for this changeset. Also note that the MountConfig enum starts with "supports all" in order to support community plugins in a backwards compatible way, rather than cutting them off from volume mounting unexpectedly.

2020-05-21

volumes: return better error messages for unsupported task drivers (#8030) When an allocation runs for a task driver that can't support volume mounts, the mounting will fail in a way that can be hard to understand. With host volumes this usually means failing silently, whereas with CSI the operator gets inscrutable internals exposed in the `nomad alloc status`. This changeset adds a MountConfig field to the task driver Capabilities response. We validate this when the `csi_hook` or `volume_hook` fires and return a user-friendly error. Note that we don't currently have a way to get driver capabilities up to the server, except through attributes. Validating this when the user initially submits the jobspec would be even better than what we're doing here (and could be useful for all our other capabilities), but that's out of scope for this changeset. Also note that the MountConfig enum starts with "supports all" in order to support community plugins in a backwards compatible way, rather than cutting them off from volume mounting unexpectedly.

2020-04-20

fix spelling errors (#6985)

2020-04-09

remove vendor

2020-03-23

remove unused packages

2020-03-14

fixup! vendor: vendor.json cleanup Signed-off-by: Yoan Blanc <yoan@dosimple.ch>

2020-03-03

update grpc Upgrade grpc to v1.27.1 and protobuf plugins to v1.3.4.

2020-02-18

client: support no_pivot_root in exec driver configuration

2020-02-17

vendor: golang/protobuf 1.1.0 -> 1.3.2

2020-02-13

Update github.com/shirou/gopsutil and docker libraries Update github.com/shirou/gopsutil and docker related libraries. I had to ugprade docker, because gopsutil required upgrades to shared libraries (golang.org/x/sys/..) that broke our old vendoring of docker libraries.

2020-02-07

vendor: golang/protobuf 1.1.0 -> 1.3.2

2019-12-03

vendor: Update go-discover for AWS SDK change Update go-discover to pull in latest AWS SDK change through its vendored path, along with other provider changes.

2019-11-12

nomad exec: Allow customizing environment variables When execing into a task, it's useful to be able to set some special environment variables, namely `TERM` and language environment variable. In this PR, we expose `env` parameter that takes an array of `KEY=VALUE` values that gets thread all the way through to executors exec handling. The main purpose is to allow configuring `TERM` for purposes of web ui, as the appropriate value would depend on the terminal library and not necessarily user friendly. CLI ---------- This PR doesn't expose this configuration to CLI due to some complexities: * It's not as valuable to set environment variables there (i.e. users can export their environment variables before running commands) * The `-e` flag is already used for escape character, and I'm unsure if how to best proceed; I can see the appeal of repurposing `-e` to env-variables as it's the most common thing * We use the standard library `flags` package that doesn't support string array flag to allow for multiple environment variables nicely. As such, I've punted on supporting the CLI for another time as the need arises. External drivers and compatibility ---------- This PR makes some changes to the exec endpoint in drivers. These changes are binary backward compatible: existing drivers continue to run against latest nomad binary as the grpc handler ignores the new `env` field. The drivers continue to compile as normal against the version they vendored or depended on. Though, when plugin authors upgrade the dependency they will need to update code to either handle the `env` field or explicitly ignore it.

2019-10-04

upgrade go-plugin to latest, 8091134 Upgrade go-plugin to latest to pick up Windows fix in https://github.com/hashicorp/go-plugin/pull/125 .

2019-09-25

driver/networking: don't recreate existing network namespaces

2019-08-26

govendor fetch github.com/hashicorp/go-getter@f5101da, protobuf 1.2

2019-07-31

Driver networking support Adds support for passing network isolation config into drivers and implements support in the rawexec driver as a proof of concept

ar: fix lint errors

Add network lifecycle management Adds a new Prerun and Postrun hooks to manage set up of network namespaces on linux. Work still needs to be done to make the code platform agnostic and support Docker style network initalization.

2019-07-08

Driver networking support Adds support for passing network isolation config into drivers and implements support in the rawexec driver as a proof of concept

ar: fix lint errors

Add network lifecycle management Adds a new Prerun and Postrun hooks to manage set up of network namespaces on linux. Work still needs to be done to make the code platform agnostic and support Docker style network initalization.

2019-07-01

mend

2019-06-17

Driver networking support Adds support for passing network isolation config into drivers and implements support in the rawexec driver as a proof of concept

ar: fix lint errors