Proto commits in ten-nancy/porto

These 43 commits are when the Protocol Buffers files have changed:

Commit:2d42c95
Author:dmitrygx
Committer:dmitrygx

remove Network IfUp script invocation from Network Watchdog commit_hash:e14a6e2bdcfd9c1b84e128f758a460139c1141bc

The documentation is generated from this commit.

Commit:d1fbd00
Author:alexperevalov
Committer:alexperevalov

portod: docker_allowed_registry docker\_allowed\_registry now supports wildcards, like "\*" - means any registries are allowed \*.io - means all registries in domain io exact names also supported docker\_images\_support are preserved, since it used in docker-images, docker-rmi, docker-run commit_hash:375136ee7cc3ee2d762ec2e7d4f798894f0f16c7

Commit:659a249
Author:ovov
Committer:ovov

rpc, config: remove autostart_pattern commit_hash:9ecda21472fba9f37139bc5454585f167bd83499

Commit:53b0180
Author:konstantimp
Committer:konstantimp

portoctl: remove unsupported cpu_set params from help message commit_hash:6279cc1f5b5bf4cc33ff232b32afe913d724fc3c

Commit:15e6675
Author:ovov
Committer:ovov

rpc, config: add autostart_pattern Some containers created via CreateFromSpec should be force-started automatically. Add a configuration option, autostart_pattern, which defines a name pattern used to identify containers that must be started on launch, regardless of their original creation settings. commit_hash:aba40e88a141ce797f197728a4e42ad3ef653381

Commit:943837a
Author:ovov
Committer:ovov

portod: add timeout to graceful shutdown Start shutdown after a specified timeout, even if some requests are still active. commit_hash:cef9fe3c5781167a45199c811e4cfc337e07bda5

Commit:d71a349
Author:warwish
Committer:warwish

container: add implicit security session tracking commit_hash:9e9932da3a04f44e026c13d699ffb773066e5e0b

Commit:c493426
Author:dschervov
Committer:dschervov

container, network: ability to inherit network spec If our spec states that ipvlan must have: MTU/ECN/autoconf configuration parameters, then the current container will have it, but its children will not. We want child containers to be able to inherit the network properties of their parent. To enable this, we introduce a new Porto configuration parameter. It is disabled by default, so nothing will change unless this feature is explicitly turned on. commit_hash:8b13df407c460d85224f1ed21d09c3f0073e49be

Commit:1e827db
Author:konstantimp
Committer:konstantimp

rpc: support net_limit_soft in CreateFromSpec request feat: support net_limit_soft in CreateFromSpec request commit_hash:b44c9a25aa65c7274d9b941febbd6962ede8b941

Commit:c819b68
Author:ovov
Committer:ovov

portod: add prometheus metrics Report portod metrics in prometheus format via http server listening on unix socket. commit_hash:64f8e901004b5036f00ccc2d38f64c193a897582

Commit:1a00bb0
Author:warwish
Committer:warwish

porto: allow to run portod inside porto container * cgroup: parametrize cgroupfs root via config * master: allow running in container commit_hash:4600449f812662d42d55c0e18bdfe9ffca716649

Commit:e0d590d
Author:dschervov
Committer:dschervov

reduce cgroup v2 controllers quantity As for now, use of cgroup v2 is decreasing porto-api perfomance in YT enviroment. This is happening because there are too much controllers. In this PR we want to decrease their quantity by delete redundant sub-cgroup LEAF when there are no need fro nested virtualization. commit_hash:3ea54d2301ad39378b3ecde1b959770538e2a93b

Commit:135be42
Author:alexperevalov
Committer:alexperevalov

Insecure registries support in portod This PR introduces porto config option ``` daemon { docker_insecure_registry : hostname } ``` where docker\_insecure\_registry is a list of docker registries, to which portod will connect through HTTP instead of HTTPS. And implementation to support it. #### </section> commit_hash:1b35cf427406d02d208eecbc0357a4c2b81bc2bc

Commit:ad49591
Author:alexperevalov
Committer:alexperevalov

Target Platform for docker-pull This PR ads the same option as docker pull --platform and sets default arch (ISA) as portod binary Platform as an option is necessary for CI/CD, e.g. <http://test-docker-images.py> uses predefined for x86\_64 image id, but being running on arm64 these ID would be different. So either keep list for platform for such test or use target platform option in porto API. Also it could be useful for preparing images for x86\_64 servers on ARM64 workstations. Without proper default value, simple portoctl docker-pull on ARM64 doesn't work, it downloads image for x86\_64 instead of ARM64. <section id="quibbler-autodescription"> #### Add support for Docker image platform selection in `docker-pull` ๐Ÿ“ - โœ… Added support for specifying a target platform (e.g., `amd64`, `arm64`) when pulling Docker images. - ๐Ÿ”„ Updated the `PullDockerImage` API to accept a platform parameter, allowing users to select the desired architecture. - ๐Ÿงฉ Updated the `docker-pull` command to include a `-T` option for defining the target platform, useful for multi-architecture environments or CI/CD workflows. - ๐Ÿงช Updated unit tests in Python and Go to validate new platform functionality and ensure backward compatibility. - ๐Ÿ”— Updated related components and libraries to reflect new API signature and logic. --- **You can rate the quality of the description [here](https://forms.yandex-team.ru/ext/surveys/13760736)** <a href="https://nda.ya.ru/t/qa0kX64r7DqvtN"><font size="2">Autodescription by Code Assistant</font></a> </section> commit_hash:d34497ea225624acbee3905f970c2f31ec7b482e

Commit:6b0fd1d
Author:dmitrygx
Committer:dmitrygx

launch NetwrokIfUpScript inside network watchdog and turn on for YATI RoCE hosts commit_hash:250b77a7de886d3fe359bfdb86921b2592422352

Commit:dcb405d
Author:ovov
Committer:ovov

container: refactor cpu limit/guarantee * Drop legacy features and configs: cfs_reserve, guarantee propogation from children to parent, cpu_limit_total. * Fix cpu limit/guarantee propogation from parent to children. commit_hash:d77acf5774e1e231012b8858bb48a7efb958bcaa

Commit:7101e5c
Author:ovov
Committer:ovov

container: remove place_usage/place_limit Place accounting is not used, so remove it commit_hash:663cb7b20dd97595561d36acaed2c0f12045eaa5

Commit:425e62f
Author:ovov
Committer:ovov

rpc, storage: add CleanupPlace call Add CleanupPlace call that cleanups all stale storages/volume/layers in specified place. This can be useful to call after hard poweroff. commit_hash:2f284c66f12731baae76f0523b07775202ebd449

Commit:6864fbf
Author:ovov
Committer:ovov

task: remove ptrace_on_start implementation It is obsolete feature and is not used. commit_hash:c9ea953e1dfa5b0e370fcdb6b041197f8b359900

Commit:ca0011b
Author:kndrvt
Committer:kndrvt

porto: remove virt_mode=docker commit_hash:6fbc8c01b2e4f54a40124ade76fca0aa19a46474

Commit:66fac7e
Author:ovov
Committer:ovov

devices: rework inheritance semantics Add devices_explicit flag, which controls how devices are inherited: * devices_explicit=false โ€” user specified devices applied as "merge" to parent devices. * devices_explicit=true โ€” user specified devices applied "as is". Also fix bug with stale devices cleanup & refactor devices rule application. commit_hash:55230d27d68a13aa7b6b2c0f0b245cbae747be47

Commit:f8f513f
Author:ovov
Committer:ovov

container: add config option to restrict starting with stopped parent Starting all ancestor containers on self start is not robust behavior as it can lead to race. When some process stops hierarchy of containers, other process can perform start of some container in this hierarchy. As a result part of the hierarchy would not be stopped. 254272d808c8e5786ca8c4735c8762a09aadd89f

Commit:f56cb37
Author:kndrvt
Committer:kndrvt

capabilities: add action in spec to modify caps partially 1a78c347a85c507f737ad9149bbacd5c18ff5f65

Commit:1bf0f6e
Author:ovov
Committer:ovov

container: use ct cpu_limit instead of host nproc to scale memory.high margin memory.high margin calculated as nproc * 64 * PAGE_SIZE (as in kernel) can be too big for containers with cpu_limit considerably less than nproc. Using this margin can lead to such containers hanging in memory reclaim instead of oom kill. Use cpu_limit * 64 * PAGE_SIZE insead. 0ba18579d18cb43dc976a412f7c7a0d32fe217ef

Commit:ca74b9b
Author:kndrvt
Committer:kndrvt

task: add info about session to container process 9009cf6896dab7a02b499e0d2d550f0716d86957

Commit:606136c
Author:ivan42
Committer:ivan42

netlimit_soft: show stats in a `net_limit_soft_stat` container property a4b64694cc472e06a74429568ae153b1f477f838

Commit:28d880a
Author:ovov
Committer:ovov

seccomp: add seccomp profiles Add support for setting seccomp rules/profile for containers. Container can have either specific set of rules (format described in seccomp.proto) or predefined (in config) profile. Any branch of containers can have at most one container with seccomp rules/profile for now. All descendant containers reuse seccomp rules/profile of first ancestor with seccommp rules/profile set. 45860feec409b2545a6c2a7eeeb4ce40c23cbac0

Commit:23dce62
Author:ivan42
Committer:ivan42

network: add support of configurable bpf clsact filter program for container interfaces * Adds net_limit_soft property to a container, that can be changed at runtime. * Adds support for `network { network_limit_soft_bpf_elf_path: "/path/to/bpf-elf.o" }` option to portod.conf. * Adds a configured bpf program to container interfaces. * Adds setting up of a bpf map named "netlimit_soft_map" and writing net_limit_soft property of a container to a network inode key in the map. 02c107343d3e4f0d09cec3141945b4303bcef2e8

Commit:9b432c6
Author:kndrvt
Committer:kndrvt

container: terminate fuse connections and mounts in virt_mode=fuse 5af1be76a7900a8f13a1a535799f04a58fac29a0

Commit:b40e4e2
Author:kndrvt
Committer:kndrvt

config: add enable_nbd option to disable nbd volume backend 162e597b99a993eaefb2244fcc6d7dfdaa147cf7

Commit:7243586
Author:ovov
Committer:ovov

volume: add fs_type property Parametrize file system type of volumes with loop/nbd backend: portoctl vcreate -A backend=loop storage=/path/to/image fs_type=ext4 portoctl vcreate -A backend=loop storage=/path/to/image fs_type=squashfs portoctl vcreate -A backend=loop storage=/path/to/image fs_type=erofs portoctl vcreate -A backend=loop storage=unix+tcp:/path/to/socket&export=foobar fs_type=erofs

Commit:61b4fe4
Author:warwish
Committer:warwish

property: add net_limit_bound and net_rx_limit_bound props New container properties net_limit_bound and net_rx_limit_bound added. Properties propagated from nearest network-isolated parents net_limit and net_rx_limit.

Commit:ecedcfa
Author:ovov
Committer:ovov

nbd: add error codes specific to nbd/squash/loop backends Add following codes for better error handling on client side: InvalidFilesystem : nbd/loop device does not contain valid filesystem NbdSocketTimeout : negotiation with nbd server timed out NbdSocketUnavaliable : failed to connect to nbd server NbdSocketError : other nbd connection error NbdProtoError : invalid answer from nbd server

Commit:04036d5
Author:dmitrygx
Committer:dmitrygx

network: support applying mac-/ip-vlan settings from parent to child containers

Commit:60132ea
Author:ovov
Committer:ovov

volume: cleanup optimizations storage: move storage cleanup to background thread Move all storage/layers/volumes cleanups to AsyncRemoveWatchDog. util/path: move RemoveRecursive and ClearRecursive to TPath methods * replace external call to find with nftw in ClearRecursive * move RemoveRecursive and ClearRecursive to TPath::RemoveAll and TPath::ClearDirectory respectively util/path: replace remove with rmdir/unlink in nftw callbacks volume: remove Path after internal storage: check place once storage: add support for cleanup interrupt

Commit:1c05dc3
Author:conquistador
Committer:conquistador

porto: export hfsc burst_duration to config porto: export hfsc burst_duration to config

Commit:e5a2239
Author:flown4qqqq
Committer:flown4qqqq

rpc: add pattern to path in vlist Author: flown4qqqq Branch: users/flown4qqqq/ticket-porto-724

Commit:4a94d88
Author:dimdim11
Committer:dimdim11

cgroup: enable SHED_IDLE for cgroups if cpu.idle presents cgroup: enable SHED_IDLE for cgroups if cpu.idle presents

Commit:669f6b4
Author:dimdim11
Committer:dimdim11

property: add io guarantee properties Add io guarantee properties

Commit:20f393e
Author:kndrvt

storage: add async removing of storages and it can be set via config excluding docker layers

Commit:a4512ec
Author:ovov

property: add cpu, memory and io pressure stall information https://docs.kernel.org/accounting/psi.html Signed-off-by: Alexander Ovechkin <ovov@yandex-team.ru> Link: https://st.yandex-team.ru/

Commit:aab0fc4
Author:mskhristenko

Coredump_filter add property: coredump_filter

Commit:1e67ca1
Author:Dmitry Yakunin

Initial OpenSource export