Proto commits in Snowflake-Labs/sansshell

These commits are when the Protocol Buffers files have changed: (only the last 100 relevant commits are shown)

2025-05-09

Remove excessive printing

Implement redacting of certain fields for MPA

Merge main

2025-04-28

Remove defunct FDB emergency move support

2025-04-25

Remove emergency tenant move

2025-04-23

Add DNS lookup latency to the response (#559)

The documentation is generated from this commit.

2025-04-17

SNOW-1991908: test(fdb) add fdbbackup v2 and add proto files

2025-04-15

SNOW-1991908: test(fdb) add fdbbackup v2

2025-04-01

MPA method-wide mode for sansshell

2025-03-24

Add authz policy dry run (#543)

2025-02-12

Add API versioning and localfile parallel read with grep (#526)

2025-01-29

Add new fdbcli command "configure blob_granules_enabled=1" (#522)

2024-12-19

Configurable timeout for dmesg read (#518)

2024-12-16

Added back-compatibility fix

2024-12-13

Support for multiple args in rm and rmdir, -r flag for rmdir

2024-12-02

Add File Shred command (#513)

2024-11-28

Add option to overwrite dial address (#515) * Add option to overwrite dial address * Add readme and update client code * Update utils interface

2024-11-26

Add readme and update client code

Add option to overwrite dial address

2024-11-22

Add way to specify user for exec command. (#511) * Add way to specify user for exec command. * Respect user for streaming version of exec. * Fix exec uid vs gid * Add README.md

2024-10-10

Add command to get and set data in files of specific formats (#492) * SNOW-1665681 Add set/get operators

2024-08-28

Expose raw cert info through tlsinfo (#479) * Expose raw cert info through tlsinfo In some types of debugging, it's useful to get a more detailed view of a certificate than what `tlsinfo get-cert` currently provides. The existing fields that we parse out are the most commonly useful ones so they're good to show by default, but I'd like it to be possible to display more. My solution is to pass back the full raw certificate and allow printing it if a flag is set. We transfer the cert over the wire in binary ASN.1 DER format because it's more compact and it's easy to get from the libraries we're using, and we print it in PEM to avoid issues with unprintable characters and to keep it in a format that many CLI tools understand. I confirmed that generated PEMs were readable with `keytool -printcert -file` and `openssl x509 -in`. Example commands: ``` $ go run ./cmd/sanssh -targets localhost tlsinfo get-certs www.example.com:443 ---Server Certificate--- 0 Issuer: CN=DigiCert Global G2 TLS RSA SHA256 2020 CA1,O=DigiCert Inc,C=US Subject: CN=www.example.org,O=Internet Corporation for Assigned Names and Numbers,L=Los Angeles,ST=California,C=US NotBefore: 2024-01-29 16:00:00 -0800 PST NotAfter: 2025-03-01 15:59:59 -0800 PST DNS Names: [www.example.org example.net example.edu example.com example.org www.example.com www.example.edu www.example.net] IP Addresses: [] ---Server Certificate--- 1 Issuer: CN=DigiCert Global Root G2,OU=www.digicert.com,O=DigiCert Inc,C=US Subject: CN=DigiCert Global G2 TLS RSA SHA256 2020 CA1,O=DigiCert Inc,C=US NotBefore: 2021-03-29 17:00:00 -0700 PDT NotAfter: 2031-03-29 16:59:59 -0700 PDT DNS Names: [] IP Addresses: [] $ go run ./cmd/sanssh -targets localhost tlsinfo get-certs -pem www.example.com:443 -----BEGIN CERTIFICATE----- MIIHbjCCBlagAwIBAgIQB1vO8waJyK3fE+Ua9K/hhzANBgkqhkiG9w0BAQsFADBZ MQswCQYDVQQGEwJVUzEVMBMGA1UEChMMRGlnaUNlcnQgSW5jMTMwMQYDVQQDEypE aWdpQ2VydCBHbG9iYWwgRzIgVExTIFJTQSBTSEEyNTYgMjAyMCBDQTEwHhcNMjQw MTMwMDAwMDAwWhcNMjUwMzAxMjM1OTU5WjCBljELMAkGA1UEBhMCVVMxEzARBgNV BAgTCkNhbGlmb3JuaWExFDASBgNVBAcTC0xvcyBBbmdlbGVzMUIwQAYDVQQKDDlJ bnRlcm5ldMKgQ29ycG9yYXRpb27CoGZvcsKgQXNzaWduZWTCoE5hbWVzwqBhbmTC oE51bWJlcnMxGDAWBgNVBAMTD3d3dy5leGFtcGxlLm9yZzCCASIwDQYJKoZIhvcN AQEBBQADggEPADCCAQoCggEBAIaFD7sO+cpf2fXgCjIsM9mqDgcpqC8IrXi9wga/ 9y0rpqcnPVOmTMNLsid3INbBVEm4CNr5cKlh9rJJnWlX2vttJDRyLkfwBD+dsVvi vGYxWTLmqX6/1LDUZPVrynv/cltemtg/1Aay88jcj2ZaRoRmqBgVeacIzgU8+zmJ 7236TnFSe7fkoKSclsBhPaQKcE3Djs1uszJs8sdECQTdoFX9I6UgeLKFXtg7rRf/ hcW5dI0zubhXbrW8aWXbCzySVZn0c7RkJMpnTCiZzNxnPXnHFpwr5quqqjVyN/aB KkjoP04Zmr+eRqoyk/+lslq0sS8eaYSSHbC5ja/yMWyVhvMCAwEAAaOCA/IwggPu MB8GA1UdIwQYMBaAFHSFgMBmx9833s+9KTeqAx2+7c0XMB0GA1UdDgQWBBRM/tAS TS4hz2v68vK4TEkCHTGRijCBgQYDVR0RBHoweIIPd3d3LmV4YW1wbGUub3Jnggtl eGFtcGxlLm5ldIILZXhhbXBsZS5lZHWCC2V4YW1wbGUuY29tggtleGFtcGxlLm9y Z4IPd3d3LmV4YW1wbGUuY29tgg93d3cuZXhhbXBsZS5lZHWCD3d3dy5leGFtcGxl Lm5ldDA+BgNVHSAENzA1MDMGBmeBDAECAjApMCcGCCsGAQUFBwIBFhtodHRwOi8v d3d3LmRpZ2ljZXJ0LmNvbS9DUFMwDgYDVR0PAQH/BAQDAgWgMB0GA1UdJQQWMBQG CCsGAQUFBwMBBggrBgEFBQcDAjCBnwYDVR0fBIGXMIGUMEigRqBEhkJodHRwOi8v Y3JsMy5kaWdpY2VydC5jb20vRGlnaUNlcnRHbG9iYWxHMlRMU1JTQVNIQTI1NjIw MjBDQTEtMS5jcmwwSKBGoESGQmh0dHA6Ly9jcmw0LmRpZ2ljZXJ0LmNvbS9EaWdp Q2VydEdsb2JhbEcyVExTUlNBU0hBMjU2MjAyMENBMS0xLmNybDCBhwYIKwYBBQUH AQEEezB5MCQGCCsGAQUFBzABhhhodHRwOi8vb2NzcC5kaWdpY2VydC5jb20wUQYI KwYBBQUHMAKGRWh0dHA6Ly9jYWNlcnRzLmRpZ2ljZXJ0LmNvbS9EaWdpQ2VydEds b2JhbEcyVExTUlNBU0hBMjU2MjAyMENBMS0xLmNydDAMBgNVHRMBAf8EAjAAMIIB fQYKKwYBBAHWeQIEAgSCAW0EggFpAWcAdABOdaMnXJoQwzhbbNTfP1LrHfDgjhuN acCx+mSxYpo53wAAAY1b0vxkAAAEAwBFMEMCH0BRCgxPbBBVxhcWZ26a8JCe83P1 JZ6wmv56GsVcyMACIDgpMbEo5HJITTRPnoyT4mG8cLrWjEvhchUdEcWUuk1TAHYA fVkeEuF4KnscYWd8Xv340IdcFKBOlZ65Ay/ZDowuebgAAAGNW9L8MAAABAMARzBF AiBdv5Z3pZFbfgoM3tGpCTM3ZxBMQsxBRSdTS6d8d2NAcwIhALLoCT9mTMN9OyFz IBV5MkXVLyuTf2OAzAOa7d8x2H6XAHcA5tIxY0B3jMEQQQbXcbnOwdJA9paEhvu6 hzId/R43jlAAAAGNW9L8XwAABAMASDBGAiEA4Koh/VizdQU1tjZ2E2VGgWSXXkwn QmiYhmAeKcVLHeACIQD7JIGFsdGol7kss2pe4lYrCgPVc+iGZkuqnj26hqhr0TAN BgkqhkiG9w0BAQsFAAOCAQEABOFuAj4N4yNG9OOWNQWTNSICC4Rd4nOG1HRP/Bsn rz7KrcPORtb6D+Jx+Q0amhO31QhIvVBYs14gY4Ypyj7MzHgm4VmPXcqLvEkxb2G9 Qv9hYuEiNSQmm1fr5QAN/0AzbEbCM3cImLJ69kP5bUjfv/76KB57is8tYf9sh5ik LGKauxCM/zRIcGa3bXLDafk5S2g5Vr2hs230d/NGW1wZrE+zdGuMxfGJzJP+DAFv iBfcQnFg4+1zMEKcqS87oniOyG+60RMM0MdejBD7AS43m9us96Gsun/4kufLQUTI FfnzxLutUV++3seshgefQOy5C/ayi8y1VTNmujPCxPCi6Q== -----END CERTIFICATE----- -----BEGIN CERTIFICATE----- MIIEyDCCA7CgAwIBAgIQDPW9BitWAvR6uFAsI8zwZjANBgkqhkiG9w0BAQsFADBh MQswCQYDVQQGEwJVUzEVMBMGA1UEChMMRGlnaUNlcnQgSW5jMRkwFwYDVQQLExB3 d3cuZGlnaWNlcnQuY29tMSAwHgYDVQQDExdEaWdpQ2VydCBHbG9iYWwgUm9vdCBH MjAeFw0yMTAzMzAwMDAwMDBaFw0zMTAzMjkyMzU5NTlaMFkxCzAJBgNVBAYTAlVT MRUwEwYDVQQKEwxEaWdpQ2VydCBJbmMxMzAxBgNVBAMTKkRpZ2lDZXJ0IEdsb2Jh bCBHMiBUTFMgUlNBIFNIQTI1NiAyMDIwIENBMTCCASIwDQYJKoZIhvcNAQEBBQAD ggEPADCCAQoCggEBAMz3EGJPprtjb+2QUlbFbSd7ehJWivH0+dbn4Y+9lavyYEEV cNsSAPonCrVXOFt9slGTcZUOakGUWzUb+nv6u8W+JDD+Vu/E832X4xT1FE3LpxDy FuqrIvAxIhFhaZAmunjZlx/jfWardUSVc8is/+9dCopZQ+GssjoP80j812s3wWPc 3kbW20X+fSP9kOhRBx5Ro1/tSUZUfyyIxfQTnJcVPAPooTncaQwywa8WV0yUR0J8 osicfebUTVSvQpmowQTCd5zWSOTOEeAqgJnwQ3DPP3Zr0UxJqyRewg2C/Uaoq2yT zGJSQnWS+Jr6Xl6ysGHlHx+5fwmY6D36g39HaaECAwEAAaOCAYIwggF+MBIGA1Ud EwEB/wQIMAYBAf8CAQAwHQYDVR0OBBYEFHSFgMBmx9833s+9KTeqAx2+7c0XMB8G A1UdIwQYMBaAFE4iVCAYlebjbuYP+vq5Eu0GF485MA4GA1UdDwEB/wQEAwIBhjAd BgNVHSUEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIwdgYIKwYBBQUHAQEEajBoMCQG CCsGAQUFBzABhhhodHRwOi8vb2NzcC5kaWdpY2VydC5jb20wQAYIKwYBBQUHMAKG NGh0dHA6Ly9jYWNlcnRzLmRpZ2ljZXJ0LmNvbS9EaWdpQ2VydEdsb2JhbFJvb3RH Mi5jcnQwQgYDVR0fBDswOTA3oDWgM4YxaHR0cDovL2NybDMuZGlnaWNlcnQuY29t L0RpZ2lDZXJ0R2xvYmFsUm9vdEcyLmNybDA9BgNVHSAENjA0MAsGCWCGSAGG/WwC ATAHBgVngQwBATAIBgZngQwBAgEwCAYGZ4EMAQICMAgGBmeBDAECAzANBgkqhkiG 9w0BAQsFAAOCAQEAkPFwyyiXaZd8dP3A+iZ7U6utzWX9upwGnIrXWkOH7U1MVl+t wcW1BSAuWdH/SvWgKtiwla3JLko716f2b4gp/DA/JIS7w7d7kwcsr4drdjPtAFVS slme5LnQ89/nD/7d+MS5EHKBCQRfz5eeLjJ1js+aWNJXMX43AYGyZm0pGrFmCW3R bpD0ufovARTFXFZkAdl9h6g4U5+LXUZtXMYnhIHUfoyMo5tS58aI7Dd8KvvwVVo4 chDYABPPTHPbqjc1qCmBaZx2vN4Ye5DUys/vZwP9BFohFrH/6j/f3IL16/RZkiMN JCqVJUzKoZHm1Lesh3Sz8W2jmdv51b2EQJ8HmA== -----END CERTIFICATE----- ``` * Update styling and add explicit error on blank cert

2024-08-27

SNOW-1571136 Add `network tcp-check` command (#472) * SNOW-1571136 Add `network tcp-check` command * SNOW-1571136 Add set up dev env to readme * SNOW-1571136 Add golangci to pre-commit

2024-07-23

Tweak debug_redact behavior (#460) * Tweak debug_redact behavior In sansshell we use redact to avoid logging sensitive info primarily for avoiding leaking secrets, not to provide privacy. I've switched our redaction from `--REDACTED--` to `REDACTED-<sha256hash>` so that it's possible to figure out if a field matches a known value while not being possible to learn the value purely from reading the field. The empty value is left unredacted because it should never be secret. I've also added support for bytes in addition to strings. Now that we have a more generous redaction, I've added it to a couple more fields that seem like they might accidentally contain secrets. * Add comment and more tests

2024-06-03

Add support for service action reload (#443) * Add support for service action reload Adds support for reloading a systemd/unknown service via an `ACTION_RELOAD` service action request. * Update ReloadRemoteService to support multiple targets Updates the `ReloadRemoteService` helper function to support multiple targets in its invocation. * Remove inaccurate comment on ReloadRemoteService This function now _does_ support multiple targets.

2024-05-06

Add timestamp option (recent time reach the status) to service status (#429) * Add timestamp option (recent time reach the status) to service status * address comments * fix typo

2024-05-03

Add timestamp option (recent time reach the status) to service status

2024-04-15

Add network capabilities: packet capture and listing interfaces Network service implements a basic packet capture functionality, supporting network inteface selection, packet truncation and server-side filtering. The main motivation for this capability is addressing a common anti-pattern of allowing users to `exec tcpdump`. Unrestricted use of tcpdump can led to privilege escalation. Implementing packet capture natively within SansShell mitigates that risk and enables creating more fine-grained policies that, for example, can limit the amount of data being captured. Additionally, support listing network interfaces to aid with the remote packet capture.

2024-02-09

add tcpoverrpc service

2024-01-10

Add repo and package name to remove command (#396) * add repo and package to remove * add test for remove. validate args

2023-11-03

Fix redact logic when recuring into list/map fields (#367)

2023-10-24

Add a MPA design and proto definition. (#350) * Add a MPA design and proto definition. I've prototyped this enough in https://github.com/stvnrhodes/sansshell/tree/mpa to have reasonable confidence that this design is implementable. I'm separating out the design and proto into an initial PR so that we can make sure that we agree on the high-level direction before we deeply review any implementation. The readme is meant to be used as long-term documentation on how to use MPA. Part of https://github.com/Snowflake-Labs/sansshell/issues/346 * Address feedback

2023-10-18

Add an option to skip tls verification in HTTPOverRPC action (#356)

2023-10-11

Redact http header values in logs (#343)

2023-09-21

Restore old proto tags (#327) In https://github.com/Snowflake-Labs/sansshell/pull/297, we changed a proto tag number from 19 to 20, breaking backwards compatibility. This restores the old tag number. I'm reserving tag 20 because I prefer never reusing fields that were used before, even briefly.

2023-09-01

Add generic knob parsing for fdbcli (#297) * add generic knob parsing for fdbcli * fix spacing and add to fdbcli_test.go

2023-08-30

Add a Sansshell service for fetching TLS server certificate details (#312)

2023-08-11

Add tenant emergency move to SansShell fdbcli modules (#285) * add tenant emergency move to fdbcli modules * add status command * partial implementation of FDB data move * fix incorrect copy/paste * add server for fdbdatamove * add numProcs argument * register package * WIP for env variables * WIP implementation of start and wait * look for error on cmd.Wait() * WIP implementation of streaming stdout/stderr * move mu unlock location and add link to script location * try to release mutex deliberately and fix string message * fix streaming stdout/stderr to client * add test file * adjust test case * wrap clear operations with mutex * defer unlock and reorder test to be easier to follow * avoid directly overwriting pipes * copy stdout/stderr before passing to anonymous goroutine * avoid duplicate error message * account for EOF test result in races * revert test change and use sync.WaitGroup * move Wait ahead of command Wait * address code review comments * update unit test * update module with finalized paths * add more log lines * add more logging * merge log lines

2023-08-10

Add journalctl action to sansshell with os/exec (#293) * Add journalctl action to sansshell with os/exec * remove commment out lines * address comments * define const and put in sysinfo.go file

2023-08-09

Update sansshell httpproxy to make it have ability to send http/https… (#290) * Update sansshell httpproxy to make it have ability to send http/https to specified remote host * validate port

2023-08-01

Protobuf reorder (#281) * reorder * adjust the output format of package list * Add unit test to newer version sanssh with older version sansshell in package list action --------- Co-authored-by: Edbert Linardi <edbert.linardi@snowflake.com>

2023-07-26

Add sysinfo dmesg action to Sansshell (#277) * Add sysinfo dmesg action to Sansshell * mock KmsgParser to pass integration test * nit: change name for req * change timestampe display format * add comment to getKernelMessages

2023-07-21

Add packages search by name (#271) * add packages search by name * make some adjustments(verify the cmdline) to test files * make adjustments to PackageInfo to make it fit with SearchReply * change package search proto define and address comments * change the way to extract EVR in package list output * change package search action proto definition * name change, add unit test for extractEVR

Add file mkdir to Sansshell (#275) * Add file mkdir to Sansshell * Address PR comments: name change, mkdir proto definition change * create directory directly without renaming * name fix

Add wait option to sanssh fdb fdbcli exclude action (#273) * Add wait option to sanssh fdb fdbcli exclude action * minor changes to fdbcli client

Add /Exec.Exec/StreamingRun (#274) * Add /Exec.Exec/StreamingRun `/Exec.Exec/StreamingRun` is a variant of `/Exec.Exec/Run` that streams back output as the command runs instead of collecting output and returning it at the end. Many commands like ping and tcpdump are commonly used in an interactive way where output is continually printed out and are badly supported by the existing Run call. Streaming the response allows continually printing out output during a call without widening attack surfaces. StreamingRun uses the exact same request and response as Run. The only difference is that the response is a stream that may be returned many times, each time with a portion of stdout and/or stderr. The final message in the response contains a nonzero exit code if needed. StreamingRun is a much better experience for anyone using `sanssh exec run`, but we don't want to break backwards compatibility or cause issues when preexisting policies don't allow the new calls or preexisting sansshell nodes haven't been built with the new feature. It can be enabled by passing in `--streaming` to the command or modifying the DefaultStreaming package-level variable in services/exec/client to change the defaults. * change flag name, remove comment left from testing

2023-07-12

Add a sansshell service for proxying HTTP requests. (#266) * Add a sansshell service for proxying HTTP requests. This adds a structured way to make an HTTP request to a port on localhost. The structure allows fine-grained access control on the http request. The httpoverrpc proxy subcommand launches a web server that translates http calls into grpc calls, creating a http->grpc->http flow. The webserver only supports one host at a time. The httpoverrpc get subcommand allows passing in exact arguments to call, giving a way to query the same port across many machines. A few design notes: - I've limited the rpc interface to localhost because sansshell is normally used for host-local actions and I don't have a clear use case for making this call cross-host. If we want to make this more general, I could rename things or we could add more RPCs. - I'm only supporting http calls. Supporting https might require some thought about how to handle certificates and server identity. - We don't automatically follow redirects because we expect users of the RPCs to do their own redirect-following logic. - Sufficiently-large http body sizes that are too big to fit into a single grpc message will fail. We could get around size limitations by streaming the response, but that adds complexity to the API. To test this out, try running the following with each command in a separate terminal. ``` python -m http.Server go run ./cmd/sansshell-server go run ./cmd/sanssh -targets localhost httpoverrpc proxy -addr localhost:8001 8000 curl localhost:8001 ``` * Fix lint errors * Address some review comments

2023-07-11

add fdbserver version lookup (#264) * add fdbserver version lookup * address comments: change version from flag to action & typo fix * set fdbserver binary as a flag for sansshell server * remove unnecessary lines for fdbserver command join

2023-06-23

add sysinfo service, uptime action (#257) * add sysinfo service, uptime action * fix: 1. use unix.Sysinfo() to fetch system uptime rather than read through /proc/uptime 2. make sysinfo proxy-able * fix: add blank line in rego file & change print format of uptime

2023-06-12

Implement power service (#253) * add: power service * doc: live in a current year * fix: example in the flag description * fix: cleanup client messages * add: power server to sansshell-server --------- Co-authored-by: Edbert Linardi <edbert.linardi@snowflake.com>

2023-06-07

add: RebootRequied field to packages.install reply (#251)

2023-05-24

Add remove method to package service (#248) * add: remove call to package service * fix: messages and spelling

2023-05-10

Fix `fdbcli fdb kill` (#238) Suppose we want to kill an FDB process. One can try to do it naively via sansshell in a following way but it would fail: ``` $ sanssh --targets=${IP?} fdb fdbcli kill 1.2.3.4:4500 fdbcli error: rpc error: code = Internal desc = error from running - <nil> stdout: stderr: ERROR: process `1.2.3.4:4500' not recognized. ``` This is because fdbcli requires kill to be done in two phases: first we get a list of processes by running kill without arguments which fetches list of processes and caches it in the client, so kill <process> works after that. But it’s not convenient to do this via sanshell as it’s not designed for interactive commands. We can do sanssh --targets=${IP?} fdb fdbcli --exec 'kill; kill [proc]' However any kind of --exec reduces our ability to have fine-graned ACLs and auditing. Sansshell client even documents that `fdbcli kill [proc]` will fetch process list first, however it is not implemented on the server side. This PR fixes it: * Make help of the fdbcli kill command shorter * Update the autogenerated protobuf files for FDB * Prepend kill all and kill <proc> with 'kill ; ' as CLI documentation suggests * Add -sleep flag to the fdbcli kill command * Add some of the cSpell false-positives to a local dictionary * Address Steven's comments - style improvements

2023-04-03

Improve symlink handling. (#211) Fixes https://github.com/Snowflake-Labs/sansshell/issues/210 I've made dealing with symlinks nicer through adding the following 1. Use lstat when listing files in directory 2. Use lstat by default for `file stat`, imitating the behavior of `stat` 3. Add a new readlink command so that we can read symbolic links 4. Add a new symlink command so that we can create symbolic links 5. Add -l and -d commands to `file ls` because my muscle memory keeps typing them Some manual testing on macos: ``` % ./sanssh --targets=localhost file ls -l /tmp/repro -rw-r--r-- - 501 0 0 Mon Apr 3 16:11:57 UTC 2023 /tmp/repro/a Lrwxr-xr-x - 501 0 12 Mon Apr 3 16:13:10 UTC 2023 /tmp/repro/b Lrwxr-xr-x - 501 0 6 Mon Apr 3 16:12:55 UTC 2023 /tmp/repro/c -rw-r--r-- - 501 0 0 Mon Apr 3 16:11:57 UTC 2023 /tmp/repro/d % ./sanssh --targets=localhost file readlink /tmp/repro/b /tmp/repro/d % ./sanssh --targets=localhost file readlink /tmp/repro/a readlink client error: rpc error: code = FailedPrecondition desc = file is not a symlink ```

2022-11-10

Adds convenience command to issue DNS queries (#175) Fix https://github.com/Snowflake-Labs/sansshell/issues/167

2022-10-18

Add package cleanup command. (#174) In some cases a YUM operation can leave things partially finished. i.e. doing a live update of the sansshell YUM package could result in a server restart before yum has completed. Allow this to fix via Cleanup which for YUM implements yum-complete-transaction --cleanup-only Add client support and tests

2022-10-14

Add service enable/disable support and tests (#171) * Add utils for all service permutations

2022-06-28

Cleanup client packages code where it could send bad data and cause a crash. (#144) Add disablerepo as an option to install/update as well. There are times you need to skip a broken repo definition. Regen all protoes

2022-06-16

Implement fdbcli missing commands (#137) * Add missing fdbcli commands (10) Realized that transaction isn't needed as it's own message. Instead we simply take one or more commands and then chain them together with exec. Implement begin and commit so transactions will work in this fashion. Fix kill (it has options I missed). Add tests for everything. * Add a multi-command test * Refactor to account for N commands separated by semicolon and new Request structure. Also fix Kill since we changed it's proto struct as well. TODO: Remaining new commands and some sanity testing for N commands. * regen * Implement all the remaining missing CLI commands. Standardize usage Fix help/flags/commands to work correctly. Rename fdbcli help to fdbclihelp as we have a conflict otherwise. * Make sure for basic top level help/flags/commands we still pass an execute state. * Fix tabwriter so the output actually looks good when we're nested a bit. * Update reserved syntax to actual proto3 style * Add default cases for internal oneof's so we error if one gets added we don't handle * Remove transaction. it's no longer a distinct message * Rewrite parse loop to be a lot cleaner

2022-05-26

With the proxy pushing all connection logic after stream reply we can add a dial timeout now (#130) Move Dial into Run() as well and plumb a timeout into here (optional). This allow a per target dial timeout if wanted. Blocking doesn't matter since all connection logic is handled inside a per target go routine so we can safely do the separate blocking call from Dial now. Add the ability for DialContext in TargetDialer to take additional options at calling time. This way we can slip in a WithBlock() but only in cases it's needed. Add tests proving timeouts work and don't error in the normal case * Rework stream setup in client. Now that the server returns quicker it's possible for this to happen: reply 0 reply 1 close 1 reply 2 i.e. we get a close before all replies. Close needs to be processed and inserted into errors and removed from tracking. * Maintain a list of streams we've previously had but are now closed. The client is not required to sync and likely isn't as this is a bidi stream. Therefore it's valid for this sequence to happen: ->start <-reply <-close ->data as the client may read at the reply point but not read close before sending data. Close conditions are asynch in nature. The Recv() we're blocking on may return for a TCP reset at any time. Though in testing it was most often a dial problem showing immediately in NewStream and racing. * Add test proving parallel works. 2 working targets 1 dial error 1 timeout All happen in parallel and don't leak onto other targets even with short (<20s) timeouts. * Add debugging to parallel work * Errors can be different between environments. Just count errors vs healthy.

2022-05-24

Convert logfile return from fdbcli into a streaming RPC instead of unary (#122) * Convert logfile return from fdbcli into a streaming RPC instead of unary. Logfiles can be arbitrary sizes and we don't want unbounded memory consumption for a large file. Instead stream them over in reasonable chunks keyed by filename. * No need to defer log removal, we're at the end of function. * Add stdout check that was missing

2022-05-18

Add rename (i.e. mv) API to localfile. (#126)

2022-05-13

Proto for FDB services: (#114) * Proto for FDB services: Conf - edit configuration data FDBCli - Wrapper around fdbcli * Add copyrights * Proper license header. * Exclude proto dir from testing

2022-05-05

Add a version RPC and CLI arg for returning the embedded one. (#108) * Add a version RPC and CLI arg for returning the embedded one. Expose in CLI/proxy-server as it's own method (similar to logging). Integrate both methods into integration test * Add license block

2022-05-02

Add kill to RPCs in process module (#103) * Add kill to process to send signals to a given pid. No support for wildcarding with negative pids/signals (too dangerous) * Client side of kill * Integration testing for kill. shfmt also redid some lines

2022-04-13

Add support to file ops so we can do things by name instead of requiring uid/gid. (#100) * Add support to file ops so we can things by name instead of requiring uid/gid. i.e. setting permissions with chown or copying files up. This allows easier use from the CLI where one may not know the uid/gid already. The server will lookup values when presented with names. The options are exclusive. Can't set both uid and username together. Flesh out some tests and integration tests to validate. * Make sure nobody exists. * Run creates under sudo on github * Only need group on github * Make sure nobody's group defaults to nobody also on github

2022-03-10

Fix naming to reflect sansshell (#91) re: https://github.com/Snowflake-Labs/sansshell/commit/0ef6d53f2a66d78575cb648749a30dd9eba4052a

2022-02-28

Fixup accidental references to bad paths/names. (#81)

2022-02-22

Implement a logging module. (#77) * Implement a logging module. This can safely set the logging verbosity level on a server using stdr package. Implement basic unit tests and client/integration tests. All client side support in sanssh to do this on the proxy itself. Wire into proxy as an RPC it'll serve in addition to /Proxy.Proxy This means we need to add interceptors for unary RPCs here now too. * Rename from logging to sansshell as the service breakdown so it's obvious this is internal state. * Fix client command to be sansshell instead of logging

2022-01-27

Implementation for rm and rmdir operations (#62) * Add proto definitions and placeholders for service for rm and rmdir. Regen all protos * Provide a fake hook for kqueue/kevent for darwin and add OS specific tests to validate. Caught a bug where we weren't returning error correctly on kevent setup. * Remove a debugging print and add hooks for testing inotify/epoll failures * Add tests for linux specific inotify/epoll fakes so can validate error cases. Caught fact in errors closer() was getting set to nil which would panic() * Move osStat as a var so we can swap out for tests. * Implement rm/rmdir * Add tests for chmod failing (due to inaccessible path) Add more tests to ls where stat fails and various directory/file ops can fail. Add tests for rm/rmdir * Bring coverage up to 90% as all get to there now. * Add formatting checks * Add basic go vet check * Add client side commands for rm/rmdir and add integration tests. Integration more manual since 2 commands to the same host will have one fail every time with ENOENT. * Convert to subcommands style for rm/rmdir * Make sure and return a structured error for rm/rmdir * Add a test to validate rmdir fails if files are in the directory.

2022-01-21

Merge branch 'main' into jallie-services-implementation

Add the required license block to every .go/.proto/.sh file here. (#54) The generation for proto will copy this into .pb.go so include those here as a rebuild. Add checks in integration check to make sure new files can't get checked in without the license.

2022-01-20

add implementation for "service" module

2022-01-11

review comments

2022-01-07

Add initial definition for service management

2022-01-06

Write and Copy implementation (#47) * Implement server side for Write and unit tests * Regen for comments * Add comments to immutableState * Implement Copy Do this mostly in terms of the code we used in Write. Unfortunately due to differing RPC's and proto the test cases is a little copy/paste from Write. Proto needed an update to separate out bucket/key. * Update comments. * Implement CloseAndRecv correctly as it needs the proxy Recv not the standard one. Just implement the header and reuse the Recv code below. * Regen for CloseAndRecv fixes * Correctly process errors for memory dumps * Make sure we close the bucket * Implement a test for remote dump to an s3 bucket * Client side cp implementation * Fix option processing (was requiring bucket) Also adjust errors to ignore io.EOF * Add fixes for CloseAndRecv since we have to loop over all the responses till we get one for each target. There is no other recv that'll happen (it's not even a generated message). Otherwise we lose results. * Regen for CloseAndRecv changes * Fix bucket prefix checking (check against bucket..) Fix error checking (ignore EOF) * Move functions to one place in file. Add tests for cp including local and remote (via s3) modes. Add a remote coredump which pushes to s3 test. * Rework logic where we must read a description first for write then contents. Simplifies the for loop for grabbing contents.

2021-12-02

Implement chown/chgrp/chmod/immutable commands (#42) * Implement tail support and update tests. Also add tests for a few missing cases we can without service mocks. * Fix immutable testing * Move immutable from direct stat as well. * Implement cancel correctly for tail. We can't poll it turns out because a file is always ready to read even at EOF. It'll just return EOF over and over.. So we have to do an old school sleep loop so we can break out to check cancel state. Update unit tests to accomodate. * Implementation for SetFileAttributes. Change proto to have a cleaner API so people aren't accidentally setting user/group to root. i.e. defaults can't be 0 for int fields this way. Add separate client commands for all ops to map to unix: chmod chown chgrp immutable Add unit tests (with mocks for some system things since it's useful). Add e2e tests * Adjust mask to be correct so we can pass it to chown

Misc cleanups/optimizations (#41) * Convert healthcheck to google.protobuf.Empty instead of our own version * Move all custom flags into util, add tests and standardize behavior better. i.e. allow for var's of these to "just work" without requiring additional setup. * Standardize unit tests: All table driven tests are tc vars All table driven tests use non exported names for the members of the anon struct. All table driven tests use anon structs inline with the range. All table driven tests use t.Run (with variable shadowing). * Convert a test that got missed. * Remove dead code I accidentally left commented out as a test. * Tail in current form needs slightly longer timeout.

2021-11-30

Add tail and immutable support to stat (#40) * API features/changes for LocalFile: Change Read to take a one_of indicating reading a file or tailing a file. Add List which allows you to implement ls on a directory (with optional recursion). Add immutable options to StatReply and FileAttributes (so one can set or remove it). * go mod tidy * Move path testing to util and convert things to use it. * Implement tail support and update tests. Also add tests for a few missing cases we can without service mocks. * Update comment on immutability. * Move stat support to os specific implementations. This way immutable state can be abstracted out as OS/X and linux do it differently. In addition Linux requires a 4.11+ kernel to do this via a syscall. Otherwise you need an ioctl... Unfortunately no way to unit test immutable as it requires root. Will add to end-end. * Add immutable state to stat output * Add tests for immutable files and tail. Also suppress terminated message for process deaths by using SIGINT. Make sure this aborts for non-linux. Move policy into $LOGS and don't delete it on each run. Make job terminations quiet to avoid noise. * Fix disown since shell isn't tracking the sudo -b, only the proxy-server * Adjust constant to be more clear and actually use it. * Update a comment * Update function comment to be correct. * Move disown and document it better * Fix immutable testing * Use the same stat to check that we'll be using server side (it isn't always os.Stat) * Move immutable from direct stat as well. * Add test case and handling for not sending a valid request. * Implement cancel correctly for tail. We can't poll it turns out because a file is always ready to read even at EOF. It'll just return EOF over and over.. So we have to do an old school sleep loop so we can break out to check cancel state. Update unit tests to accomodate.

2021-11-24

API features/changes for LocalFile (#39) * API features/changes for LocalFile: Change Read to take a one_of indicating reading a file or tailing a file. Add List which allows you to implement ls on a directory (with optional recursion). Add immutable options to StatReply and FileAttributes (so one can set or remove it). * Fix policy/tests for moving Read msg type around. * Actually make the tests work this time... * Log policy input fully (stringified) * Handle errors better for remote file reads (read what we can, record errors, exit non-zero on any). * turn tail into a TailRequest which has no length field. This way it's just an offset to start at and then we return to EOF and continue from there. * Remove recursion and cleanup List so it's just a list of entries. i.e. List on a file returns one thing just like ls <file> does. * Update ListReply comment

2021-11-10

API skeleton for Write/Copy (#30) * Skeleton API for Write and Copy functions. Allow write to process N files in one stream. Open question if copy should be unary or also allowing N * Update to more structured proto types. Add SetFileAttributes to do chown/chgrp/chmod as a single rpc as needed (since it reuses so much of what we need for write/copy) * Rename a few fields and add comments for clarity.

2021-11-09

Implement remote bucket destinations for core/heap (#29) * Fixes for RunCommand handling stderr output now. * Add blob support for heap/core and testing for cores. NOTE: This is ridiculous on how much code is common. The only reason most can't be shared is distinct types for RPCs. This really should be one RPC "get memory dump" and an option for "this is java". * Revamp for core/heaps: 1 RPC with an option saying "this is java" Java - use jmap Everything else - use gcore If somehow someone gets it wrong a heapdump can be pulled from a core anyways. Vastly simplifies duplicate code * Add blob packages * Rename TestCore to TestMemoryDump * Remove getcore/etc as shouldn't have slipped into policy * Merge updates: Somehow lost these changes on rebase * Fix logging * Fixes for one RPC instead of 2 and only needing 1 client impl. * Make sure errors are pretty. Include support for buckets instead of local file output.. * Remove memory dump that slipped in * Turn dump type into an enum vs a bool for "is this java?" * Fix typos * Convert DumpType into a proper enum flag * Rework destination into a one-of so we don't have extra params that only work "sometimes". Fix bugs in client related to enum flags.

2021-11-08

Implementation for core/heap dump collection (#27) * Move all command execution into a utility function. This can enforce invariants such as "must be an absolute and clean path" Move TrimString into util as well. Provide a testutil function to resolve a naked binary name (sh, false, cat) to a real path. Replace all test binary referecnes with lookups for absolute paths. * Skeleton for GetCore * Implement basic gcore/jmap to a file which we then stream. * Add a comment that currently one must have disk space for the core/heap. * Move streaming size into util * Cleanup from merge * Cleanup from merge * Fix path bug for core dumps. gcore expects /path/file and it will always append .PID to the end. * Implement support for core and heapdump generation. * Add tests for core/heapdumps. Unfortunately these are literally the same tests but due to protobuf typing/etc had to do 2 copies. Also cleanup some other tests for a bit more consistency and coverage. * Test input file * Minor cleanup from review. Move stderr testing into utils.RunCommand since it's used a lot. * Fixes for RunCommand handling stderr output now. * document failing due to stderr output better. Turn it into a proper options builder pattern and update all callers.

2021-11-02

Merge branch 'main' into jallie-proxy-proto-server

pre-review

2021-11-01

adding initial tests

Implement support for pstack/jstack (#19) * Implement pstack support. Allow testing on darwin but disable in real binary by default. Lots of tests data for various scenarios. Hook into client with it's own sub-command and cleanup this for futureproofing. * Remove a log line not needed * Add a trimString function here for clipping error text we send back. * Convert flags to foo-bar form * While everything will have ps installed not everything has pstack. Skip if we can't stat the binary. * Use the right path for pstack... * Implement jstack support: Times are floats. Convert flags to foo-bar style and add gcore/jmap in (upcoming) * Simplify parsing into a tiny state decision tree which picks out state and applies at the end. Fixed a bug where we weren't parsing state correctly on OS threads. * Add TODO to make a single entry point for exec'ing commands * Fixes from review: Use ReadFile instead of open+read Cleanup some error strings * Implement client jstack and add to default policy

2021-10-29

remove stream descriptor information (client/server stream) from startstream

2021-10-28

add proxy protocol buffer

2021-10-26

Implement Package package (#15) * Convert to status/code style for errors * Implement RepoList * Implement ListInstalled Rework structure so we can reuse a bit of setup logic between service impls. * ListInstalled test was accidentally replacing generateRepoList. * Fix comment * First pass impl for install. Needs tests * Add a test case to prove unknown repo status parses * Document for upgrade that versions must be in neva format also. * Correct a comment typo * Implement client support and hook into client/server binaries * Add verbose mode for repo list * Convert to service/server directory structure * Collapse UNKNOWN package to YUM in each RPC instead of N cases checking for it. For places we parse output from yum add an abstraction layer so other systems can be plugged in. Move setup for tests above even the basic test (otherwise it could pass when not expected) * Rework commands into a distinct package. Also add auto generated of enum values for package_system. * Split flag for yumBin into linux vs other OSs. This way it can default to something that won't work !linux and is obvious from the description. Tests still pass since we fake this up for everything and never execute the real binary. * Errors in grpc generally shouldn't be longer than a few K. Make sure where we potentially copy unbounded output back we truncate it first. * Merge from main to get correct server paths * Update flags to dash-dash form. Also futureproof client connection objects by casting to the interface instead of the concrete type.

2021-10-21

Implement Ansible playbook support (#12) * Implement Ansible playbook support Require an absolute path to a playbook. Playbook is assumed to have hosts: localhost in it and we only do connection=local. Allow setting user, vars, check, diff, verbose Tests for everything (complete coverage). Include a test playbook yml file so one can do local testing with a real server. * Remove return_code from request as it goes in reply... * Prevent key=value from containing anything except a-zA-Z0-9 * Make sure playbook is a valid path and file so malicious content can't be injected. * Allow _ in key/value pairs and update tests * Make sure user is valid as well and no spaces, etc * Use cmd.String

2021-10-19

Merge pull request #10 from Snowflake-Labs/sanshell-file-impl LocalFile: implementation for`Stat` and `Sum`

2021-10-18

Implement List call (#9) * Stat has N code appended to it when you ask for "state". Parse those up structured instead of a raw string. * Turn flags into an int64. Instead of interpreting it as OS/X vs linux treat it differently. * Begin of impl for List. Enough that it stubs out and creates server/client hooks. Assuming linux today so doesn't work on other OS yet. * Add new names for client/server to gitignore * Move ps options to OS specific mappings. Add OS/X support (doesn't work yet but compiles) Fix deadlock. Can't use ioutil to read all of a stdout/stderr buffer as the process won't exit if it's stuck buffering to the other one. Instead we need to use bytes.Buffer instead * Add OS specific parsers and implement for OS/X * Add client support for specifying a single pid * Add a helper for setting up a blank ProcessEntry so we can set defaults for empty fields * Implement so it can print ps output Add --pids (instead of --pid) to allow N pids. Print a header and PS style output with the returned data. * Revamp darwin processing Jon pointed out that -M after -o with the right args will do what we want. Convert to those options and simplify parsing into a table driven effort. Fix thread counting bug on first entry (we're double counting the first line). Visually looks correct (needs tests) * Add tests which cover darwin and sets up an easy framework for linux/etc. * Add linux test data * Use prototext package for text protos The other one is deprecated and this is the replacement for this functionality. * More test file updates * Regen for proto package changes * Add a go1.16+ build constraint Package embed didn't appear until 1.16 * A few fields need to be uint and not int. In the output we have to parse 0xffffffffffffffff which isn't a valid int64. * Updates for int64 and implement linux support * Use cmp/protocmp to print better diffs on errors * Make sure tests cover all non-default values * Updates for uint64 changes * Need a few blank lines to trigger parsing errors now * Add lwp in and adjust field counts * Updates for protocmp/cmp * Add all the state codes (linux only as darwin doesn't do this). Adjust tests to provide sorters via protocmp to have stable compares. * Simplify thread logic as we were off by one on the last entry. Make sure and add up cpu/mem for each entry as ps reports per thread. Remove one case in state as it's the default by default already. * Bunch of testdata update to make sure we test all fields. Lots of fields end up with default values so parsing doesn't prove a lot. Make an entry with all fields filled in non-default we haven't tested. Also put a singleton entry first as assuming pid 1 was multi-threaded was a bug for thread counts. * Add a test which runs the real PS and asks for output. This breaks on darwin right now due to path. Adjust linux support as the realtime scheduler doens't print a nice value. Instead it prints "-" which doesn't parse as a number. Leave it a zero in that case and change client support to print - for those classes. * Large refactor to split OS specific items into their own files. Remove all uses of runtime.GOOS and instead use separate files. Each impl has to expose a psOptions var which is a callable function. It's a var so testing can replace it as needed. Also expose a parser function and then everything works. Rework test to do the same where each OS has to expose some strings for finding test files. * use bytes.Buffer.String() instead of string(bytes.Buffer.Bytes()) * Avoid shadowing vars Break parsing class,state into helper funcs * Convert to status.Errorf instead of fmt.Sprintf embedded * Remove error from psOptions siganture. Not needed * Add tests for a bad command and stderr output * Remove a case statement we don't need.

2021-10-16

Skeleton for Ansible playbook support (#3) * Skeleton for Ansible playbook support Support running against a specific module with args passed. Allow diff/check, become and fork args. Today just return the output as a single string since debug output is determined per module. * Refactor to local ansible-playbook only with key/value substitutions. * Oops. Forgot to add a playbook arg to request. Fixed * Add verbose option * Whitespace update * Add return code to ansible proto Also document that a non-zero return from playbook doesn't mean an RPC failed.

2021-10-14

implementation for LocalFile Stat and Sum

2021-10-12

Update proto with various niceties discovered while implementing cross platform (#8) * Stat has N code appended to it when you ask for "state". Parse those up structured instead of a raw string. * Turn scheduling class into a real enum. Also note this may return empty/guard values if the OS doesn't support gathering the values. * Rename lwp to threads to be a bit more platform agnostic * Turn flags into an int64. Instead of interpreting it as OS/X vs linux treat it differently.

swap tags for sum filename and type

2021-10-11

Merge branch 'main' into jallie-sanshell-commands

switch stat/sum to bidi streaming, update readme

Skeleton for Process service (#5) * Skeleton for Process service Allows getting remote ps output Also stacks and cores from both regular processes and java versions (using jstack/jmap) * Break up process listings a bit better. Return discrete fields for each header/value broken out by entry. Also update comments to indicate some ops have sensitive data. * Make core/heap specifiy a destination so we can write to a remote URL instead of streaming back. * Add comments for command lines having sensitive data. Flesh out process list so we get structured data back instead of simple parsed ps. Add a blob field for heap/core where one can stash destination specific data (ala creds). * Update comment (we don't pass options to ps now)