Proto commits in openbao/openbao

These commits are when the Protocol Buffers files have changed: (only the last 100 relevant commits are shown)

2024-10-05

Add transactions to plugin GRPC layer (#497) This adds support for transactions to the plugin GRPC layer, opportunistically upgrading the client storage implementation to a transactional aware one if the underlying storage implementation on the server supports it. This retains bidirectional interoperability with upstream, allowing plugins compiled with OpenBao's SDK to continue to work with upstream server implementations and visa-versa. Signed-off-by: Alexander Scheel <ascheel@gitlab.com>

The documentation is generated from this commit.

2024-08-30

Fix api, sdk modules with v2.0.1 (#425) * Update protobuf definitions to use v2 format Signed-off-by: Alexander Scheel <ascheel@gitlab.com> * Fix api, sdk modules with v2.0.1 This retracts the v2.0.0 module version that we pushed and finishes the Go module at /v2. In the future, the following module tagging order will need to be used: - api - api/auth/* - sdk - Any external dependencies (go-kms-wrapping, openbao-template, &c) - Finally, the main module In this case, the api/auth/* modules aren't currently buildable until the v2.0.1 tag is pushed for api. Signed-off-by: Alexander Scheel <ascheel@gitlab.com> --------- Signed-off-by: Alexander Scheel <ascheel@gitlab.com>

2024-07-24

Fix api, sdk modules with v2.0.1 (#425) * Update protobuf definitions to use v2 format Signed-off-by: Alexander Scheel <ascheel@gitlab.com> * Fix api, sdk modules with v2.0.1 This retracts the v2.0.0 module version that we pushed and finishes the Go module at /v2. In the future, the following module tagging order will need to be used: - api - api/auth/* - sdk - Any external dependencies (go-kms-wrapping, openbao-template, &c) - Finally, the main module In this case, the api/auth/* modules aren't currently buildable until the v2.0.1 tag is pushed for api. Signed-off-by: Alexander Scheel <ascheel@gitlab.com> --------- Signed-off-by: Alexander Scheel <ascheel@gitlab.com>

2024-06-12

Revert "Remove Server Side Consistent Tokens (SSCTs)" This reverts commit 1f2635c3d1ac90a60bc4193668fd6309da2f06ab. As discussed on #openbao-general, this breaks existing migrations: anyone with SSCT tokens present in token store would lose all existing tokens and need to re-auth everything. This is moderately more disruptive for root tokens in particular, as `operator generate-root` would need to be taken (and sometimes these root tokens are stored but not used, as they don't necessarily expire). This reasonably breaks the "drop-in migration" guarantees of a Raft storage backend, and thus will be reverted for the time being. Other than the protobuf regeneration (which makes sense as it is an auto-generated file anyways), this was a clean revert. Resolves: #297 Signed-off-by: Alexander Scheel <alexander.m.scheel@gmail.com>

2024-03-25

Remove Enterprise features from api/, sdk/ This removes various references of WALs, Enterprise-only Auto Pilot configuration, Performance Secondaries, and Disaster Recovery from api/ and sdk/, including dependent pieces in command, agent, and proxy (around enforcing consistency, FIPS status, and other related topics). This also fixes the README to update references to DockerTestHelper to remove Vault Enterprise support. Signed-off-by: Alexander Scheel <alexander.m.scheel@gmail.com>

Add ListPage(...) as a storage operation This adds the ability to list a specified number of entries, seeking until after after a given reference entry, to the storage backends. With efficient implementations, this should allow reducing the memory overhead of API LIST operations, though at the lack of consistency between fetches. This should allow more expensive tidy operations (such as PKI's) to more correctly release resources between operations and gracefully handle larger numbers of certificates. Additionally, this functionality will be useful for implementing better pagination on list views (again, like PKI's), as finer control over fetched entries from the server can be had, avoiding the need to keep the entire list in the browser's memory. This has broader applicability beyond PKI. Signed-off-by: Alexander Scheel <alexander.m.scheel@gmail.com>

2024-03-14

Remove Server Side Consistent Tokens (SSCTs) SSCTs were a Vault Enterprise feature, implemented everywhere, to introduce tokens which encoded information about Performance Secondary replication state into requests. This let callers pass this token to any other server and guarantee their state is at least as fresh as this token (by virtue of it containing a WAL index number). However, this didn't apply to HA clusters and only to Performance Secondary cluster types, meaning it is of limited utility to us. We revert to the old token prefixes (`s.`, `b.`, and `r.`), rather than continuing to use the SSC Token prefixes (`hvs.`, `hvb.`, and `hvr.`). This ensures any utilities attempting to decode Vault's SSCTs knows these do not have the required internal structure. Signed-off-by: Alexander Scheel <alexander.m.scheel@gmail.com>

2024-03-12

Remove eventing & experimental subsystems The eventing subsystem was the only pre-release experiment present in upstream's 1.14.x branch; this wasn't stabilized and more widely used until later release (1.15.x and the upcoming 1.16.x); and so makes little sense to keep in its partial form. Additionally, no existing plugins within this distribution currently emit such events. By removing eventing, we can also remove the experiments subsystem; this was only used for enabling the eventing subsystem in this release. See also: https://developer.hashicorp.com/vault/docs/concepts/events Signed-off-by: Alexander Scheel <alexander.m.scheel@gmail.com>

Remove activity log from OpenBao The activity log subsystem was an upstream specific mechanism to handle counting of "clients" (for some definition there-of, mostly focused around combining the concept of an entity with some equivalence relation for non-entity tokens to reduce the total number), for the express purpose of billing customers. This code was thus in OSS so that, on purchasing an Enterprise license, an idea of pricing could be established. As we don't have Vault Enterprise, removing this code is likely best as it allows us to avoid having to maintain it going forward and avoids any incompatibilities with upstream if they were to change the definitions. See also: https://developer.hashicorp.com/vault/docs/concepts/client-count/counting See also: https://developer.hashicorp.com/vault/api-docs/system/internal-counters Signed-off-by: Alexander Scheel <alexander.m.scheel@gmail.com>

Remove Performance Secondary, DR Secondary consts From the core vault/ and http/ space, remove references to Performance Secondary and Disaster Recovery Secondary clusters as they are Vault Enterprise only functionality. This lets us simplify some of the semantics around forwarding of requests (removing many no-ops in the process) and execution of tasks (again, mostly removing no-ops and unnecessary channels). Also regenerates all protobuf files, causing a few extraneous checkins with the new version. Signed-off-by: Alexander Scheel <alexander.m.scheel@gmail.com>

2024-02-09

Point lf-edge->openbao for GH organization Signed-off-by: Alexander Scheel <alexander.m.scheel@gmail.com>

2024-02-03

Add 'builtin/logical/kv/' from commit '49393bae3947b42057533b54f4977bcc4ed313b6' git-subtree-dir: builtin/logical/kv git-subtree-mainline: 397dbb90ad87ab3ed616e6f38e8160bf3362673a git-subtree-split: 49393bae3947b42057533b54f4977bcc4ed313b6

2024-01-26

Request Limiter (#25093) This commit introduces two new adaptive concurrency limiters in Vault, which should handle overloading of the server during periods of untenable request rate. The limiter adjusts the number of allowable in-flight requests based on latency measurements performed across the request duration. This approach allows us to reject entire requests prior to doing any work and prevents clients from exceeding server capacity. The limiters intentionally target two separate vectors that have been proven to lead to server over-utilization. - Back pressure from the storage backend, resulting in bufferbloat in the WAL system. (enterprise) - Back pressure from CPU over-utilization via PKI issue requests (specifically for RSA keys), resulting in failed heartbeats. Storage constraints can be accounted for by limiting logical requests according to their http.Method. We only limit requests with write-based methods, since these will result in storage Puts and exhibit the aforementioned bufferbloat. CPU constraints are accounted for using the same underlying library and technique; however, they require special treatment. The maximum number of concurrent pki/issue requests found in testing (again, specifically for RSA keys) is far lower than the minimum tolerable write request rate. Without separate limiting, we would artificially impose limits on tolerable request rates for non-PKI requests. To specifically target PKI issue requests, we add a new PathsSpecial field, called limited, allowing backends to specify a list of paths which should get special-case request limiting. For the sake of code cleanliness and future extensibility, we introduce the concept of a LimiterRegistry. The registry proposed in this PR has two entries, corresponding with the two vectors above. Each Limiter entry has its own corresponding maximum and minimum concurrency, allowing them to react to latency deviation independently and handle high volumes of requests to targeted bottlenecks (CPU and storage). In both cases, utilization will be effectively throttled before Vault reaches any degraded state. The resulting 503 - Service Unavailable is a retryable HTTP response code, which can be handled to gracefully retry and eventually succeed. Clients should handle this by retrying with jitter and exponential backoff. This is done within Vault's API, using the go-retryablehttp library. Limiter testing was performed via benchmarks of mixed workloads and across a deployment of agent pods with great success.

2024-01-25

Remove HCP Link from OpenBao HCP Link is a service to link local Vault Enterprise instances to HashiCorp's HCP cloud offering. This isn't useful for consumers of OpenBao and we do not wish to have to maintain it. Thus, removal is most prudent. Signed-off-by: Alexander Scheel <alexander.m.scheel@gmail.com>

2024-01-18

systemview: adds method for plugins to generate identity tokens (#24929) * systemview: adds method for plugins to generate identity tokens * change test name and godoc * adds changelog * make proto to include comment

2024-01-04

Update licensing across various source files (#24672)

2023-12-12

rebase to development Signed-off-by: Jeremiah Butler <jeremiah.butler@ibm.com>

2023-12-08

rebase to development Signed-off-by: Jeremiah Butler <jeremiah.butler@ibm.com>

2023-12-04

Use our heartbeat echo RPCs to estimate clock skew, expose it in status APIs (#24343)

2023-10-23

Add support for plugins to specify binary request paths (#23729) * wip * more pruning * Integrate OCSP into binary paths PoC - Simplify some of the changes to the router - Remove the binary test PKI endpoint - Switch OCSP to use the new binary paths backend variable * Fix proto generation and test compilation * Add unit test for binary request handling --------- Co-authored-by: Scott G. Miller <smiller@hashicorp.com>

2023-09-20

VAULT-19237 Add mount_type to secret response (#23047) * VAULT-19237 Add mount_type to secret response * VAULT-19237 changelog * VAULT-19237 make MountType generic * VAULT-19237 clean up comment * VAULT-19237 update changelog * VAULT-19237 update test, remove mounttype from wrapped responses * VAULT-19237 fix a lot of tests * VAULT-19237 standby test

2023-08-24

Add vault community changes for the Seal HA project. (#22515) * Seal HA: Use new SealWrappedValue type to abstract seal wrapped values Introduce SealWrappedValue to abstract seal wrapped values. Make SealWrappedValue capable of marshalling into a BlobInfo, when there is plaintext or a single encryption, or to a custom serialization consisting of a header, length and a marshalled MultiWrapValue protobuf. * Vault-13769: Support configuring and using multiple seals for unsealing * Make sealWrapBackend start using multiple seals * Make seal.Access no longer implement wrapping.Wrapper. Instead, add the Encrypt and Decrypt methods to the Access interface. * Make raft snapshot system use funcs SealWrapValue + UnsealWrapValue. Move the snapshot.Sealer implementation to the vault package to avoid circular imports. * Update sealWrapBackend to use multiple seals for encryption. Use all the encryption wrappers when storing seal wrapped values. Try do decrypt using the highest priority wrapper, but try all combinations of encrypted values and wrappers if necessary. * Allow the use of multiple seals for entropy augmentation Add seal_name variable in entropy stanza Add new MultiSourcer to accommodate the new entropy augmentation behavior. * Individually health check each wrapper, and add a sys/seal-backend-status endpoint. * Address a race, and also a failed test mock that I didn't catch * Track partial wrapping failures... ... where one or more but not all access.Encrypts fail for a given write. Note these failures by adding a time ordered UUID storage entry containing the path in a special subdirectory of root storage. Adds a callback pattern to accomplish this, with certain high value writes like initial barrier key storage not allowing a partial failure. The followup work would be to detect return to health and iterate through these storage entries, rewrapping. * Add new data structure to track seal config generation (#4492) * Add new data structure to track seal config generation * Remove import cycle * Fix undefined variable errors * update comment * Update setSeal response * Fix setSealResponse in operator_diagnose * Scope the wrapper health check locks individually (#4491) * Refactor setSeal function in server.go. (#4505) Refactor setSeal function in server.go. * Decouple CreateSecureRandomReaderFunc from seal package. Instead of using a list of seal.SealInfo structs, make CreateSecureRandomReaderFunc use a list of new EntropySourcerInfo structs. This brakes the denpency of package configutil on the seal package. * Move SealGenerationInfo tracking to the seal Access. * Move SealGenerationInfo tracking to the seal Access. The SealGenerationInfo is now kept track by a Seal's Access instead of by the Config object. The access implementation now records the correct generation number on seal wrapped values. * Only store and read SealGenerationInfo if VAULT_ENABLE_SEAL_HA_BETA is true. * Add MultiWrapValue protobuf message MultiWrapValue can be used to keep track of different encryptions of a value. --------- Co-authored-by: Victor Rodriguez <vrizo@hashicorp.com> * Use generation to determine if a seal wrapped value is up-to-date. (#4542) * Add logging to seal Access implementation. * Seal HA buf format run (#4561) * Run buf format. * Add buf.lock to ensure go-kms-wrapping module is imported. * Vault-18958: Add unit tests for config checks * Add safety logic for seal configuration changes * Revert "Add safety logic for seal configuration changes" This reverts commit 7fec48035a5cf274e5a4d98901716d08d766ce90. * changes and tests for checking seal config * add ent tests * remove check for empty name and add type into test cases * add error message for empty name * fix no seals test --------- Co-authored-by: divyapola5 <divya@hashicorp.com> * Handle migrations between single-wrapper and multi-wrapper autoSeals * Extract method SetPhysicalSealConfig. * Extract function physicalSealConfig. The extracted function is the only code now reading SealConfig entries from storage. * Extract function setPhysicalSealConfig. The extracted function is the only code now writing SealConfig entries from storage (except for migration from the old recovery config path). * Move SealConfig to new file vault/seal_config.go. * Add SealConfigType quasy-enumeration. SealConfigType is to serve as the typed values for field SealConfig.Type. * Rename Seal.RecoveryType to RecoverySealConfigType. Make RecoverySealConfigType return a SealConfigType instead of a string. * Rename Seal.BarrierType to BarrierSealConfigType. Make BarrierSealConfigType return a SealConfigType. Remove seal.SealType (really a two-step rename to SealConfigType). * Add Seal methods ClearBarrierConfig and ClearRecoveryConfig. * Handle autoseal <-> multiseal migrations. While going between single-wrapper and multiple-wrapper autoseals are not migrations that require an unwrap seal (such as going from shamir to autoseal), the stored "barrier" SealConfig needs to be updated in these cases. Specifically, the value of SealConfg.Type is "multiseal" for autoSeals that have more than one wrapper; on the other hand, for autoseals with a single wrapper, SealConfig.Type is the type of the wrapper. * Remove error return value from NewAutoSeal constructor. * Automatically rewrap partially seal wrapped values on an interval * Add in rewrapping of partially wrapped values on an interval, regardless of seal health/status. * Don't set SealGenerationInfo Rewrapped flag in the partial rewrap call. * Unexport the SealGenerationInfo's Rewrapped field, add a mutex to it for thread safe access, and add accessor methods for it. * Add a success callback to the manual seal rewrap process that updates the SealGenerationInfo's rewrapped field. This is done via a callback to avoid an import cycle in the SealRewrap code. * Fix a failing seal wrap backend test which was broken by the unexporting of SealGenerationInfo's Rewrapped field. * Nil check the seal rewrap success callback before calling it. * Change SealGenerationInfo rewrapped parameter to an atomic.Bool rather than a sync.RWMutex for simplicity and performance. * Add nil check for SealAccess before updating SealGenerationInfo rewrapped status during seal rewrap call. * Update partial rewrap check interval from 10 seconds to 1 minute. * Update a reference to SealGenerationInfo Rewrapped field to use new getter method. * Fix up some data raciness in partial rewrapping. * Account for possibly nil storage entry when retrieving partially wrapped value. * Allow multi-wrapper autoSeals to include disabled seal wrappers. * Restore propagation of wrapper configuration errors by setSeal. Function setSeal is meant to propagate non KeyNotFound errors returned by calls to configutil.ConfigureWrapper. * Remove unused Access methods SetConfig and Type. * Allow multi-wrapper autoSeals to include disabled seal wrappers. Make it possible for an autoSeal that uses multiple wrappers to include disabled wrappers that can be used to decrypt entries, but are skipped for encryption. e an unwrapSeal when there are disabled seals. * Fix bug with not providing name (#4580) * add suffix to name defaults * add comment * only change name for disabled seal * Only attempt to rewrap partial values when all seals are healthy. * Only attempt to rewrap partial values when all seals are healthy. * Change logging level from info to debug for notice about rewrap skipping based on seal health. * Remove stale TODOs and commented out code. --------- Co-authored-by: rculpepper <rculpepper@hashicorp.com> Co-authored-by: Larroyo <95649169+DeLuci@users.noreply.github.com> Co-authored-by: Scott G. Miller <smiller@hashicorp.com> Co-authored-by: Divya Pola <87338962+divyapola5@users.noreply.github.com> Co-authored-by: Matt Schultz <matt.schultz@hashicorp.com> Co-authored-by: divyapola5 <divya@hashicorp.com> Co-authored-by: Rachel Culpepper <84159930+rculpepper@users.noreply.github.com>

2023-08-11

Fix activity log generator license (#22301)

VAULT-17145, VAULT-17146: Write intent logs and event type in testing (#21132) * add entry and intent log * test for intent logs

VAULT-15396: Client count testing library (#20774) * fluent generation of client count testing data input * godocs * add VerifyInput and complete testing * pr fixes * pr fixes * buf lint

[COMPLIANCE] License changes (#22290) * Adding explicit MPL license for sub-package. This directory and its subdirectories (packages) contain files licensed with the MPLv2 `LICENSE` file in this directory and are intentionally licensed separately from the BSL `LICENSE` file at the root of this repository. * Adding explicit MPL license for sub-package. This directory and its subdirectories (packages) contain files licensed with the MPLv2 `LICENSE` file in this directory and are intentionally licensed separately from the BSL `LICENSE` file at the root of this repository. * Updating the license from MPL to Business Source License. Going forward, this project will be licensed under the Business Source License v1.1. Please see our blog post for more details at https://hashi.co/bsl-blog, FAQ at www.hashicorp.com/licensing-faq, and details of the license at www.hashicorp.com/bsl. * add missing license headers * Update copyright file headers to BUS-1.1 * Fix test that expected exact offset on hcl file --------- Co-authored-by: hashicorp-copywrite[bot] <110428419+hashicorp-copywrite[bot]@users.noreply.github.com> Co-authored-by: Sarah Thompson <sthompson@hashicorp.com> Co-authored-by: Brian Kassouf <bkassouf@hashicorp.com>

2023-07-31

Migrate protobuf generation to Buf (#22099) * Migrate protobuf generation to Buf Buf simplifies the generation story and allows us to lean into other features in the Buf ecosystem, such as dependency management, linting, breaking change detection, formatting and remote plugins. * Format all protobuf files with buf Also add a CI job to ensure formatting remains consistent * Add CI job to warn on proto generate diffs Some files were not regenerated with the latest version of the protobuf binary. This CI job will ensure we are always detect if the protobuf files need regenerating. * Add CI job for linting protobuf files

2023-05-30

backport of commit 7490a2ebb7f5950321ec7930ee5f911c86febb04 (#20856) Co-authored-by: Milena Zlaticanin <60530402+Zlaticanin@users.noreply.github.com>

Add support for client cert credential type (#20425) * poc: client cert credential type * go mod tidy * fix typo * fix newUserReqToProto * add changelog * add newline * fix changelog * add test for the client cert generator * Fix formatting * unset signing bundle URLs * set BasicConstraintsValidForNonCA to false * backdate cert by 30s * remove empty creation params URLs * check cert BasicConstraintsValid * set default key bits in newClientCertificateGenerator * fix client cert gen test with default values * Add default for key_type * fix default key_type * update test with default key type * update test * Update changelog/20425.txt Co-authored-by: Austin Gebauer <34121980+austingebauer@users.noreply.github.com> * set default key bits and sig bits * remove the default for key type ad fix the test * make fmt + add comments for each exported field * restart test --------- Co-authored-by: Austin Gebauer <agebauer@hashicorp.com> Co-authored-by: Austin Gebauer <34121980+austingebauer@users.noreply.github.com>

2023-05-23

VAULT-14735: generate mock clients for activity log (#20252) * first part of segment client generation * fix imports * initial pr fixes * refactor and fix * update comments * assign client type

2023-05-19

Add client_type field to EntityRecord protobuf (#20626) * Add client_type field to EntityRecord protobuf * changelog * Add ACME clientType verification

2023-04-12

VAULT-14734: activity log write endpoint (#20019) * add noop endpoint with testonly build flag * add tests for endpoint * cleanup * fix test name * add changelog * pr fixes

2023-03-15

adding copyright header (#19555) * adding copyright header * fix fmt and a test

2023-02-17

events: WS protobuf messages should be binary (#19232) The [WebSockets spec](https://www.rfc-editor.org/rfc/rfc6455) states that text messages must be valid UTF-8 encoded strings, which protobuf messages virtually never are. This now correctly sends the protobuf events as binary messages. We change the format to correspond to CloudEvents, as originally intended, and remove a redundant timestamp and newline. We also bump the eventlogger to fix a race condition that this code triggers.

2023-02-10

backport of commit 34b3d0406d34586b201512fee739581007a30156 (#19136) Co-authored-by: Tom Proctor <tomhjp@users.noreply.github.com>

Convert events metadata type to google.protobuf.Struct (#19130)

2023-02-09

events: Add websockets and command (#19057) Also updates the event receieved to include a timestamp. Websockets support both JSON and protobuf binary formats. This can be used by either `wscat` or the new `vault events subscribe`: e.g., ```sh $ wscat -H "X-Vault-Token: $(vault print token)" --connect ws://127.0.0.1:8200/v1/sys/events/subscribe/abc?json=true {"event":{"id":"5c5c8c83-bf43-7da5-fe88-fc3cac814b2e", "note":"testing"}, "eventType":"abc", "timestamp":"2023-02-07T18:40:50.598408Z"} ... ``` and ```sh $ vault events subscribe abc {"event":{"id":"5c5c8c83-bf43-7da5-fe88-fc3cac814b2e", "note":"testing"}, "eventType":"abc", "timestamp":"2023-02-07T18:40:50.598408Z"} ... ``` Co-authored-by: Tom Proctor <tomhjp@users.noreply.github.com>

2023-02-03

Add events sending routed from plugins (#18834) This isn't perfect for sure, but it's solidifying and becoming a useful base to work off. This routes events sent from auth and secrets plugins to the main `EventBus` in the Vault Core. Events sent from plugins are automatically tagged with the namespace and plugin information associated with them.

2023-02-01

add ClusterName to meta GetClusterStatusResponse (#18944) * add ClusterName to meta GetClusterStatusResponse * make proto

2023-01-24

Allow Token Create Requests To Be Replicated (#18689) * Allow Token Create Requests To Be Replicated * adding a test * revert a test

2023-01-23

named Login MFA methods (#18610) * named MFA method configurations * fix a test * CL * fix an issue with same config name different ID and add a test * feedback * feedback on test * consistent use of passcode for all MFA methods (#18611) * make use of passcode factor consistent for all MFA types * improved type for MFA factors * add method name to login CLI * minor refactoring * only accept MFA method name with its namespace path in the login request MFA header * fix a bug * fixing an ErrorOrNil return value * more informative error message * Apply suggestions from code review Co-authored-by: Nick Cabatoff <ncabatoff@hashicorp.com> * feedback * test refactor a bit * adding godoc for a test * feedback * remove sanitize method name * guard a possbile nil ref Co-authored-by: Nick Cabatoff <ncabatoff@hashicorp.com>

2023-01-20

Add path based primary write forwarding (PBPWF) - OSS (#18735) * Add WriteForwardedStorage to sdk's plugin, logical in OSS This should allow backends to specify paths to forward write (storage.Put(...) and storage.Delete(...)) operations for. Notably, these semantics are subject to change and shouldn't yet be relied on. Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com> * Collect paths for write forwarding in OSS This adds a path manager to Core, allowing tracking across all Vault versions of paths which could use write forwarding if available. In particular, even on OSS offerings, we'll need to template {{clusterId}} into the paths, in the event of later upgrading to Enterprise. If we didn't, we'd end up writing paths which will no longer be accessible post-migration, due to write forwarding now replacing the sentinel with the actual cluster identifier. Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com> * Add forwarded writer implementation to OSS Here, for paths given to us, we determine if we need to do cluster translation and perform local writing. This is the OSS variant. Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com> * Wire up mount-specific request forwarding in OSS Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com> * Clarify that state lock needs to be held to call HAState in OSS Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com> * Move cluster sentinel constant to sdk/logical Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com> * Expose ClusterID to Plugins via SystemView This will let plugins learn what the Cluster's ID is, without having to resort to hacks like writing a random string to its cluster-prefixed namespace and then reading it once it has replicated. Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com> * Add GRPC ClusterID implementation For any external plugins which wish to use it. Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com> Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

Use schema for events in event broker (#18693) For the new events schema. Based on the CloudEvents schema.

2023-01-06

change quorum warning to a singular string (#18619)

2023-01-05

VAULT-11829: Add GetClusterStatus rpc to meta capability (#18316) * add GetClusterStatus rpc to meta capability * consolidate HA-related info * add storage type

VAULT-11827: Add new Link node level fields (#18203) * add addition Link node-level status fields * pin protoc to 3.21.9 * make proto * change LogLevel to be a string * add RaftStatus; IsActive -> Activetime * use an enum for LogLevel

2022-12-09

use meta and link_control from link proto (#18290)

copy over link_control and meta to link proto module (#18285)

2022-12-08

Link OSS (#18228) * add Link config, init, and capabilities * add node status proto * bump protoc version to 3.21.9 * make proto * adding link tests * remove wrapped link * add changelog entry * update changelog entry

2022-09-15

Add plugin version to GRPC interface (#17088) Add plugin version to GRPC interface Added a version interface in the sdk/logical so that it can be shared between all plugin types, and then wired it up to RunningVersion in the mounts, auth list, and database systems. I've tested that this works with auth, database, and secrets plugin types, with the following logic to populate RunningVersion: If a plugin has a PluginVersion() method implemented, then that is used If not, and the plugin is built into the Vault binary, then the go.mod version is used Otherwise, the it will be the empty string. My apologies for the length of this PR. * Placeholder backend should be external We use a placeholder backend (previously a framework.Backend) before a GRPC plugin is lazy-loaded. This makes us later think the plugin is a builtin plugin. So we added a `placeholderBackend` type that overrides the `IsExternal()` method so that later we know that the plugin is external, and don't give it a default builtin version.

2022-09-09

node status as a module to be importable by HCP cloud (#17089)

2022-05-20

Add autopilot automated upgrades and redundancy zones (#15521)

2022-05-17

secrets/database: adds ability to manage alternative credential types and configuration (#15376)

2022-04-14

VAULT-5422: Add rate limit for TOTP passcode attempts (#14864) * VAULT-5422: Add rate limit for TOTP passcode attempts * fixing the docs * CL * feedback * Additional info in doc * rate limit is done per entity per methodID * refactoring a test * rate limit OSS work for policy MFA * adding max_validation_attempts to TOTP config * feedback * checking for non-nil reference

2022-04-05

Adds Vault version prerelease and metadata to logical.PluginEnvironment (#14851)

2022-02-18

OSS Port: Activity log breakdowns (#14160) * Activity log breakdowns * s/path/namespace_path * Remove ent specific tests

2022-02-17

Login MFA (#14025) * Login MFA * ENT OSS segragation (#14088) * Delete method id if not used in an MFA enforcement config (#14063) * Delete an MFA methodID only if it is not used by an MFA enforcement config * Fixing a bug: mfa/validate is an unauthenticated path, and goes through the handleLoginRequest path * adding use_passcode field to DUO config (#14059) * add changelog * preventing replay attack on MFA passcodes (#14056) * preventing replay attack on MFA passcodes * using %w instead of %s for error * Improve CLI command for login mfa (#14106) CLI prints a warning message indicating the login request needs to get validated * adding the validity period of a passcode to error messages (#14115) * PR feedback * duo to handle preventing passcode reuse Co-authored-by: hghaf099 <83242695+hghaf099@users.noreply.github.com> Co-authored-by: hamid ghaf <hamid@hashicorp.com>

SSCT Tokens Feature [OSS] (#14109) * port SSCT OSS * port header hmac key to ent and generate token proto without make command * remove extra nil check in request handling * add changelog * add comment to router.go * change test var to use length constants * remove local index is 0 check and extra defer which can be removed after use of ExternalID

feature: multiplexing support for database plugins (#14033) * feat: DB plugin multiplexing (#13734) * WIP: start from main and get a plugin runner from core * move MultiplexedClient map to plugin catalog - call sys.NewPluginClient from PluginFactory - updates to getPluginClient - thread through isMetadataMode * use go-plugin ClientProtocol interface - call sys.NewPluginClient from dbplugin.NewPluginClient * move PluginSets to dbplugin package - export dbplugin HandshakeConfig - small refactor of PluginCatalog.getPluginClient * add removeMultiplexedClient; clean up on Close() - call client.Kill from plugin catalog - set rpcClient when muxed client exists * add ID to dbplugin.DatabasePluginClient struct * only create one plugin process per plugin type * update NewPluginClient to return connection ID to sdk - wrap grpc.ClientConn so we can inject the ID into context - get ID from context on grpc server * add v6 multiplexing protocol version * WIP: backwards compat for db plugins * Ensure locking on plugin catalog access - Create public GetPluginClient method for plugin catalog - rename postgres db plugin * use the New constructor for db plugins * grpc server: use write lock for Close and rlock for CRUD * cleanup MultiplexedClients on Close * remove TODO * fix multiplexing regression with grpc server connection * cleanup grpc server instances on close * embed ClientProtocol in Multiplexer interface * use PluginClientConfig arg to make NewPluginClient plugin type agnostic * create a new plugin process for non-muxed plugins * feat: plugin multiplexing: handle plugin client cleanup (#13896) * use closure for plugin client cleanup * log and return errors; add comments * move rpcClient wrapping to core for ID injection * refactor core plugin client and sdk * remove unused ID method * refactor and only wrap clientConn on multiplexed plugins * rename structs and do not export types * Slight refactor of system view interface * Revert "Slight refactor of system view interface" This reverts commit 73d420e5cd2f0415e000c5a9284ea72a58016dd6. * Revert "Revert "Slight refactor of system view interface"" This reverts commit f75527008a1db06d04a23e04c3059674be8adb5f. * only provide pluginRunner arg to the internal newPluginClient method * embed ClientProtocol in pluginClient and name logger * Add back MLock support * remove enableMlock arg from setupPluginCatalog * rename plugin util interface to PluginClient Co-authored-by: Brian Kassouf <bkassouf@hashicorp.com> * feature: multiplexing: fix unit tests (#14007) * fix grpc_server tests and add coverage * update run_config tests * add happy path test case for grpc_server ID from context * update test helpers * feat: multiplexing: handle v5 plugin compiled with new sdk * add mux supported flag and increase test coverage * set multiplexingSupport field in plugin server * remove multiplexingSupport field in sdk * revert postgres to non-multiplexed * add comments on grpc server fields * use pointer receiver on grpc server methods * add changelog * use pointer for grpcserver instance * Use a gRPC server to determine if a plugin should be multiplexed * Apply suggestions from code review Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com> * add lock to removePluginClient * add multiplexingSupport field to externalPlugin struct * do not send nil to grpc MultiplexingSupport * check err before logging * handle locking scenario for cleanupFunc * allow ServeConfigMultiplex to dispense v5 plugin * reposition structs, add err check and comments * add comment on locking for cleanupExternalPlugin Co-authored-by: Brian Kassouf <bkassouf@hashicorp.com> Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>

2022-01-26

Add remote_port in the audit logs when it is available (#12790) * Add remote_port in the audit logs when it is available The `request.remote_port` field is now present in the audit log when it is available: ``` { "time": "2021-10-10T13:53:51.760039Z", "type": "response", "auth": { "client_token": "hmac-sha256:1304aab0ac65747684e1b58248cc16715fa8f558f8d27e90fcbcb213220c0edf", "accessor": "hmac-sha256:f8cf0601dadd19aac84f205ded44c62898e3746a42108a51105a92ccc39baa43", "display_name": "root", "policies": [ "root" ], "token_policies": [ "root" ], "token_type": "service", "token_issue_time": "2021-10-10T15:53:44+02:00" }, "request": { "id": "829c04a1-0352-2d9d-9bc9-00b928d33df5", "operation": "update", "mount_type": "system", "client_token": "hmac-sha256:1304aab0ac65747684e1b58248cc16715fa8f558f8d27e90fcbcb213220c0edf", "client_token_accessor": "hmac-sha256:f8cf0601dadd19aac84f205ded44c62898e3746a42108a51105a92ccc39baa43", "namespace": { "id": "root" }, "path": "sys/audit/file", "data": { "description": "hmac-sha256:321a1d105f8c6fd62be4f34c4da4f0e6d1cdee9eb2ff4af0b59e1410950fe86b", "local": false, "options": { "file_path": "hmac-sha256:2421b5bf8dab1f9775b2e6e66e58d7bca99ab729f3f311782fda50717eee55b3" }, "type": "hmac-sha256:30dff9607b4087e3ae6808b4a3aa395b1fc064e467748c55c25ddf0e9b150fcc" }, "remote_address": "127.0.0.1", "remote_port": 54798 }, "response": { "mount_type": "system" } } ``` Closes https://github.com/hashicorp/vault/issues/7716 * Add changelog entry * Empty commit to trigger CI * Add test and explicit error handling * Change temporary file pattern in test

2021-11-30

Add "operator members" command to list nodes in the cluster. (#13292)

2021-10-15

Local aliases OSS patch (#12848) * Local aliases OSS patch * build fix

2021-10-14

Port: add client ID to TWEs in activity log [vault-3136] (#12820) * port for tracking twes as clients * comment clean up * changelog * change changelog entry phrasing

2021-10-07

Adds ability to define an inline policy and internal metadata on tokens (#12682) * Adds ability to define an inline policy and internal metadata to tokens * Update comment on fetchEntityAndDerivedPolicies * Simplify handling of inline policy * Update comment on InternalMeta Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com> * Improve argument name Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com> * Use explicit SkipIdentityInheritance token field instead of implicit InlinePolicy behavior * Add SkipIdentityInheritance to pb struct in token store create method * Rename SkipIdentityInheritance to NoIdentityPolicies * Merge latest from main and make proto Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>

Update plugin proto to send tls.ConnectionState (Op.2) (#12581)

2021-09-30

Update protobuf & grpc libraries and protoc plugins (#12679)

2021-09-17

Add Custom metadata field to alias (#12502) * adding changes * removing q.Q * removing empty lines * testing * checking tests * fixing tests * adding changes * added requested changes * added requested changes * added policy templating changes and fixed tests * adding proto changes * making changes * adding unit tests * using suggested function

2021-08-11

Add custom key metadata (#48) * set proto go_package to github.com/hashicorp/kv * include paths=source_relative in proto make target * Add CustomMetadata to KeyMetadata proto message * compile proto changes with CustomMetadata field * change protoc-gen-go version from v1.27.1-devel->v1.26.0 * add read/write of custom_metadata field to metadata path * add custom metadata input validation logic * perform POST/PUT custom_metadata validation prior to fetch from storage * Fix custom_metadata not being written to storage after refactor * Add custom_metadata and metadata merge behavior unit tests * Only overwrite custom_metadata if key provided in request * use quoted character format for validation errors * simplify custom metadata check for writing custom metadata to storage * early return in validateCustomMetadata for too many keys * include unquoted key in unprintable key error string

2021-07-15

Remove vendor, go mod tidy, update lib locations

This commit does not contain any .proto files.

Remove vendor, go mod tidy, update lib locations

This commit does not contain any .proto files.

Remove vendor, go mod tidy, update lib locations

Remove vendor, go mod tidy, update lib locations

This commit does not contain any .proto files.

Remove vendor, go mod tidy, update lib locations

This commit does not contain any .proto files.

2021-06-11

Remove vendor/ (RELENG-355) (#11544)

2021-05-03

Introduce Logical Unrecoverable Error, Use it in Expiration Manager (#11477) * build out zombie lease system * add typo for CI * undo test CI commit * time equality test isn't working on CI, so let's see what this does... * add unrecoverable proto error, make proto, go mod vendor * zombify leases if unrecoverable error, tests * test fix: somehow pointer in pointer rx is null after pointer rx called * tweaks based on roy feedback * improve zombie errors * update which errors are unrecoverable * combine zombie logic * keep subset of zombie lease in memory

2021-03-03

Autopilot: Server Stabilization, State and Dead Server Cleanup (#10856) * k8s doc: update for 0.9.1 and 0.8.0 releases (#10825) * k8s doc: update for 0.9.1 and 0.8.0 releases * Update website/content/docs/platform/k8s/helm/configuration.mdx Co-authored-by: Theron Voran <tvoran@users.noreply.github.com> Co-authored-by: Theron Voran <tvoran@users.noreply.github.com> * Autopilot initial commit * Move autopilot related backend implementations to its own file * Abstract promoter creation * Add nil check for health * Add server state oss no-ops * Config ext stub for oss * Make way for non-voters * s/health/state * s/ReadReplica/NonVoter * Add synopsis and description * Remove struct tags from AutopilotConfig * Use var for config storage path * Handle nin-config when reading * Enable testing autopilot by using inmem cluster * First passing test * Only report the server as known if it is present in raft config * Autopilot defaults to on for all existing and new clusters * Add locking to some functions * Persist initial config * Clarify the command usage doc * Add health metric for each node * Fix audit logging issue * Don't set DisablePerformanceStandby to true in test * Use node id label for health metric * Log updates to autopilot config * Less aggressively consume config loading failures * Return a mutable config * Return early from known servers if raft config is unable to be pulled * Update metrics name * Reduce log level for potentially noisy log * Add knob to disable autopilot * Don't persist if default config is in use * Autopilot: Dead server cleanup (#10857) * Dead server cleanup * Initialize channel in any case * Fix a bunch of tests * Fix panic * Add follower locking in heartbeat tracker * Add LastContactFailureThreshold to config * Add log when marking node as dead * Update follower state locking in heartbeat tracker * Avoid follower states being nil * Pull test to its own file * Add execution status to state response * Optionally enable autopilot in some tests * Updates * Added API function to fetch autopilot configuration * Add test for default autopilot configuration * Configuration tests * Add State API test * Update test * Added TestClusterOptions.PhysicalFactoryConfig * Update locking * Adjust locking in heartbeat tracker * s/last_contact_failure_threshold/left_server_last_contact_threshold * Add disabling autopilot as a core config option * Disable autopilot in some tests * s/left_server_last_contact_threshold/dead_server_last_contact_threshold * Set the lastheartbeat of followers to now when setting up active node * Don't use config defaults from CLI command * Remove config file support * Remove HCL test as well * Persist only supplied config; merge supplied config with default to operate * Use pointer to structs for storing follower information * Test update * Retrieve non voter status from configbucket and set it up when a node comes up * Manage desired suffrage * Consider bucket being created already * Move desired suffrage to its own entry * s/DesiredSuffrageKey/LocalNodeConfigKey * s/witnessSuffrage/recordSuffrage * Fix test compilation * Handle local node config post a snapshot install * Commit to storage first; then record suffrage in fsm * No need of local node config being nili case, post snapshot restore * Reconcile autopilot config when a new leader takes over duty * Grab fsm lock when recording suffrage * s/Suffrage/DesiredSuffrage in FollowerState * Instantiate autopilot only in leader * Default to old ways in more scenarios * Make API gracefully handle 404 * Address some feedback * Make IsDead an atomic.Value * Simplify follower hearbeat tracking * Use uber.atomic * Don't have multiple causes for having autopilot disabled * Don't remove node from follower states if we fail to remove the dead server * Autopilot server removals map (#11019) * Don't remove node from follower states if we fail to remove the dead server * Use map to track dead server removals * Use lock and map * Use delegate lock * Adjust when to remove entry from map * Only hold the lock while accessing map * Fix race * Don't set default min_quorum * Fix test * Ensure follower states is not nil before starting autopilot * Fix race Co-authored-by: Jason O'Donnell <2160810+jasonodonnell@users.noreply.github.com> Co-authored-by: Theron Voran <tvoran@users.noreply.github.com>

2021-02-18

Revert "Vault Dependency Upgrades [VAULT-871] (#10903)" (#10939) This reverts commit eb74ca61fc4dcb7038f39defb127d5d639ba0ca1.

Vault Dependency Upgrades [VAULT-871] (#10903) * upgrade vault dependency set * etcd and grpc issues: * better for tests * testing * all upgrades for hashicorp deps * kubernetes plugin upgrade seems to work * kubernetes plugin upgrade seems to work * etcd and a bunch of other stuff * all vulnerable packages upgraded * k8s is broken in linux env but not locally * test fixes * fix testing * fix etcd and grpc * fix etcd and grpc * use master branch of go-testing-interface * roll back etcd upgrade * have to fix grpc since other vendors pull in grpc 1.35.0 but we cant due to etcd * rolling back in the replace directives * a few more testing dependencies to clean up * fix go mod vendor

2020-10-29

Backport some OSS changes (#10267) * Backport some OSS changes * go mod vendor

2020-10-23

Merge PR #10192: Auto-Join: Configurable Scheme & Port (and add k8s provider)

2020-10-15

DBPW - Copy `newdbplugin` package to `dbplugin/v5` (#10151) This is part 1 of 4 for renaming the `newdbplugin` package. This copies the existing package to the new location but keeps the current one in place so we can migrate the existing references over more easily.

2020-10-13

Merge PR #10095: Integrated Storage Cloud Auto-Join

2020-09-18

[DBPW 4/X] Update DB engine to support v4 and v5 interfaces with password policies (#9878)

2020-09-15

Run CI tests in docker instead of a machine. (#8948)

2020-09-08

Merge activity log work to date on enterprise back into oss. (#9900) * Added stub class for activity logging. (#1435) * Define activity fragments and starter methods for manipulating them. (#1441)

2020-08-28

Add Database v5 interface with gRPC client & server (#9641) * Add new Database v5 interface with gRPC client & server This is primarily for making password policies available to the DB engine, however since there are a number of other problems with the current interface this is getting an overhaul to a more gRPC request/response approach for easier future compatibility. This is the first in a series of PRs to add support for password policies in the combined database engine

2020-07-21

Adds gsuite provider specific extension for fetching groups and user information (#123)

2020-07-09

Backport the pieces of the replication API changes (#9425)

2020-07-01

Update go-kms-deps & run go mod vendor (#9366)

2020-06-26

Stepwise docker env (#9292) * add first stepwise test env, Docker, with example transit test * update transit stepwise test * add other tests that use stepwise * cleanup test, make names different than just 'transit' * return the stderr if compile fails with error * minor cleanups * minor cleanups * go mod vendor * cleanups * remove some extra code, and un-export some fields/methods * update vendor * remove reference to vault.CoreConfig, which really wasn't used anyway * update with go mod vendor * restore Precheck method to test cases * clean up some networking things; create networks with UUID, clean up during teardown * vendor stepwise * Update sdk/testing/stepwise/environments/docker/environment.go haha thanks :D Co-authored-by: Michael Golowka <72365+pcman312@users.noreply.github.com> * Update sdk/testing/stepwise/environments/docker/environment.go Great catch, thanks Co-authored-by: Michael Golowka <72365+pcman312@users.noreply.github.com> * fix redundant name * update error message in test * Update builtin/credential/userpass/stepwise_test.go More explicit error checking and responding Co-authored-by: Michael Golowka <72365+pcman312@users.noreply.github.com> * Update builtin/logical/aws/stepwise_test.go `test` -> `testFunc` Co-authored-by: Michael Golowka <72365+pcman312@users.noreply.github.com> * Update builtin/logical/transit/stepwise_test.go Co-authored-by: Michael Golowka <72365+pcman312@users.noreply.github.com> * fix typos * update error messages to provide clarity * Update sdk/testing/stepwise/environments/docker/environment.go Co-authored-by: Michael Golowka <72365+pcman312@users.noreply.github.com> * update error handling / collection in Teardown * panic if GenerateUUID returns an error * Update sdk/testing/stepwise/environments/docker/environment.go Co-authored-by: Michael Golowka <72365+pcman312@users.noreply.github.com> * Update builtin/credential/userpass/stepwise_test.go Co-authored-by: Calvin Leung Huang <cleung2010@gmail.com> * Update builtin/logical/aws/stepwise_test.go Co-authored-by: Calvin Leung Huang <cleung2010@gmail.com> * Update builtin/logical/transit/stepwise_test.go Co-authored-by: Calvin Leung Huang <cleung2010@gmail.com> * Update sdk/testing/stepwise/environments/docker/environment.go Co-authored-by: Calvin Leung Huang <cleung2010@gmail.com> * import ordering * standardize on dc from rc for cluster * lowercase name * CreateAPIClient -> NewAPIClient * testWait -> ensure * go mod cleanup * cleanups * move fields and method around * make start and dockerclusternode private; use better random serial number * use better random for SerialNumber * add a timeout to the context used for terminating the docker container * Use a constant for the Docker client version * rearrange import statements Co-authored-by: Michael Golowka <72365+pcman312@users.noreply.github.com> Co-authored-by: Calvin Leung Huang <cleung2010@gmail.com>

2020-05-27

Add password policy support (#12) * Allows users to specify a password policy to generate passwords from rather than hard coded to base62 * If no policy is specified, it defaults back to base62 * The PasswordLength field has been deprecated. If both the length and a policy are specified, the configuration will error on save.

Add user configurable password policies available to secret engines (#8637) * Add random string generator with rules engine This adds a random string generation library that validates random strings against a set of rules. The library is designed for use as generating passwords, but can be used to generate any random strings.

2020-05-22

Bump go-kms-wrapping to remove proto warning, and vendor (#9066)

2020-05-14

Update to latest go-kms-wrapping and fix protos/etcd (#8996)

2020-04-27

Add helper for aliasmetadata and add to AWS auth (#8783) * add aliasmetadata sdk helper and add to aws auth * split into ec2_metadata and iam_metadata fields * fix tests * strip pointer * add test of default metadata * more test <3 * switch from interface to custom marshallers * add tests for marshalling * store nil when selected fields are default * separate loop into pieces * separate acc test into multiple * Update builtin/credential/aws/path_login.go Co-Authored-By: Jim Kalafut <jkalafut@hashicorp.com> * changes from feedback * update aws test * refactor to also populate auth metadata * update how jsonification is tested * only add populated metadata values * add auth_type to ec2 logins Co-authored-by: Jim Kalafut <jkalafut@hashicorp.com>

2020-02-17

Update api and sdk deps

Update api and sdk deps

Update api and sdk deps

Update sdk dep

2020-02-16

Update api and sdk deps