Proto commits in unkeyed/unkey

These 32 commits are when the Protocol Buffers files have changed:

2025-04-08

chore: clean up after changing ratelimit implementation (#3081) * chore: clean up after changing ratelimit implementation * revert: css * test: remove super long tests

The documentation is generated from this commit.

2025-03-18

ratelimit clustering (#2986) * ci: release agent * perf: add permission and key caches * test: run clustered ratelimit tests * [autofix.ci] apply automated fixes --------- Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>

2025-02-10

service discovery (#2882) * wip * wip * wip * wip * fix: feedback

2025-02-07

fix: transaction deadlock in trpc (#2843) * fix: transaction deadlock in trpc Fixes transaction deadlocks around audit log buckets. This moves the workspace fetch to the beginning of the trpc handler and loads the bucket id once, to be reused. In theory we can now cache the workspace lookup but I'm not sure if Next.js does that automatically now or if we need to do it. * fix: load the correct workspace id * fix: race condition with deleting permissions * fix: race condition with deleting roles * fix: use correct id * fix: remove duplicate query * feat: add error log * fix: populate workspace * fix: populate workspace * fix: remove opt in * [autofix.ci] apply automated fixes * fix: remove opt in * revert: timeout * revert: name * revert: name * revert: name --------- Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>

2024-08-30

feat: sliding window (#2021) * fix(job_test_api_local.yaml): add TEST_LOCAL environment variable fix(go.mod): remove unused github.com/shirou/gopsutil/v4 dependency fix(go.sum): remove github.com/shirou/gopsutil/v4 dependency fix(fixed_window.go): change buildKey function to public BuildKey function feat(sliding_window.go): add sliding_window ratelimit implementation refactor(flush_push_pull.go): remove unused ratelimitNodeKey function refactor(ratelimit.go): add randomization to ratelimit check for testing fix(tests): update test cases for ratelimit accuracy tests fix(ecs): update watchPaths in flightcontrol.json for agent configuration * fix(ratelimit): update BuildKey function to include duration in key fix(ratelimit): fix function calls to BuildKey in sliding_window.go fix(ratelimit): update BuildKey function to include duration in commit_lease.go fix(ratelimit): update BuildKey function to include duration in flush_push_pull.go fix(ratelimit): update BuildKey function to include duration in ratelimit.go * wip: sliding window ratelimiting * [autofix.ci] apply automated fixes * refactor(ratelimit_test.go): remove unused imports and variables feat(ratelimit_test.go): add new test for accuracy with fixed time and cluster size of 9 * chore(agent): fix typo in .golangci.yaml comment feat(agent): add logging of port information when starting service chore(agent): update dependencies in go.mod file feat(agent): update connectrpc.com/otelconnect to v0.7.1 feat(agent): update github.com/aws/aws-sdk-go-v2 to v1.30.4 feat(agent): update github.com/aws/aws-sdk-go-v2/config to v1.27.28 feat(agent): update github.com/aws/aws-sdk-go-v2/credentials to v1.17.28 feat(agent): update github.com/aws/aws-sdk-go-v2/service/s3 to v1.60.0 feat(agent): update github.com/axiomhq/axiom-go to v0.20.2 feat(agent): update github.com/danielgtaylor/huma/v2 to v2.22.1 feat(agent): update github.com/grafana/pyroscope-go to v1.1.2 feat(agent): update github.com/maypok86/otter to v1 chore(agent): update dependencies in go.mod file chore(agent): update dependencies feat(agent): update otelconnect to v0.7.1 feat(agent): update mergo to v1.0.1 feat(agent): update go-fuzz-headers to v0.0.0-20240806141605-e8a1dd7889d6 feat(agent): update go-118-fuzz-build to v0.0.0-20230306123547-8075edf89bb0 feat(agent): update survey/v2 to v2.3.7 feat(agent): update semver/v3 to v3.2.1 feat(agent): update hcsshim to v0.12.6 feat(agent): update go-expect to v0.0.0-20220104043353-73e0943537d2 feat(agent): update logrus-bugsnag to v0.0.0-201712042 chore(agent): update dependencies for aws-sdk-go-v2 to version 1.30.4 chore(agent): update dependencies for various aws-sdk-go-v2 packages to latest versions chore(agent): update dependencies feat(agent): add new dependencies for bugsnag and cloudflare/cfssl feat(agent): add new dependencies for codahale/rfc6979 and compose-spec/compose-go/v2 feat(agent): add new dependencies for containerd/cgroups, containerd/console, containerd/containerd, containerd/nydus-snapshotter, containerd/stargz-snapshotter, containerd/ttrpc, containerd/typeurl/v2 chore(go.sum): update dependencies versions for docker-related packages chore(agent): update go.sum with new dependencies for go-testdb, fsnotify, and fvbommel/sortorder feat(agent): add support for go-testdb, fsnotify, and fvbommel/sortorder dependencies in the agent application feat(go.sum): add new dependencies for jsonpointer, jsonreference, swag, mysql, slim-sprig, mapstructure, flock, googleapis, certificate-transparency-go, gnostic-models, go-cmp, go-querystring, gofuzz chore(agent): update dependencies for google/pprof, google/shlex, gorilla/mux, gorilla/websocket, grafana/pyroscope-go, and hashicorp/go-cleanhttp feat(agent): add new versions for google/pprof, google/shlex, gorilla/mux, gorilla/websocket, grafana/pyroscope-go, and hashicorp/go-cleanhttp feat(agent): add new version for hashicorp/go-version and hashicorp/hcl feat(go.sum): add new dependencies for vt10x, mergo, in-toto-golang, tdigest, gorm, inflection, now, clockwork, intern, loggo, go-shellquote and influxdata chore(agent): update dependencies in go.sum file feat(agent): add new dependencies for godebug, lib/pq, plan9stats, properties, easyjson, and go-colorable chore(agent): update dependencies in go.sum file chore(agent): update go.sum with new versions of prometheus libraries feat(agent): add support for prometheus/client_golang v1.1.0 and v1.20.1 feat(agent): add support for prometheus/client_model v0.6.1 feat(agent): add support for prometheus/common v0.55.0 and v0.6.0 feat(agent): add support for prometheus/procfs v0.15.1 feat(agent): add support for redis/go-redis/v9 v9.6.1 feat(agent): add support for rivo/uniseg v0.4.7 feat(agent): add support for rs/dnscache v0.0.0-20230804202142-fc85eb664529 feat(go.sum): add new dependencies for secure-systems-lab/go-securesystemslib, serialx/hashring, shibumi/go-pathspec, sirupsen/logrus, skratchdot/open-golang, spdx/tools-golang, spf13/afero, spf13/cast, spf13/viper, streadway/quantile, and stretchr/testify chore(agent): update dependencies feat(agent): add new version of gotenv library feat(agent): update testcontainers-go to version 0.33.0 feat(agent): add new modules for testcontainers-go feat(agent): add new version of notary library feat(agent): add new version of tilt-dev/fsnotify library feat(agent): add new version of tilt-dev/go-csvvalue library feat(agent): add new version of tilt-dev/units library feat(agent): add new version of tilt-dev/vt100 library feat(agent): add new version of tsenart/vegeta/v12 library feat(agent): update urfave/cli/v2 to version 2.27.4 feat(agent): add new version of vbatts/tar-split library feat(agent): add new version of x448/float16 library feat(agent): add new version of yuin/goldmark library feat(agent): add new version of feat(agent): update dependencies for opencensus and opentelemetry packages feat(agent): add new versions for opencensus and opentelemetry packages feat(agent): add new goleak and mock packages versions chore(agent): update go.sum with new versions of golang.org/x/crypto dep and golang.org/x/mod dep feat(agent): add golang.org/x/crypto v0.0.0-20210921155107-089bfa567519 feat(agent): add golang.org/x/crypto v0.0.0-20200302210943-78000ba7a073 feat(agent): add golang.org/x/crypto v0.0.0-20201117144127-c1f2f97bffc9 feat(agent): add golang.org/x/crypto v0.0.0-20210921155107-089bfa567519 feat(agent): add golang.org/x/crypto v0.0.0-20200302210943-78000ba7a073 feat(agent): add golang.org/x/crypto v0.0.0-20201117144127-c1f2 chore(agent): update go.sum file with latest golang.org/x/net and golang.org/x/oauth2 versions chore(agent): update go.sum file with latest versions of dependencies feat(sys): add new versions of golang.org/x/sys in go.sum file feat(term): update golang.org/x/term versions in go.sum file chore(agent): update go.sum file with new versions of dependencies feat(agent): update golang.org/x/text to v0.4.0 feat(agent): update golang.org/x/text to v0.17.0 feat(agent): update golang.org/x/time to v0.6.0 feat(agent): update golang.org/x/tools to v0.24.0 feat(agent): add new dependencies gonum.org/v1/gonum and gonum.org/v1/netlib feat(agent): update google.golang.org/genproto and google.golang.org/grpc versions chore(agent): update dependencies feat(agent): add gopkg.in/airbrake/gobrake.v2 and gopkg.in/cenkalti/backoff.v1 feat(agent): add gopkg.in/cenkalti/backoff.v2 and gopkg.in/gemnasium/logrus-airbrake-hook.v2 feat(agent): add gopkg.in/inf.v0 and gopkg.in/ini.v1 feat(agent): add gopkg.in/rethinkdb/rethinkdb-go.v6 and update gopkg.in/ini.v1 feat(agent): update gopkg.in/yaml.v2 to v2.4.0 feat(agent): add new dependencies for Kubernetes client and utilities feat(tests): add new test for ratelimits accuracy in cluster/docker package feat(tests): rename TestIdentitiesShareRatelimits to TestClusterRatelimitAccuracy refactor(tests): update port handling in cluster, harness, and axiom tests feat(containers): add NewAgent function to start Agent containers feat(containers): add ComposeUp function to start docker-compose stack feat(containers): update NewRedis function to skip reaper in test container refactor(s3.go): remove unnecessary comments and whitespace feat(s3.go): add support for InternalURL field in S3 struct feat(s3.go): update NewS3 function to accept network configurations feat(s3.go): add logging for S3 container name and IP address chore(ratelimit_real_time_test.go): remove real-time ratelimit test file test(ratelimit_test.go): update test cases for accuracy and cluster size test(sliding_window_test.go): refactor testAccuracy function for better accuracy feat(s3.go): add logging for S3 bucket creation with S3 URL information * [autofix.ci] apply automated fixes * feat: mitigate broadcast when a window is full * chore(pr.yaml): update workflow to install dependencies and run tests for agent integration * fix(pr.yaml): specify agent_lb service when running docker compose up command * refactor(ratelimits_test.go): remove inaccurate test case with limit 1 and 1 second duration * test(identities): use require.NotEmpty for environment variables in test files * chore(agent): fix typo in inamedparam linter setting comment * chore(pr.yaml): update go test command to include specific integration tests for identities and cluster features * chore(pr.yaml): update 'go test' command to include verbose output * chore(pr.yaml): comment out test_agent_integration job for now --------- Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>

The documentation is generated from this commit.

2024-08-08

feat: leases and fixes (#2013) * feat: leases and fixes * feat(api): add handler for v1 ratelimit commitLease endpoint feat(api): add test for commitLease handler feat(testutil): add test harness for API testing feat(ratelimit): add tests for fixed window ratelimit implementation feat(ratelimit): add CommitLease function to ratelimit service feat(nginx): add nginx configuration for agent and api services * feat(agent): add v1RatelimitCommitLease route to agent API for commit lease functionality * refactor(agent.go): remove unnecessary empty line feat(client.ts): improve logging for sync rate limiting conditions

2024-08-06

feat(proto): add Pass field to PushPullEvent message for tracking ratelimit decision chore(proto): update proto file with new fields and messages feat(proto): add pass field to PushPullEvent message for ratelimit tracking feat(metrics): add ratelimitAccuracy metric to track ratelimit accuracy feat(pushpull): update ratelimitAccuracy metric based on ratelimit pass result

2024-07-23

chore: latency metrics for pushpull

2024-07-19

feat(gossip.proto): add JoinRequest, JoinResponse, LeaveRequest, LeaveResponse message types chore(gossip.pb.go): update generated Go code for Gossip service feat(gossip.pb.go): add new message types for JoinRequest, JoinResponse, LeaveRequest, LeaveResponse, and LeaveResponse feat(gossip.pb.go): add support for self field in JoinRequest, LeaveRequest, and LeaveRequest feat(gossip.pb.go): add support for LeaveRequest message type feat(gossip.pb.go): add support for LeaveResponse message type feat(gossip.pb.go): add support for LeaveResponse message type feat(gossip.pb.go): add support for LeaveResponse message type feat(gossip.pb.go): add support for LeaveResponse message type feat(gossip.pb.go): add support for GossipService Ping method feat(gossip.pb.go): add support for PingRequest and PingResponse message types feat(gossip.pb.go): add support for Join method feat(gossip.pb.go): add support for JoinRequest and JoinResponse message types chore(gossip.pb.go): update number of message types to 14 in gossip.proto feat(gossip.pb.go): add JoinRequest, JoinResponse, LeaveRequest, and LeaveResponse message types to gossip.proto chore(gossip): add Join and Leave procedures to GossipService feat(cache): change ReportCacheHealth to Record in memory cache feat(cache): change ReportCacheHit to Record in metrics middleware refactor(ratelimit): remove logging of latency in ratelimit functions refactor(service): update comments in Listen method regarding timeouts feat(gossip): implement cluster package for managing gossip communication feat(gossip): add cluster struct with methods for managing cluster members and communication feat(gossip): implement Config struct with default values for gossip configuration feat(gossip): add New function to create a new cluster instance with provided config feat(gossip): implement run method to start the cluster's gossip loop and background tasks feat(gossip): add methods for joining, shutting down, and subscribing to cluster events feat(gossip): implement randomPeers method to select random peers for gossip communication feat(gossip): add methods to add and remove members from the cluster state feat(gossip): implement gossip method for exchanging membership information with peers feat(gossip): add clusterServer struct to handle gossip service feat(gossip): implement CreateHandler method in clusterServer feat(gossip): implement Serve method in clusterServer feat(gossip): implement Join method in clusterServer feat(gossip): implement Leave method in clusterServer feat(gossip): implement Ping method in clusterServer feat(gossip): implement IndirectPing method in clusterServer feat(gossip): implement SyncMembers method in clusterServer feat(gossip): add Member struct with Hash method feat(gossip): add Cluster interface with SubscribeJoinEvents and SubscribeLeaveEvents feat(gossip): add tests for cluster membership and event emission feat(gossip): add rpc.go file with functions for handling join, leave, ping, indirectPing, and syncMembers in the gossip package feat(gossip): add test for membership changes propagation to hash ring feat(gossip): add test utility function to simulate failure in cluster fix(load): change variable name from SystemLoadReport to SystemLoad fix(metrics): refactor axiom methods to use Record instead of specific reports feat(metrics): add Metric interface for all metrics to implement feat(proto): add Join and Leave RPC methods to GossipService feat(scripts): add deploy script for scaling regions in fly platform refactor(ratelimit): remove unused time tracking code in service methods refactor(ratelimit): update ratelimit fields to use integer type instead of number feat(api): update key schema fields to use integer type instead of number feat(api): update ratelimit fields to use integer type instead of number fix(routes): specify that limit and duration fields should be integers

2024-07-14

test: cluster

2024-07-03

test: multiratelimit (#1852)

test: multiratelimit

2024-07-02

test: membership (#1844)

feat: batch ratelimit syncs per peer (#1842)

2024-06-28

chore: remove old code

2024-06-27

feat: use hashring for ratleimits (#1791) * feaT: use hashring for ratleimits * refactor(Ratelimit.bru): update Ratelimit service URL to use port 8081 refactor(Ratelimit.bru): update bearer token to 'agent-auth-secret' refactor(agent.go): add logging of environment variables refactor(agent.go): update Cluster initialization with AuthToken refactor(agent.go): update Ratelimit service initialization with Cluster refactor(agent.go): update Port type to string in configuration refactor(agent.go): update Cluster initialization with AuthToken refactor(agent.go): update Cluster initialization with AuthToken refactor(agent.go): update Cluster initialization with AuthToken refactor(agent.go): update Cluster initialization with AuthToken refactor(agent.go): update Cluster initialization with AuthToken refactor(agent.go): update Cluster initialization with AuthToken refactor(agent.go): update Cluster initialization with AuthToken refactor(agent.go): update Cluster initialization with AuthToken refactor(agent.go): update Cluster initialization with AuthToken fix(ratelimit.go): remove redundant empty lines fix(fixed_window.go): change count variable type from int32 to int64 feat(fixed_window.go): add SetCurrent method to update current count feat(interface.go): add SetCurrentRequest struct fix(token_bucket.go): change remaining, max, refillRate, and refillInterval types to int64 feat(token_bucket.go): change newTokenBucket parameters to int64 feat(ring.go): remove 'tokens' field from logger.Info() call feat(service.proto): add PushPull rpc description fix(schema.json): change port type to string and remove format field feat(service.go): add pushPullEvent struct and pushPullC channel feat(service.go): initialize pushPullC channel and runPushPullSync goroutine feat(sync.go): add ratelimitNodeKey and runPushPullSync methods fix(docker-compose.gossip.yaml): update AGENT_CONFIG_FILE and RPC_ADDR values feat(deployment): add agent_1, agent_2, and agent_3 services feat(deployment): update ports for agent_1, agent_2, and agent_3 services feat(deployment): add environment variables for NODE_ID, RPC_ADDR, and REDIS_URL feat(deployment): add redis service with port mapping 6379:6379 feat(deployment): update AGENT_URL and TINYBIRD_PROXY_URL for api service * feat: ratelimits pass tests * feat(api): add accuracy and consistency tests for v1_ratelimit_limit route * fix: cluster config * chore(lint.yaml): add Go vet step to lint workflow for agent app * fix: docker networking * chore(workflows): add step to dump docker logs before uploading feat(workflows): change artifact path to docker.logs for upload step * fix(workflows): update docker compose command to include --no-color flag * chore(deployment): update AGENT_URL to use http://agent_2:8080 chore(deployment): update TINYBIRD_PROXY_URL to use http://agent_3:8080 * chore(workflows): remove unused TEST_LOCAL environment variable chore(tests): remove conditional skipping of test based on TEST_LOCAL variable * fix(workflows): add TEST_LOCAL environment variable to job_test_api_local.yaml feat(agent): add config.docker.json to Dockerfile for agent service feat(agent): log environment variables in agent.go feat(agent): add config.docker.json for agent service fix(api): update test to skip based on TEST_LOCAL environment variable feat(deployment): update agent service in docker-compose.yaml to use config.docker.json feat(deployment): update agent service in docker-compose.yaml to have 3 replicas feat(tools): update startContainers function in main.ts to use "agent" service

2024-06-21

feat: gossip (#1789) * feat: gossip * fix(agent): update logger configuration to include full config object fix(agent): update variable names and configuration structure for cluster settings fix(agent): update cluster configuration structure and usage in the code fix(agent): update cluster configuration structure in JSON files fix(agent): update dependencies in go.mod and go.sum files refactor(cluster): change variable name 'nodes' to 'ring' for clarity feat(cluster): add support for RpcAddr in Config struct feat(cluster): add support for RpcAddr in Membership struct feat(cluster): add support for RpcAddr in Tags struct feat(ring): add ring package with Node, Config, Token, and Ring structs feat(ring): implement New, AddNode, RemoveNode, hash, Members, and FindNode feat(ring): add ring_test.go with TestRing function for unit testing feat(schema.json): update schema to include clusters.join and clusters.serfAddr * refactor(cluster.go): modify Join method to exclude self from the list of addresses * fix(agent): update run function to handle Join method correctly feat(agent): add service for connect.NewClusterServer feat(agent): add new proto files for cluster v1 service and openapi definition feat(proto): add cluster v1 service definition with JoinClusterRequest, JoinClusterResponse, LeaveClusterRequest, and LeaveClusterResponse types chore(proto): update protoc-gen-go version to v1.34.2 in service.pb.go files fix(cluster.go): change Join method signature to return clusterSize and err feat(cluster.go): add new file cluster.go for connect package test(serf_test.go): update Region value to "test" feat(service.proto): add proto definition for RatelimitService refactor(schema.json): remove unnecessary newlines and fix formatting * refactor(agent.go): remove unnecessary logging of 'configFile' and 'config' details * fix(cluster.go): improve error message when failing to join serf cluster * fix(serf.go): add check to ensure Membership was started before shutting down * fix(serf.go): fix error message capitalization for consistency fix(serf.go): add check for nil serf instance before leaving feat(serf.go): add logging when creating serf instance * fix(serf.go): add error logging for failed address resolution and tag conversion * chore(agent): update serfAddr and rpcAddr in config.gossip.json to include port numbers * refactor(cluster): rename RatelimitService to ClusterService feat(cluster): add support for ClusterService in client and handler implementations refactor(proto): rename RatelimitService to ClusterService feat(proto): update method names in ClusterService to JoinCluster and LeaveCluster fix(proto): update package go_package paths for vault and cluster services fix(cluster): update UnimplementedRatelimitServiceHandler to UnimplementedClusterServiceHandler fix(cluster): update CreateHandler to return NewClusterServiceHandler feat(proto): rename RatelimitService to ClusterService in service.proto * fix(agent): change variable name from SerfAddr to RedisUrl in agent.go fix(agent): remove unused SerfAddr field and update RedisUrl field in config feat(agent): add github.com/redis/go-redis/v9 v9.5.3 to go.mod feat(agent): add github.com/cespare/xxhash/v2 v2.2.0 to go.mod feat(agent): add github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f to go.sum refactor(cluster.go): change membership field type to Membership interface feat(cluster.go): add shutdownCh field for graceful shutdown handling feat(interface.go): add Membership interface and Member struct feat(redis.go): implement Membership interface for Redis-based membership chore(serf.go): remove unused membership package and test file * fix(redis.go): refactor redis client creation to use parsed URL options refactor(uid.go): remove unused byteLength parameter from New function refactor(uid.go): simplify Node function by removing byteLength parameter * fix(agent.go): set cfg.NodeId based on HOSTNAME environment variable or generate it using uid.IdFromHash function feat(uid/hash.go): add IdFromHash function to generate unique IDs from a hash * fix(redis.go): set default database to 0 in Redis client options fix(hash.go): encode hash to base64 before converting to string * fix(redis.go): set InsecureSkipVerify to true in TLSConfig for Redis connection * feat(redis.go): add logging when connecting to redis server * chore(agent): update go.mod to add github.com/btcsuite/btcutil v1.0.2 chore(agent): remove indirect dependencies from go.mod file chore(agent): update go.sum with new dependencies feat(agent): add new dependencies for siphash, btcd, btclog, btcutil, go-spew, fsnotify, goleveldb, snappy-go, websocket, winsvc, and davecgh chore(agent): update go.sum file with new dependencies feat(agent): add new dependency github.com/hpcloud/tail v1.0.0 feat(agent): add new dependency github.com/jessevdk/go-flags v1.4.0 feat(agent): add new dependency github.com/jrick/logrotate v1.0.0 feat(agent): add new dependency github.com/kkdai/bstream v0.0.0-20161212061736-f391b8402d23 chore(agent): update go.sum with new versions of ginkgo and gomega chore(agent): add new version of golang.org/x/crypto to go.sum chore(agent): update go.sum with new dependencies for fsnotify and tomb packages refactor(uid): switch from base64 encoding to base58 encoding for UID generation * fix(config.gossip.json): update rpcAddr port from 10001 to 8080 feat(cluster.go): add heartbeat function to send HTTP requests to cluster members * feat(cluster.go): add debug mode with periodic heartbeat functionality * refactor(cluster.go): add condition to skip self in heartbeat loop refactor(redis.go): remove unused tls configuration in New function * fix(agent.go): change environment variable from HOSTNAME to FLY_PRIVATE_IP fix(config.gossip.json): update rpcAddr to use FLY_MACHINE_ID and FLY_APP_NAME feat(fly.toml): add fly.toml configuration file for unkey-agent-dev app refactor(cluster.go): improve logging in heartbeat function refactor(schema.json): simplify required fields structure feat(benchmarks.ts): add middleware for benchmarking requests feat(index.ts): export benchmarks middleware refactor(worker.ts): add benchmarks middleware to app middleware stack * chore(pr.yaml): rename api_integration_local.yaml to job_test_api_local.yaml * build(workflows): add 'ts: true' flag to the Install step in build.yaml, lint.yaml, and unit_test.yaml workflows * fix(docs): correct typo in listKeys API reference link in recovering-keys.mdx

2024-06-14

feat: improve configuration and copy vault code (#1750) * feat: improve configuration and copy vault code * fix(config): remove unnecessary whitespace in logging object fix(config): remove unkey.json file feat(config): add AGENT_CONFIG_FILE and PORT env variables to flightcontrol.js feat(config): add AGENT_CONFIG_FILE and PORT env variables to flightcontrol.json * fix(agent): update logger and header middleware to use REGION environment variable feat(flightcontrol): add REGION environment variable to config for each region * feat(flightcontrol.js): add preview environment as an exact copy of us-east-1 region with specific configurations feat(flightcontrol.json): add preview environment configuration with specific settings for the Agent service * chore(flightcontrol.json): simplify watchPaths array by removing unnecessary line breaks and spaces

2024-06-11

fix(Dockerfile): update CMD to enable ratelimit service feat(agent): add support for enabling ratelimit service in agent command docs(ratelimit): update OpenAPI schema to include cost field feat(proto): add Cost field to RatelimitRequest struct feat(proto): add GetCost method to RatelimitRequest struct fix(proto): correct field name from 'duracion' to 'cost' in service.proto feat(proto): add support for 'cos' field in service.proto fix(auth): add fmt.Println statements for authToken and authorizationHeader fix(ratelimit): add tokens parameter to bucket.take method feat(server): update AddService method to log pattern when adding service feat(server): update Listen method to use s.mux instead of creating new mux feat(ratelimit): add Cost field to RatelimitRequest struct feat(ratelimit): log req and res in Ratelimit method of Service

2024-06-10

feat: playing with errors

2024-06-02

feat: wip (#1710) * feat: wip * feat(benchmarks.ts): add platform field with value "koyeb" to metrics feat(index.ts): add platform field to metricSchema with type z.string() * fix(benchmarks.ts): add country field to metrics.emit function feat(index.ts): add country field to metricSchema for "metric.ratelimit.latency" * refactor(agent.go): update import paths to reflect new package structure

2024-05-30

feat: migration queue (#1697) * feat(api): queued migrations * feat(key_migrations.ts): add keyMigrations and keyMigrationErrors schemas feat(key_migrations.ts): define relations for keyMigrations and keyMigrationErrors * chore(wrangler.toml): enable consumers for preview, canary, and production environments * feat(api): queued migrations * fix(routes): remove unnecessary console.error statement fix(routes): correct endpoint URL from /v1/migrations.createKeys to /v1/migrations.enqueueKeys fix(tests): update expected status codes to 202 instead of 200 * chore(wrangler.custom.toml): add queue configuration for key-migrations-production feat(wrangler.custom.toml): add producer configuration for key-migrations-production

2024-05-21

style: fmt (#1659)

2024-05-20

feat: vault (#1656) * chore remove Unkeyed, Inc from 404 * feat: vault wip * fix(Dockerfile): remove unnecessary go install command feat(Dockerfile): update CMD to include "agent" argument feat(Encrypt.bru): add Authorization header with Bearer token feat(Liveness.bru): add Liveness endpoint configuration feat(agent.go): add flag to specify .env file path refactor(proto): rename EncryptedDEK to DataEncryptionKey feat(proto): add Key field to DataEncryptionKey message feat(proto): add EncryptedDataEncryptionKey message feat(proto): add Encrypted field to EncryptedDataEncryptionKey message feat(proto): add KeyEncryptionKey message feat(proto): add GetKey method to DataEncryptionKey message feat(proto): add GetEncrypted method to EncryptedDataEncryptionKey message feat(proto): add GetKey method to KeyEncryptionKey message refactor(proto): remove unused EncryptionKey methods and descriptors feat(proto): add new fields to EncryptionKey message refactor(object.pb.go): remove EncryptionKey message type and update message dependencies refactor(proto): update field names in proto messages for consistency and clarity refactor(vault): update message types indices in service.pb.go feat(vault): add support for GetKeyring method in ReEncryptRequest struct feat(vault): add support for ReEncryptDEKsRequest and ReEncryptDEKsResponse types in service.pb.go fix(proto): fix typo in 'LivenessRequest' message name fix(proto): fix typo in 'LivenessResponse' message name fix(proto): fix typo in 'DecryptRequest' message name fix(proto): fix typo in 'DecryptResponse' message name refactor(service.pb.go): reorder message types to match updated service methods fix(service.pb.go): correct message type export for service methods refactor(vaultv1connect): add Liveness method to VaultServiceClient interface feat(vaultv1connect): implement Liveness method in vaultServiceClient feat(vaultv1connect): add Liveness handler to NewVaultServiceHandler chore(vault): update dependencies in go.sum file feat(vault): add support for github.com/creack/pty package feat(vault): remove unused dependencies from go.sum file feat(vault): add support for github.com/pkg/diff package chore(vault): remove unused dependencies from go.sum feat(vault): update key variable names in coldstart_test.go and migrate_deks_test.go feat(vault): add authorization logic in auth package feat(vault): add authorization check in service methods feat(vault): add keyring package with CreateKey and DecodeAndDecryptKey functions feat(keyring): add functions to encrypt and encode data encryption keys feat(keyring): add functions to get, create, and roll keys in keyring feat(service): add functions to create, decrypt, and encrypt data encryption keys refactor(encryption.go): remove encryption functionality from service package feat(get_dek.go): add getDEK method to retrieve Data Encryption Key feat(reencrypt.go): implement ReEncrypt method for decrypting and re-encrypting data feat(roll_deks.go): add RollDeks method to rotate Data Encryption Keys refactor(service.go): refactor New function to load master keys and create keyring feat(service.go): add keyring package and integrate with Service struct feat(service.go): add loadMasterKeys function to load encryption and decryption keys feat(interface.go): rename ListObjects to ListObjectKeys for consistency fix(storage): update PutObject method parameter name from 'data' to 'b' fix(storage): update GetObject method to handle object not found case fix(storage): update ListObjectKeys method to correctly filter object keys feat(storage): add support for S3 storage implementation feat(storage): add support for creating S3 bucket if necessary feat(storage): initialize S3 storage and log initialization feat(proto): add DataEncryptionKey message feat(proto): remove EncryptionKey message feat(service): add Liveness RPC method to VaultService chore(deployment): comment out clickhouse services in docker-compose.yaml feat(deployment): add vault and minio services to docker-compose.yaml * chore(api_integration_local.yaml): remove ENCRYPTION_KEYS variable feat(build.yaml): add support for ENCRYPTION_KEYS variable feat(api): add @unkey/vault package as a dependency feat(env.ts): add VAULT_URL and VAULT_AUTH_SECRET environment variables feat(hono/env.ts): import Vault type from @unkey/vault package feat(middleware/init.ts): create vault client using VAULT_URL and VAULT_AUTH_SECRET feat(testutil/harness.ts): add storeEncryptedKeys fields for entities feat(routes): add support for fetching keyAuth data in various routes feat(schema.ts): add plaintext field to keySchema feat(v1_apis_listKeys.happy.test.ts): add test for retrieving key in plaintext feat(routes): add support for decrypting and displaying raw key in plaintext fix(routes): handle cases where both hash and plaintext are provided, provide either one feat(routes): encrypt plaintext key before storing in database fix(vault): remove unnecessary newline at end of bruno.json fix(vault): add error handling for storage.ErrObjectNotFound in GetKey method fix(vault): update S3_URL in docker-compose.yaml to point to s3 service feat(vault): add support for s3 service in docker-compose.yaml feat(vault): add new buf.gen.yaml and package.json for vault-sdk feat(vault-sdk): add generated proto files for object and service connect feat(vault-sdk): add generated TypeScript classes for vault service messages feat(vault-sdk): add createVaultClient function to create Vault client fix(api): change createdAt property in components interface to be required feat(rbac): add "decrypt_key" action to apiActions enum * fix(api_integration_local.yaml): update docker compose command to include planetscale and vault services feat(api_integration_local.yaml): add VAULT_URL and VAULT_AUTH_SECRET environment variables feat(v1_apis_listKeys.happy.test.ts): remove unnecessary console.log statement fix(env.ts): remove ENCRYPTION_KEYS environment variable parsing fix(create.ts): comment out unused imports and code related to encryption fix(createGateway): update createGateway mutation to remove encryption logic fix(createSecret): update createSecret procedure to remove encryption logic fix(decrypt.ts): update decryptSecret procedure to remove encryption logic fix(updateSecret): update updateSecret procedure to remove encryption logic fix(createWebhook): update createWebhook procedure to remove encryption logic refactor(env.ts): remove ENCRYPTION_KEYS field from zEnv object refactor(index.ts): comment out encryption key decryption logic refactor(index.ts): comment out encryption key decryption logic refactor(index.ts): comment out encryption key decryption logic refactor(index.ts): comment out encryption key decryption logic refactor(index.ts): comment out encryption key decryption logic refactor(index.ts): comment out encryption key decryption logic refactor(index.ts): comment out encryption key decryption logic refactor(index.ts): comment out encryption key decryption logic refactor(index.ts): comment out encryption key decryption logic refactor(index.ts): comment out encryption key decryption logic refactor(index.ts): comment out encryption key decryption logic refactor(index.ts): comment out encryption key decryption logic refactor(index.ts): comment out encryption key decryption logic refactor(index.ts): comment out encryption key decryption logic refactor(index.ts): comment out encryption key decryption logic refactor(encryption): remove unused encryption environment file refactor(encryption): remove unused encryption environment functions refactor(dashboard.ts): remove encryption key generation from environment variables

2023-11-02

refactor: use services

wip

2023-11-01

wip

2023-10-29

refactor: mostly just renaming

2023-10-22

feat: verifyKey in service

2023-10-20

feat: removeKey uses proto

feat: createKey uses proto

wip