Proto commits in rudderlabs/rudder-server

These 79 commits are when the Protocol Buffers files have changed:

2026-07-03

feat: add control-plane connection management for processor (#7126) # Description Opens the processor's dataplane-initiated gRPC connection to **cp-router**, reusing the shared `controlplane.ConnectionManager` (the same mechanism rudder-sources and warehouse use). This makes the processor reachable from cp-router so that, test-flow requests can be forwarded to the per-workspace `rudder-pytransformer`. Only the **node-0** pod opens the connection, so cp-router holds a single inbound connection per dataplane (not one per replica) and node-0 becomes the sole owner of any privileged work served over it. The whole bootstrap is gated behind a config flag and defaults off — this PR is inert until enabled. The `Forward` RPC itself is intentionally **left unimplemented in this PR** (a follow-up PR adds op-routing + the HTTP forward to pyt). Until then cp-router receives `codes.Unimplemented`. ## What's in here - **`proto/processor/processor.proto`** (+ generated `.pb.go` / `_grpc.pb.go`): a generic `ProcessorService` with a single RPC — `Forward(op, workspace_id, payload) → (status_code, body)`. The op is an opaque string and payload/body are JSON bytes, so cp-router stays transformation-agnostic and never grows per-op code. Generated with the repo's pinned plugin versions (`protoc-gen-go` v1.33.0, `protoc-gen-go-grpc` v1.3.0). - **`processor/cpservice/cpservice.go`** — `Service`, the `ProcessorService` gRPC handler. Embeds `UnimplementedProcessorServiceServer` so it registers and serves safely while `Forward` is pending. - **`processor/cpservice/connection.go`** — `Connector`, which wraps `controlplane.ConnectionManager`, registers the handler on the dataplane-side gRPC server, and drives the connection from the workspace connection flags published by the control plane (the warehouse pattern). `ShouldConnect` = feature-enabled **&&** node-0. - **`app/apphandlers/{processor,embedded}AppHandler.go`** — gated bootstrap wired into both modes that run the processor pipeline. GATEWAY mode is intentionally excluded (no processor pipeline; the LLD keeps this off the horizontally-scaled gateway). ## Node-0 detection Reuses `misc.GetInstanceID()` — the same instance-ID ordinal signal `app/cluster/state` uses. It returns the trailing ordinal of `INSTANCE_ID` (`"0"`, `"1"`, …); an unset `INSTANCE_ID` yields `""`, so local runs never connect. ## Config | Key | Default | Purpose | |-----|---------|---------| | `Processor.cpRouterConnection.enabled` | `false` | Master switch for opening the cp-router connection | | `Processor.cpRouterConnection.retryInterval` | `0` (→ 1s) | Reconnect backoff for the connection manager | | `CP_ROUTER_USE_TLS` (reused) | `true` | TLS for the cp-router dial | | `INSTANCE_ID` (reused) | — | Node-0 detection + `AuthInfo.InstanceID` | ## Linear Ticket pipe-3143 ## Security - [ ] The code changed/added as part of this pull request won't create any security issues with how the software is being used.

2026-07-01

feat: introduce Op enum for processor operations in ForwardRequest 🔒 Scanned for secrets using gitleaks 8.28.0

2026-06-29

chore(jobsdb): multi-consumer partition migration (#7098) # Description Adds `GetPendingConsumerJobs`, a partition-migration-specific method to fetch pending jobs while excluding their latest status(es). - For a multi-consumer jobsdb, each returned job carries only the list of consumers still pending. - Reuses all the existing jobsdb optimizations (dataset iteration, noResultsCache, limits, MoreToken, retry-on-compaction). - Partition migration now uses this method and also communicates the `consumers` field node-to-node (new `repeated string consumers = 12` on the cluster `Job` proto message). ## Linear Ticket resolves PIPE-3053 ## Security - [x] The code changed/added as part of this pull request won't create any security issues with how the software is being used. <!-- GitButler Footer Boundary Top --> --- This is **part 3 of 5 in a stack** made with GitButler: - <kbd>&nbsp;5&nbsp;</kbd> #7122 - <kbd>&nbsp;4&nbsp;</kbd> #7121 - <kbd>&nbsp;3&nbsp;</kbd> #7098 👈 - <kbd>&nbsp;2&nbsp;</kbd> #7097 - <kbd>&nbsp;1&nbsp;</kbd> #7096 <!-- GitButler Footer Boundary Bottom -->

2026-06-28

chore(jobsdb): GetPendingConsumerJobs

2026-06-26

feat: add control-plane connection management for processor 🔒 Scanned for secrets using gitleaks 8.28.0

chore(jobsdb): GetPendingConsumerJobs

2026-06-24

chore(jobsdb): GetPendingConsumerJobs

2026-06-22

chore(jobsdb): GetPendingConsumerJobs

2026-06-19

chore(jobsdb): GetPendingConsumerJobs

2026-05-31

Deploy preview for PR #7026

This commit does not contain any .proto files.

2026-01-02

feat: partition migration grpc server, client and job executor (#6568) 🔒 Scanned for secrets using gitleaks 8.30.0 # Description Introducing a new gRPC-based partition migration system. - `PartitionMigrationServer`: sets up a gRPC multidirectional streaming service for letting clients move jobs to the target jobsdb in the context of a single migration job. Takes care of deduplication of jobs that have already been migrated for each migration job id and table prefix. - `MigrationJobExecutor`: Executes partition migration jobs by moving jobs out of a local jobsdb to a remote jobsdb using the gRPC service. ## Key implementation details - Jobs are fetched in batches (default 10k) and split into smaller chunks (default 1k) for streaming. - The server operates asynchronously: while the receiver goroutine collects incoming chunks, a separate store goroutine persists complete batches to the database. - Memory is bounded using an unbuffered channel for backpressure, ensuring at most 2 batches are in memory at any time (one being assembled, one being stored). - Each batch is acknowledged before the client proceeds sending the last chunk of the next batch. ## Linear Ticket Resolves PIPE-2650 Resolves PIPE-2651 ## Security - [x] The code changed/added as part of this pull request won't create any security issues with how the software is being used.

2025-09-08

feat: data mapper tester

2025-09-05

add proto file for processor grpc service

feat: framework changes for processor, cp router communication

2025-08-13

feat: warehouse grpc API to get namespace mappings of a destination (#6200) ### 📋 **Overview** Implements the `GetDestinationNamespaces` gRPC API method for the warehouse service . The API allows clients to retrieve the most recent namespace for each source associated with a given destination ID. ### ✅ **Changes Made** - **Proto**: Added `GetDestinationNamespaces` RPC method and corresponding request/response messages - **Model**: Added `NamespaceMapping` struct with `SourceID` and `Namespace` fields - **Repository**: Added `GetDestinationNamespaces` method using `DISTINCT ON` query to get most recent namespace per source - **gRPC Service**: Implemented API method with input validation, error handling, and structured logging - **Tests**: Added comprehensive repository and gRPC tests ## Security - [x] The code changed/added as part of this pull request won't create any security issues with how the software is being used.

The documentation is generated from this commit.

2025-07-30

feat: warehouse grpc API to get namespace mappings of a destination

The documentation is generated from this commit.

2025-02-21

feat: gRPC API to expire the warehouse schema of a destination (#5508) # Description This PR adds a GRPC API which expires all warehouse schemas of a destination in the `wh_schemas` table by setting `expiresAt` column as `now()`. This will ensure that the next time these schemas are fetched from the DB, they will re-fetched from the warehouse. ## Security - [x] The code changed/added as part of this pull request won't create any security issues with how the software is being used.

2025-02-14

feat: sync latency metrics api (#5500)

2024-07-04

feat: add api to get first aborted uploads in series of continuous aborts (#4838)

2023-10-05

feat: health dashboard (#3906)

2023-07-03

chore: remove unused code from warehouse (#3567)

2023-05-29

chore: applying 1.9.1 and 1.9.2 hotfixes to main branch (#3413)

chore: batch event schema messages for faster processing (#3406)

2023-04-21

feat: schema forwarder (#3146)

2022-11-17

chore: data residency adaptations (#2703)

2022-10-11

chore(warehouse): validate object storage credentials (#2440) * Introduced api for object storage validation * Changes data type for isValid * Resolved code issues * Changed directory of test file for upload * Comments resolution * Error print in case of upload failure * Handled case for bucketName not present * Added base implementation of fallback logic for s3 * Refactored codebase to allow for better gRPC handling * Added deletion functionality * ContainerName in case of azure blob * Updated proto files to accept json object in config field * Minor change: printin error * Added download functionality for object storage validation * Included few checks * Minor correction * Liniting fixes * Updated the function endpoint to capture status codes as part of error along with some other refactorings * Updated warehouse proto files * Updated object storage validation api * Code refactoring * Code refactoring * Updated grpc files * Updated grpc files * Build failed resolution * Code comments * made changes to pass correct generated files * Code refactoring * Code refactoring * Code refactoring * Rectified error * Rectified error * Rectified error * Format code with gofumpt * Updated go.mod and go.sum file * Corrected warehouse grpc file * Added test case for api.go * Ran make fmt * Updated test message Co-authored-by: Abhimanyu Babbar <abhimanyu.babbar89@gmail.com> Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>

2022-09-15

chore(warehouse): warehouse retry uploads count (#2275)

2022-09-08

chore: add token type for success message (#2408)

chore: change proto message names (#2394)

2022-09-07

feat: cpRouter Multitenant Support (#2233)

2022-08-19

chore(warehouse): added merge partition pruning for deltalake changes. (#2175)

2022-06-03

feat(warehouse): added support for multi retry admin api (#1889)

2022-03-12

proto files updated for WHValidationRequest and WHValidationResponse.

2022-03-11

Added validation for destination proto files for warehouse.

2022-01-16

Added databricks proto files

2021-12-08

Make some changes around databricks.proto

2021-12-05

Added execute query

Updated proto files for databricks.proto

2021-12-03

Close proto changes.

2021-11-29

Added proto files.

2021-10-13

Remove vendor files (#1285) * Remove vendor * rm -mod=vendor * Add caching

2021-09-07

Made changes for TriggerWhUploadsResponse

2021-08-10

wh dest sync now

2021-08-04

No retry for archived wh uploads

2021-06-13

badgerv2->v2.2007.2 (#1066) * badgerv2->v2.2007.2

2021-05-26

wh upload retry

2021-03-30

Authotize syncs by source config

2021-03-11

Add last_exec_at to sync detail

2021-03-08

Minor tweaks to warehouse proto

2021-03-03

WH health endpoint

Add createdAt to upload detail endpoint

2021-03-02

Few more tweaks to warehouse proto

Changes to warehouse proto

Add additional fields

2021-02-25

api response changes

2021-02-24

change api routes

2021-02-19

working warehouse apis through grpc

2021-02-15

refactor proto files

2021-02-11

warehouse grpc apis

2020-12-15

migrating dedup

2020-08-28

added google pub/sub as destination

2020-07-13

rebased for go modules with master

updated for firehose

showing information to user for firehose

5213fb7df102a5c1630888633d6bdcf382d67d73

added sending simple events to firehose

changing for firehose

2020-07-10

updated for firehose

showing information to user for firehose

5213fb7df102a5c1630888633d6bdcf382d67d73

added sending simple events to firehose

2020-06-08

Chandu/badger crash fix (#346) * badger crash fix Co-authored-by: Chandra shekar Varkala <chandra@rudderlabs.com>

2020-03-04

Revert "Postgres version compatibility check (#159)" (#160) This reverts commit 1a5df33b6a0cef09b5d63600359bf3d4d5d1f829.

Postgres version compatibility check (#159)

Postgres version compatibility check

2019-12-23

Bigquery support (#97) Redshift, BigQuery, GCS and Azure Blob support

2019-12-16

Feature hosted service (#94) Rudder's Hosted service

2019-11-26

changes from master and session change (#91) * Add enterprise edition of docker compose file which includes grafana dashboard. * Fix dashboard repo path. * Fix ops dashboard path. * Fix bug with sourceIDFilters for batchrouter * Add timestamp to filename for s3 dumps * Cleaning up config toml * use sourceIDFilters also in setting ds empty result cache * v0.1.1 * Add sourceID to csutomval in empty cache for batch router on job status update * Support for VictorOps and Pagerduty (#87) * Fix jobsdb empty cache (#86) * Add sourceid as key in nested map for meptyresultcache * Add locks to inProgressMap in batchrouter to eliminate concurrent access * Remove unused code * Fixing alert if-else * Release script and Source debugger assert only in dev * Removing debug log * Changes to ginkgo tests for segment compatability * Update README.md * Update correctness.go * Update correctness.go * Add instanceName as a tag to all stats. * Adding 204 as success status * Add all 2** statucodes as success status * Dedup events at gateway using BadgerDB (#88) Use Badger K/V store to dedup events at gateway * Order by job_id for table dumps * Update README.md * Update README.md * Update README.md * Send empty array if properties is empty * add session id to singular events * append time info to log statements * resolve conflicts * changes to add session on job limit, adding logs * sdks send shorthand AM, GA in integrations array * revert adding time info to logs * handle the bug of not initializing empty array * dst and correctness test of s3 * Revert "dst and correctness test of s3" This reverts commit 148eea998f0d3b2adf7390a2d1186df2d476644a. * Revert "sdks send shorthand AM, GA in integrations array" This reverts commit 5b1205d50d092c539103dc7cb8a116fe52860dbb. * refactor logging * change session to uuid * update log levels and logging * more cleanup

2019-11-13

Dedup events at gateway using BadgerDB (#88) Use Badger K/V store to dedup events at gateway