Proto commits in neondatabase/neon

These 94 commits are when the Protocol Buffers files have changed:

2025-08-06

Merge commit '187170be4' into problame/standby-horizon-leases

Merge commit 'cec0543b5' into problame/standby-horizon-leases

2025-07-21

pageserver: remove gRPC `CheckRelExists` (#12616) ## Problem Postgres will often immediately follow a relation existence check with a relation size query. This incurs two roundtrips, and may prevent effective caching. See [Slack thread](https://databricks.slack.com/archives/C091SDX74SC/p1751951732136139). Touches #11728. ## Summary of changes For the gRPC API: * Add an `allow_missing` parameter to `GetRelSize`, which returns `missing=true` instead of a `NotFound` error. * Remove `CheckRelExists`. There are no changes to libpq behavior.

The documentation is generated from this commit.

2025-07-20

WIP: standby_horizon leases

WIP

2025-07-16

Add note on why there is no CheckRelExists

The documentation is generated from this commit.

Update grpcurl example

2025-07-15

pageserver: remove gRPC `CheckRelExists`

2025-07-10

pageserver/page_api: include block number and rel in gRPC `GetPageResponse` (#12542) ## Problem With gRPC `GetPageRequest` batches, we'll have non-trivial fragmentation/reassembly logic in several places of the stack (concurrent reads, shard splits, LFC hits, etc). If we included the block numbers with the pages in `GetPageResponse` we could have better verification and observability that the final responses are correct. Touches #11735. Requires #12480. ## Summary of changes Add a `Page` struct with`block_number` for `GetPageResponse`, along with the `RelTag` for completeness, and verify them in the rich gRPC client.

pageserver/page_api: add attempt to GetPage request ID (#12536) ## Problem `GetPageRequest::request_id` is supposed to be a unique ID for a request. It's not, because we may retry the request using the same ID. This causes assertion failures and confusion. Touches #11735. Requires #12480. ## Summary of changes Extend the request ID with a retry attempt, and handle it in the gRPC client and server.

page_api: add attempt to GetPage request ID

page_api: include block number and rel in `GetPageResponse`

2025-07-02

Merge branch 'problame/lease-deadline-tests' into problame/standby-horizon-removal-poc-rip-out

2025-06-30

fully rip it out, see which tests break

pageserver: add gRPC LSN lease support (#12384) ## Problem The gRPC API does not provide LSN leases. ## Summary of changes * Add LSN lease support to the gRPC API. * Use gRPC LSN leases for static computes with `grpc://` connstrings. * Move `PageserverProtocol` into the `compute_api::spec` module and reuse it.

Remove mention of exclusive expiration

2025-06-29

pageserver: add gRPC LSN lease support

2025-06-28

pageserver: add gRPC LSN lease support

2025-06-25

pageserver: payload compression for gRPC base backups (#12346) ## Problem gRPC base backups use gRPC compression. However, this has two problems: * Base backup caching will cache compressed base backups (making gRPC compression pointless). * Tonic does not support varying the compression level, and zstd default level is 10% slower than gzip fastest level. Touches https://github.com/neondatabase/neon/issues/11728. Touches https://github.com/neondatabase/cloud/issues/29353. ## Summary of changes This patch adds a gRPC parameter `BaseBackupRequest::compression` specifying the compression algorithm. It also moves compression into `send_basebackup_tarball` to reduce code duplication. A follow-up PR will integrate the base backup cache with gRPC.

pageserver: payload compression for gRPC base backups

2025-06-18

pageserver: support full gRPC basebackups (#12269) ## Problem Full basebackups are used in tests, and may be useful for debugging as well, so we should support them in the gRPC API. Touches #11728. ## Summary of changes Add `GetBaseBackupRequest::full` to generate full base backups. The libpq implementation also allows specifying `prev_lsn` for full backups, i.e. the end LSN of the previous WAL record. This is omitted in the gRPC API, since it's not used by any tests, and presumably of limited value since it's autodetected. We can add it later if we find that we need it.

2025-06-17

pageserver: gRPC base backup fixes (#12243) ## Problem The gRPC base backup implementation has a few issues: chunks are not properly bounded, and it's not possible to omit the LSN. Touches #11728. ## Summary of changes * Properly bound chunks by using a limited writer. * Use an `Option<Lsn>` rather than a `ReadLsn` (the latter requires an LSN).

pageserver: support full gRPC basebackups

2025-06-13

pageserver: gRPC base backup fixes

pageserver: gRPC base backup fixes

Tweak base backups

2025-06-12

WIP

2025-06-09

WIP advertisement sending

2025-06-03

WIP

WIP proto

2025-05-27

page_api: tweak errors (#12019) ## Problem The page API gRPC errors need a few tweaks to integrate better with the GetPage machinery. Touches https://github.com/neondatabase/neon/issues/11728. ## Summary of changes * Add `GetPageStatus::InternalError` for internal server errors. * Rename `GetPageStatus::Invalid` to `InvalidRequest` for clarity. * Rename `status` and `GetPageStatus` to `status_code` and `GetPageStatusCode`. * Add an `Into<tonic::Status>` implementation for `ProtocolError`.

wip

Rename `GetPageStatus` to `GetPageStatusCode`

2025-05-26

page_api: tweak errors

wip

2025-05-19

pageserver/page_api: add binary Protobuf descriptor (#11968) ## Problem A binary Protobuf schema descriptor can be used to expose an API reflection service, which in turn allows convenient usage of e.g. `grpcurl` against the gRPC server. Touches #11728. ## Summary of changes * Generate a binary schema descriptor as `pageserver_page_api::proto::FILE_DESCRIPTOR_SET`. * Opportunistically rename the Protobuf package from `page_service` to `page_api`.

pageserver: add gRPC page service schema (#11815) ## Problem For the [communicator project](https://github.com/neondatabase/company_projects/issues/352), we want to move to gRPC for the page service protocol. Touches #11728. ## Summary of changes This patch adds an experimental gRPC Protobuf schema for the page service. It is equivalent to the current page service, but with several improvements, e.g.: * Connection multiplexing. * Reduced head-of-line blocking. * Client-side batching. * Explicit tenant shard routing. * GetPage request classification (normal vs. prefetch). * Explicit rate limiting ("slow down" response status). The API is exposed as a new `pageserver/page_api` package. This is separate from the `pageserver_api` package to reduce the dependency footprint for the communicator. The longer-term plan is to also split out e.g. the WAL ingestion service to a separate gRPC package, e.g. `pageserver/wal_api`. Subsequent PRs will: add Rust domain types for the Protobuf types, expose a gRPC server, and implement the page service. Preliminary prototype benchmarks of this gRPC API is within 10% of baseline libpq performance. We'll do further benchmarking and optimization as the implementation lands in `main` and is deployed to staging.

2025-05-15

Typo

Clarify `GET_PAGE_CLASS_PREFETCH` on v18

Add TODO about block_number ordering

Mandate shard 0 for metadata requests

Elaborate on not_modified_since_lsn

Add class TODO.

Add `GET_PAGE_CLASS_BACKGROUND`

Added `neon-compute-mode` header

Allow scattered batches, and return complete batches

Revert "Remove `GetPageRequestBatch`, add `GetPageRequest.block_count`" This reverts commit f94d64a13cd5a630d15c8b5db1f5b48a46a6d254.

2025-05-07

Add TODO for batch vs. individual responses

Remove `GetPageRequestBatch`, add `GetPageRequest.block_count`

2025-05-06

Remove `ReadLsn` TODO

2025-05-05

Rename `RequestCommon` to `ReadLsn`

Use verbs for all methods

Prefix `id` and `class` with `request_` and group

Remove `optional` on `reason`

EOF

2025-05-02

pageserver: add gRPC page service schema

service documentation

page_api: add `GetPageRequest::class`

2025-04-30

page_api: add `GetPageResponse::status`

page_api: add `GetPageRequestBatch`

page_api: add `GetPageRequest::id`

page_api: protobuf comments

Add `neon-shard-id` header

Use `authorization` header instead of `neon-auth-token`

2025-04-29

page_api: add `get_slru_segment()`

pageserver: rename `data_api` to `page_api`

include lots of changes that went missing by accident

New communicator, with "integrated" cache accessible from all processes

2025-04-23

communicator: add streaming GetPage RPC

2025-04-17

git add yet another missing file

WIP: move code around for clarity There's now a client_grpc package, fully independent of PostgreSQL and the neon extension. There's also a separate package for the "model" structs representing the API and the 'prost' generated code, which is shared by client and server

Implement the remaining API functions. Now when you set neon.enable_new_communicator feature flag, the new communicator is used for everything. The old communicator code is not even initialized anymore. Everything is super slow, because we now open a new HTTP/2 connection for every GetPage request.

WIP: Skeleton version of the new communicator It's only used for dbsize requests at the moment.

2025-04-14

git add yet another missing file

WIP: move code around for clarity There's now a client_grpc package, fully independent of PostgreSQL and the neon extension. There's also a separate package for the "model" structs representing the API and the 'prost' generated code, which is shared by client and server

Implement the remaining API functions. Now when you set neon.enable_new_communicator feature flag, the new communicator is used for everything. The old communicator code is not even initialized anymore. Everything is super slow, because we now open a new HTTP/2 connection for every GetPage request.

WIP: Skeleton version of the new communicator It's only used for dbsize requests at the moment.

2025-04-04

storcon + safekeeper + scrubber: propagate root CA certs everywhere (#11418) ## Problem There are some places in the code where we create `reqwest::Client` without providing SSL CA certs from `ssl_ca_file`. These will break after we enable TLS everywhere. - Part of https://github.com/neondatabase/cloud/issues/22686 ## Summary of changes - Support `ssl_ca_file` in storage scrubber. - Add `use_https_safekeeper_api` option to safekeeper to use https for peer requests. - Propagate SSL CA certs to storage_controller/client, storcon's ComputeHook, PeerClient and maybe_forward.

2025-02-20

pageserver: guard against WAL gaps in the interpreted protocol (#10858) ## Problem The interpreted SK <-> PS protocol does not guard against gaps (neither does the Vanilla one, but that's beside the point). ## Summary of changes Extend the protocol to include the start LSN of the PG WAL section from which the records were interpreted. Validation is enabled via a config flag on the pageserver and works as follows: **Case 1**: `raw_wal_start_lsn` is smaller than the requested LSN There can't be gaps here, but we check that the shard received records which it hasn't seen before. **Case 2**: `raw_wal_start_lsn` is equal to the requested LSN This is the happy case. No gap and nothing to check **Case 3**: `raw_wal_start_lsn` is greater than the requested LSN This is a gap. To make Case 3 work I had to bend the protocol a bit. We read record chunks of WAL which aren't record aligned and feed them to the decoder. The picture below shows a shard which subscribes at a position somewhere within Record 2. We already have a wal reader which is below that position so we wait to catch up. We read some wal in Read 1 (all of Record 1 and some of Record 2). The new shard doesn't need Record 1 (it has already processed it according to the starting position), but we read past it's starting position. When we do Read 2, we decode Record 2 and ship it off to the shard, but the starting position of Read 2 is greater than the starting position the shard requested. This looks like a gap.  To make it work, we extend the protocol to send an empty `InterpretedWalRecords` to shards if the WAL the records originated from ends the requested start position. On the pageserver, that just updates the tracking LSNs in memory (no-op really). This gives us a workaround for the fake gap. As a drive by, make `InterpretedWalRecords::next_record_lsn` mandatory in the application level definition. It's always included. Related: https://github.com/neondatabase/cloud/issues/23935

experiment: find all places that need new rclsn-by-generation facility delete all the code that uses current inmem & controlfile remote_consistent_lsn field After this removal, search for `remote_consistent_lsn` highlights the places where the new solution that tracks rclsn per generation id needs to be fitted in

2024-12-11

wal_decoder: fix compact key protobuf encoding (#10074) ## Problem Protobuf doesn't support 128 bit integers, so we encode the keys as two 64 bit integers. Issue is that when we split the 128 bit compact key we use signed 64 bit integers to represent the two halves. This may result in a negative lower half when relnode is larger than `0x00800000`. When we convert the lower half to an i128 we get a negative `CompactKey`. ## Summary of Changes Use unsigned integers when encoding into Protobuf. ## Deployment * Prod: We disabled the interpreted proto, so no compat concerns. * Staging: Disable the interpreted proto, do one release, and then release the fixed version. We do this because a negative int32 will convert to a large uint32 value and could give a key in the actual pageserver space. In production we would around this by adding new fields to the proto and deprecating the old ones, but we can make our lives easy here. * Pre-prod: Same as staging

wal_decoder: fix compact key protobuf encoding (#10074) ## Problem Protobuf doesn't support 128 bit integers, so we encode the keys as two 64 bit integers. Issue is that when we split the 128 bit compact key we use signed 64 bit integers to represent the two halves. This may result in a negative lower half when relnode is larger than `0x00800000`. When we convert the lower half to an i128 we get a negative `CompactKey`. ## Summary of Changes Use unsigned integers when encoding into Protobuf. ## Deployment * Prod: We disabled the interpreted proto, so no compat concerns. * Staging: Disable the interpreted proto, do one release, and then release the fixed version. We do this because a negative int32 will convert to a large uint32 value and could give a key in the actual pageserver space. In production we would around this by adding new fields to the proto and deprecating the old ones, but we can make our lives easy here. * Pre-prod: Same as staging

2024-11-27

safekeeper: use protobuf for sending compressed records to pageserver (#9821) ## Problem https://github.com/neondatabase/neon/pull/9746 lifted decoding and interpretation of WAL to the safekeeper. This reduced the ingested amount on the pageservers by around 10x for a tenant with 8 shards, but doubled the ingested amount for single sharded tenants. Also, https://github.com/neondatabase/neon/pull/9746 uses bincode which doesn't support schema evolution. Technically the schema can be evolved, but it's very cumbersome. ## Summary of changes This patch set addresses both problems by adding protobuf support for the interpreted wal records and adding compression support. Compressed protobuf reduced the ingested amount by 100x on the 32 shards `test_sharded_ingest` case (compared to non-interpreted proto). For the 1 shard case the reduction is 5x. Sister change to `rust-postgres` is [here](https://github.com/neondatabase/rust-postgres/pull/33). ## Links Related: https://github.com/neondatabase/neon/issues/9336 Epic: https://github.com/neondatabase/neon/issues/9329

2024-11-25

wal_decoder: generate protobuf types

2024-05-21

Propagate standby apply LSN to pageserver to hold off GC. To avoid pageserver gc'ing data needed by standby, propagate standby apply LSN through standby -> safekeeper -> broker -> pageserver flow and hold off GC for it. Iteration of GC resets the value to remove the horizon when standby goes away -- pushes are assumed to happen at least once between gc iterations. As a safety guard max allowed lag compared to normal GC horizon is hardcoded as 10GB. Add test for the feature. Co-authored-by: Konstantin Knizhnik <knizhnik@neon.tech>

2024-02-07

Replace latest with horizon in get_page request

Propagate repply_flush_lsn from SK to PS to prevent GC from collecting objects which may be still requested by replica

2023-12-19

Support custom types in broker (#5761) Old methods are unchanged for backwards compatibility. Added `SafekeeperDiscoveryRequest` and `SafekeeperDiscoveryResponse` types to serve as example, and also as a prerequisite for https://github.com/neondatabase/neon/issues/5471

2023-10-28

WIP Safekeeper broker discovery

2023-08-29

Add term and http endpoint to broker messaged SkTimelineInfo. We need them for safekeeper peer recovery https://github.com/neondatabase/neon/pull/4875

2023-04-02

Switch to safekeeper in the same AZ (#3883) Add a condition to switch walreceiver connection to safekeeper that is located in the same availability zone. Switch happens when commit_lsn of a candidate is not less than commit_lsn from the active connection. This condition is expected not to trigger instantly, because commit_lsn of a current connection is usually greater than commit_lsn of updates from the broker. That means that if WAL is written continuously, switch can take a lot of time, but it should happen eventually. Now protoc 3.15+ is required for building neon. Fixes https://github.com/neondatabase/neon/issues/3200

2022-11-23

Add storage_broker binary. Which ought to replace etcd. This patch only adds the binary and adjusts Dockerfile to include it; subsequent ones will add deploy of helm chart and the actual replacement. It is a simple and fast pub-sub message bus. In this patch only safekeeper message is supported, but others can be easily added. Compilation now requires protoc to be installed. Installing protobuf-compiler package is fine for Debian/Ubuntu. ref https://github.com/neondatabase/neon/pull/2733 https://github.com/neondatabase/neon/issues/2394

2022-11-18

Add neon_broker binary. Which ought to replace neon_broker. This patch only adds the binary and adjusts Dockerfile to include it; subsequent ones will add deploy of helm chart and the actual replacement. It is a simple and fast pub-sub message bus. In this patch only safekeeper message is supported, but others can be easily added. Compilation now requires protoc to be installed. Installing protobuf-compiler package is fine for Debian/Ubuntu. ref https://github.com/neondatabase/neon/pull/2733 https://github.com/neondatabase/neon/issues/2394

2022-10-19

Tonic based broker for SkTimelineInfo.