Proto commits in yugabyte/yugabyte-db

These commits are when the Protocol Buffers files have changed: (only the last 100 relevant commits are shown)

2026-08-21

dist-trace: propagate trace context across the RPC boundary Carry a distributed trace from an RPC caller to its callee: define the wire type, write it into the request header, decode it on the far side, and open a server span parented under the caller's client span. All of it lands together because none of the pieces is observable on its own -- a wire field nobody reads, or a reader with nothing on the wire, cannot be tested. Wire type (rpc_header.proto): - TraceContextPB (W3C trace-context) and RequestHeader.trace_context. This is the type shared by the RPC-header and shared-memory transports; the latter starts using it in a later commit. Writer (outbound_call.cc): - SetRequestParam serializes the active client span's SpanContext into the trace_context submessage while it is sizing and writing the header, so the header is still written in a single pass. ~30 bytes on the wire, and only when a trace is active; an older peer skips the unknown field. Reader (serialization.{cc,h}): - ParseTraceContext(Slice) decodes the RequestHeader.trace_context wire field; ToSpanContext(TraceContextPB) is the shared-memory equivalent; both go through BuildSpanContext and fail on a zero/invalid context. - ParseHeader captures RequestHeader.trace_context into ParsedRequestHeader. Server / inbound span (yb_rpc.{cc,h}): - YBInboundCall::ParseFrom parses the header trace_context into parent_span_context_ (best-effort: a bad context is logged, never fails the RPC). - CreateServerSpan starts a StartServerSpanWithScope child of that parent (remote) or of an explicitly passed context (local); RespondSuccess/Failure set status and end it; DropServerSpanScope releases the scope on the handler thread while the span ends later. Client local-call handoff (local_call.{cc,h}, rpc_context.cc): - LocalOutboundCall tags rpc.local_call and drops its client-span scope in the ctor (a local call never hops threads to drop it later). - otel_span_context() exposes the outbound span's context so the local inbound span can parent under it (no wire header exists locally). - RpcContext creates the server span: from the wire header for remote calls, from the outbound call's context for local calls. Process init (tserver/db_server_base.cc): - DbServerBase is the shared base of Master and TabletServer, so DbServerBase:: Init is the single place that initializes the process-wide tracer (service name = server name, node id = permanent uuid), with Shutdown tearing it down. Gated by IsDistTraceEnabled, so it is a no-op when tracing is off. Without this the server spans above are never exported, which is why the 10 lines ship here rather than as a commit of their own. Test: TestRpcSpanReachesTabletServer runs a SELECT under a known traceparent and asserts that the TabletServer's inbound "rpc yb.tserver.PgClientService.Perform" span lands in that trace as a child of the ysql backend's outbound span, with the expected rpc.service/rpc.method attributes. The new WaitForRemoteChildSpan collector helper does the caller/callee pairing check.

[pg19] merge: master commit ad7b6d8dd7cf0220a4a84fd70d3a78fdca9c6368 into pg19 Summary: Merge base: 728cbe7cd353a7cb94465a87da8b5ad327a8af74. Merged-into side: origin/pg19 (69c0aaf51b5c6371c788d756fc1de892750d8aad). Attribution key: - "master commit ..." = YB master-branch commit, i.e. what independently changed on master after the pg19 fork point. - "upstream PG commit ..." = upstream PostgreSQL commit whose change is on the pg19 branch via the initial merge (YB pg19 commit 4c01ae04c6d8e5487679d2079e44a067a66febcd merged upstream tip 90630ec42939d074ecc7b6b959b48252eed32646). - "YB pg19 commit ..." = YB-authored commit on the pg19 branch (initial-merge resolutions and follow-up pg19 fixes). - managed/src/main/resources/version.txt: - version line: - master commit 46744bbdd127f7320f89d15b0d9d06eb227e8b53 bumped master to 2.31.0.0-b0. - YB pg19 commit 478a4e5377366a4a807d27cef65aba8bbaa5f95d bumped to 2.31.0.1900-b0 (pg19-specific Jenkins version). - Kept YB pg19's 2.31.0.1900-b0. - src/lint/upstream_repositories.csv: - src/postgres row: - master commit f2875e8a94d3c4114de897061fdceb36ff33fb34 updated the src/postgres pin to b4db528e00283d572cafeacb652f8dc913326189 (CVE-2026-2005/2026-2004 imports). - YB pg19 initial merge updated the pin to the upstream tip 90630ec42939d074ecc7b6b959b48252eed32646. - Kept YB pg19's pin. - src/yb/yql/pgwrapper/ysql_binary_runner.cc: - YsqlDumpRunner::DumpSchemaAsOfTime / RunAndModifyForClone: - master commit 95306ec58b4b56c50b822f7b4d6576ef96af3b12 rewrote both functions: regex-based owner/db-name post-processing replaced by new ysql_dump --rename-database/--rename-owner options, added WrapConnectWithAllowConnections; master commit 3d54cf58a827f4161167556cbc15966eda91a635 added --dump-role-checks. - upstream PG commit 6a46089e458f2d700dd3b8c3f6fc782de933529a renamed pg_dump's --with-statistics to --statistics (and made it conflict with --schema-only), so YB pg19 commit 30bb08aecae3eb3b97f4a8beb3b2bad36e6789eb changed the clone-statistics flags to --no-data + --statistics instead of --schema-only + --statistics. - Took master's rewritten functions; re-applied the YB pg19 flag logic inside the new DumpSchemaAsOfTime. - src/yb/yql/pgwrapper/pg_libpq-test.cc: - GetCatalogTableNameFromIndexName regex: - master commit 16d281b6a6e7fa9458c2db8c74c9ff1243348b76 rewrapped the helper signature/indentation and marked it [[maybe_unused]]. - upstream PG commit 2f094e7ac691abc9d2fe0f4dcf0feac4a6ce1d9c added the pg_propgraph_* catalogs; YB pg19 commit 9251419cbda50ae7e5e0d91f3190534306fd1f3b added them to the regex. - Took master's rewrapped indentation and added the YB pg19 pg_propgraph_* entries into it; master's [[maybe_unused]] wrapper merged cleanly outside the conflict. - src/postgres/contrib/pgcrypto/ (Makefile, openssl.c, pgcrypto.c, px.c, expected/pgp-pubkey-session.out) + doc/src/sgml/pgcrypto.sgml: - master commit 2dae69dec8eb9d7917260c21b131e4e53035cb43 point-imported upstream PG commits 924d89a354750976cdd271d1dfc6c1e97cbb8851 and 035f99cbebe5ffcaf52f8370394446cd59621ab7; master commit f2875e8a94d3c4114de897061fdceb36ff33fb34 imported the back-branch pgp_pub_decrypt_bytea CVE fix (the master-line commit is 379695d3cc70d040b547d912ce4842090d917ece). All three upstream commits are ancestors of the merged tip, so the pg19 side already carries the content; took the pg19 side at every conflict. - px.c / pgp-pubkey-session.out wording: pg19 says "Session key too big" (upstream follow-up 459576303dfb47d5b9626331df1f8e8767840ede); master's back-branch import says "Public key too big". - src/postgres/contrib/pg_stat_statements/pg_stat_statements.c: - pgss_store signature and all callers: - master commit 27632aff715c7dad5c8068d0c76c7821458282e7 added a YbInstrumentation *yb_stats parameter (NULL from parse/plan hooks, &queryDesc->yb_query_stats->yb_instr from ExecutorEnd, YBGetUtilityOperationStats() from ProcessUtility). - upstream PG commits cf54a2c002544a4b7934deb44c895750aadb0a3c (parallel_workers_to_launch/launched) and 3357471cf9f5e470dfed0c7919bcf31c7efaf2b9 with e125e360020a7b0affd5bea938b749e85d8999d3 (PlannedStmtOrigin planOrigin / generic vs custom plan counters) added new pgss_store parameters; YB pg19 commit 945ea71848edb621ae23a3c7e9dcb5029dcb0567 (pg_stat_statements re-enable) carries them. - Resolved by combining: the upstream parameter list with master's yb_stats appended. - pgss_store PGSS_EXEC accumulation block: - master commit 27632aff715c7dad5c8068d0c76c7821458282e7 replaced the local YbInstrumentation yb_instr + YbUpdateSessionStats() with reads from the passed yb_stats, guarded by if (yb_stats). - upstream PG commit 8928817769de0d81758bc760333d3056c67b63c1 removed the volatile alias, renaming the accumulation target e-> to entry->. - Took master's structure with upstream's entry-> naming; no yb_instr local remains. - src/postgres/contrib/pg_stat_statements/{sql,expected}/yb.port.pg_stat_statements.* (deleted on pg19, modified on master): - master commit 61e8975d220a7d3a833c359ad5aa72c682760859 changed the initial drop to DROP EXTENSION IF EXISTS pg_stat_statements CASCADE (global views by default create gv$ dependents). - YB pg19 commit 945ea71848edb621ae23a3c7e9dcb5029dcb0567 deleted the monolithic yb.port test, replacing it with the per-topic yb.port.* split matching upstream's suite layout. - Kept the deletion; ported master's CASCADE (and its NOTICE/DETAIL output) into yb.port.select.sql/.out, which carries the equivalent initial drop. - src/postgres/contrib/postgres_fdw/postgres_fdw.c: (all master-side changes below are from master commit 98418b7bc5149e5a2253b30968e2efca05f33343, the federated global-view scan rework) - FdwScanPrivateIndex enum: - Kept master's YbFdwScanPrivateTserverUuid addition. - postgresGetForeignPaths: - master removed the skip-ORDER-BY-pushdown-for-federated guard around add_paths_with_pathkeys_for_rel. - upstream PG commit 9e9931d2bf40e2fea447d779c2e133c2c1256ef3 added the restrictlist argument to add_paths_with_pathkeys_for_rel. - Took master's unguarded call in upstream's 4-arg form. - fetch_more_data (2 conflicts): - master: routed the global-view read through the reworked YbGlobalViewReadExecScan helper (db name + tserver UUID) and set eof_reached=true with TODO(#30843) (no pagination yet), replacing the YBCPgGlobalViewReadIsEof check. - upstream PG commit 80aa9848befc13c188d2775a859deaf172fdd3a2 restructured the function body (PG_TRY/volatile removed; PGresults are context-owned via the libpqsrv_PQwrap infrastructure from 7d8f5957792421ec3bb9d1b9b6ca25d689d974b7); upstream PG commit 74e121c8dc5184318478dee587cf7d8303ab1357 changed the pgfdw_report_error signature. - Took pg19's body with master's helper call and the eof_reached=true TODO(#30843). - YbGlobalViewReadExecScan: - master: the pggate call gained database_name/tserver_uuid args and a WARNING that skips a tserver on per-tserver errors. - pg19 wraps the results in libpqsrv_PQwrap (upstream PG commit 7d8f5957792421ec3bb9d1b9b6ca25d689d974b7). - Took master's body with libpqsrv_PQwrap around both results. - src/postgres/contrib/postgres_fdw/expected/yb.orig.postgres_fdw.out: - user-object OID rows: - master commit 61e8975d220a7d3a833c359ad5aa72c682760859 added the gv$-exclusion filters to the queries and re-blessed these same rows to the first-user OIDs under global-views-by-default initdb (16400/16403/16406). - YB pg19 commit eaf9d99b8eddf503b89a942ffccfa41922d5b38f had reblessed the same rows for PG19 initdb without global views (16409/16412/16415). - Took master's OIDs: global views are created at initdb on the merged tree, so runtime produces master's values. The rest of the file keeps the pg19 rebless (PG19 HINT option lists, EXPLAIN formatting), which master's changes do not touch. TestPgRegressContribYbPostgresFdw passes. - src/postgres/contrib/postgres_fdw/expected/yb.orig.global_views.out: - all EXPLAIN plan blocks: - master commits 98418b7bc5149e5a2253b30968e2efca05f33343 and others changed the plans themselves. - YB pg19 commit eaf9d99b8eddf503b89a942ffccfa41922d5b38f reblessed the old plans for upstream PG19's EXPLAIN/subplan-naming changes (InitPlan expr_N, (InitPlan expr_N).col1 instead of $n). - Took master's golden reblessed for upstream PG19 EXPLAIN format (InitPlan expr_N subplan naming including in Remote Filters, MinMaxAggregate replaced by Merge Append + Limit, Sort under ordered aggregates). TestPgRegressContribYbGlobalViews passes. - src/postgres/src/backend/optimizer/path/costsize.c: - yb_cost_seqscan result-page/transfer-cost block: - master commits 833c71c8b1d1020568f57a7db96b9f48d882f441 and d94926814c14a7ce5e969e25bf679084f52e56af restructured the seqscan result-page/transfer-cost estimation. - YB pg19 commit caa39b2fbcc736a88a7f54ced92ce5f15ebeef16 adapted the YB cost model to upstream's disabled_nodes accounting via the PGS_* enable_mask machinery. - Took master's restructure; kept the pg19 enable_mask |= PGS_CONSIDER_NONPARTIAL inside the else branch. - (auto-merged; build fix) estimate_array_length gains root and returns double (upstream PG commit 9391f71523b6e57f1194d9f6543bc7948c16411b). - src/postgres/src/backend/optimizer/path/indxpath.c: - alt_batched_clauseset block before build_index_paths: - master commit 68b2932804ecdc297d53053660f492d8d8ff092b excludes unbatchable clauses from the batched clause set. - upstream PG commit 5bf748b86bc6786a3fc57fc7ce296c37da6564b0 removed the skip-lower-SAOP logic in get_index_paths; YB pg19 commit d6a5a0b1cc7d6f47cf2d2cd47eee7457dc408d44 trimmed the mirrored comment in yb_get_batched_index_paths accordingly. - Took master's block with the pg19 side's comment wording. - yb_get_batched_index_paths, build_index_paths calls: - master commit 68b2932804ecdc297d53053660f492d8d8ff092b changed the clause-list argument from clauses to batched_clauseset in both calls: the main ordered-path call and the conditional second pass. - upstream PG commit 5bf748b86bc6786a3fc57fc7ce296c37da6564b0 removed the skip_lower_saop out-parameter and the second pass from get_index_paths; YB pg19 mirrored that shape into yb_get_batched_index_paths. - Applied master's batched_clauseset to the surviving single call. - bitindexpaths distinct-pushdown check: - master commit ac827e08068582ac1238825177ff25504ae418cd also excludes bare distinct index scans (yb_distinct_prefixlen > 0) from bitmap candidates. - upstream PG commit 24225ad9aafc576295e210026d8ffa9f50d61145 renamed UpperUniquePath to UniquePath. - Took master's check with the rename applied; kept the initial merge's YB_TODO_PG19MERGE (the distinct-pushdown task still needs to evaluate the UniquePath change). - (auto-merged; build fix) yb_op_is_btree_equality changed to use get_op_index_interpretation/COMPARE_EQ (upstream PG commit a8025f544854ad8b865c6b4509030ee84aa8f4a0); RowCompareExpr.rctype -> cmptype (upstream PG commit 6339f6468e8217f556e38482626250dc72d7cd00; CompareType LT..GT values match btree strategy numbers). - src/postgres/src/backend/optimizer/path/joinpath.c: - try_nestloop_path / try_hashjoin_path / try_partial_hashjoin_path prechecks: - master commit 890974e32d6deaefe7da685c2dce22f2bbb7706b added yb_force_px bypasses (yb_test_force_parallel GUC) before the prechecks. - upstream PG19's precheck signatures carry workspace.disabled_nodes (disabled-nodes costing rework). - Combined: master's yb_force_px blocks with upstream's precheck signatures. - src/postgres/src/backend/optimizer/plan/planmain.c: - query_planner after add_base_rels_to_query: - master commit 8904f5fc8dc1e46294ea4e95190b2eaae668d123 added the yb_test_invalidate_relcache_in_planner hook. - upstream PG commit 430a5952deb3bfbfe1e2537315d44427b7c41fb1 moved remove_useless_groupby_columns into query_planner at the same spot. - Kept both: upstream's call first, then the YB block. - src/postgres/src/backend/optimizer/plan/setrefs.c: - T_YbBitmapIndexScan yb_idx_pushdown fixups: - master commit 509446cac8aefa593d1177da116f5757f455739a split the fixup into is_primary (base-table attnums via fix_scan_list) vs secondary (index-relative via fix_upper_expr). - upstream PG commit 2489d76c4906f4461a364ca8ad7e0751ead8aa0d added the nrm_match argument to fix_upper_expr; the pg19 initial merge added NRM_EQUAL to the parameter list for fix_upper_expr calls. - Took master's structure with NRM_EQUAL inserted in both fix_upper_expr calls. - (auto-merged; build fix) added utils/rel.h for the is_primary rd_index access. - src/postgres/src/backend/optimizer/util/pathnode.c: - forward declarations: - master commit ee367279a3e9a333544e9870603df84e64fa9f15 added yb_propagate_subqueryscan_fields (subquery-scan uniqkeys propagation); its master-side context re-presented PG15-base contain_references_to/ris_contain_references_to decls. - contain_references_to/ris_contain_references_to exist on master only via the REL_15-branch band-aid, upstream PG commit 12ec16d11c8bd75624a77ae7a456ceec8332c9d9; they never existed on the upstream master line, which instead fixed the underlying reparameterize_path_by_child problem with upstream PG commit b7e2121ab7d6166b835a46ceaab1b6a6dc589703 (postpone reparameterization to create_plan(), adding pathlist_is_reparameterizable_by_child). - Kept upstream's pathlist_is_reparameterizable_by_child decl and added only master's YB decl; the band-aid helpers are superseded. - (auto-merged; build fix) master's separate T_UpperUniquePath arms merged into T_UniquePath with a YB_TODO_PG19MERGE note (upstream PG commit 24225ad9aafc576295e210026d8ffa9f50d61145). - src/postgres/src/backend/optimizer/util/plancat.c: - get_relation_info after attr array setup: - master commit 211a32f56b6fa6295b8a15ba357fd1c378ebe7ed added the yb_prefetch_column_stats batched-statistics warmup. - upstream PG added the notnullattnums recording block. - Kept both: upstream's block first, then the YB block. - src/postgres/src/include/optimizer/paths.h: - declarations: - master commit ee367279a3e9a333544e9870603df84e64fa9f15 added yb_convert_subquery_uniqkeys. - upstream PG commit 7358abcc6076f4b2530d10126ab379f8aea612a5 (store Append-consolidation info in the final plan) changed get_singleton_append_subpath to take a child_append_relid_sets out-param. - Kept both. - src/postgres/src/backend/executor/execMain.c: - standard_ExecutorRun head: - master commit 27632aff715c7dad5c8068d0c76c7821458282e7 added YBOnExecutorOperationBegin(). - upstream PG added Assert(GetActiveSnapshot() == estate->es_snapshot). - Kept both. - ExecConstraints not-null loop: - master commits 084d6076ae8f19ee44abc320cb29a8ef94d2d055 and 8b7a26bb5c65a886b0c4ad6b32173a67bb0e46ab rewrote the YB skip-unmodified-columns logic. - upstream PG commit cdc168ad4b22ea4183f966688b245cabb5935d1f refactored reporting into ReportNotNullViolationError and added virtual-generated-column collection; the loop became for (AttrNumber attnum ...). - Took master's YB precompute + skip check inside upstream's loop and reporting structure; removed the initial-merge YB_TODO_PG19MERGE block and the per-attribute modifiedCols computation. - src/postgres/src/backend/executor/execProcnode.c (+ instrument.c, executor.h): - master commit 103f54edc9b5c2b86c5e55079c0ad6f9f89b9d2a changed distributed tracing from one span per tuple to one span per plan node (new helper YbExecProcNodeTrace, called from ExecProcNodeInstr when tracing is on; the per-tuple code is deleted on master). - upstream PG commit 544000288ec8f7dc6a1e0285821adc47324ecd33 moved ExecProcNodeInstr from execProcnode.c into instrument.c; the pg19 side carries the old per-tuple tracing at that new location. - Took master's per-node tracing, with its ExecProcNodeInstr hook placed in instrument.c where the function now lives; YbExecProcNodeTrace is declared in executor.h so instrument.c can call it. - src/postgres/src/backend/executor/nodeLockRows.c: - ExecReScanLockRows: master commit b7dbc65affb1a00f88843b2a6f9d4f7b3b3cb44a added the yb_info buffered-slots reset; upstream PG19 added the outerPlan local. Kept both. - read-ahead instrumentation (auto-merged; build fix): master's SKIP LOCKED read-ahead code types pstate->instrument as Instrumentation *; upstream PG commit 5a79e78501f46bd3ac7fbd0ff84cf1e20dbafd19 split per-node instrumentation into NodeInstrumentation (which carries yb_instr on pg19). Changed the local to NodeInstrumentation *. - src/postgres/src/backend/executor/nodeModifyTable.c: - ExecUpdateEpilogue: master commit 1ac33f1cbd76fd2cd9ca90069c062191ed22b872 added the YB comment explaining the no_row_trigger skip and transition-table capture; upstream PG added the temporal leftovers block. Kept both. - src/postgres/src/backend/commands/{copyto.c, createas.c, matview.c, explain.c}: - pg_plan_query / ExplainOneQuery calls: - master commit 89916d20b5dba5097661ea5c0bb02a88271aa631 wrapped CURSOR_OPT_PARALLEL_OK in the yb_disable_parallel_query_in_ddl conditional. - upstream PG commit c83ac02ec7309edb7561eee93895c31a54b93d3d added the ExplainState arg to pg_plan_query; upstream PG commit a04ddd077e61096da660e1c2f205e0c8277f2dcd reworked ExplainOneQuery for its hook. - Combined: master's conditional with upstream's signatures. - src/postgres/src/backend/commands/async.c: - listener loop in Exec_ListenPreCommit: - master commit d94d4130c4545eae5f21593b2f8fd53312a4bb77 added the notifications-poller runtime-error check before the loop. - upstream PG commit 024c521117579a6d356050ad3d78fdc95e44eefa (BackendId -> 0-based ProcNumber) changed the loop to the INVALID_PROC_NUMBER sentinel. - Combined: master's check + upstream's loop header. - asyncQueueIsFull: - master added the yb_test_notify_queue_max_pages test hook (with PG15 wraparound math). - upstream PG commit 2cdf131c46e631addfc386f6106e52a1b8cc3a70 removed the wraparound math and simplified the function to occupied >= max_notify_queue_pages. - Took upstream's body with the YB hook simplified to occupied >= yb_test_notify_queue_max_pages (wraparound math is obsolete). - ybTerminateSlowestListener / ybIsListenerPid / ybSignalAllListeners (auto-merged; build fix): - master commits ee5574033a371e6997727d82252dd0aad01674de / 1e522731bbd90f5e0a17d60b70bd89e16c2b08f8 added these functions using base-era BackendId loops (1-based, i > 0 sentinel) and a BackendId array. - upstream PG commit 024c521117579a6d356050ad3d78fdc95e44eefa replaced BackendId with 0-based ProcNumber (INVALID_PROC_NUMBER sentinel). - Ported the three loops and the ids array to ProcNumber, matching the file's other listener loops. - YbHasActiveOrPendingListen (auto-merged; build fix): - master added the function testing listenChannels != NIL. - upstream PG commit 282b1cde9dedf456ecf02eb27caf086023a7bb71 replaced the listenChannels list with the localChannelTable hash (LISTEN/NOTIFY shared channel map). - Ported the test to !LocalChannelTableIsEmpty(). Added a YB_TODO_PG19MERGE for additional review. - poller ReplicationSlotAcquire: - master commit 351b43d8b8207e057295e8b377a44d29d8712148 changed nowait to false (advisory-lock cleanup wait). - upstream PG commit f41d8468ddea34170fe19fdc17b5a247e7d3ac78 added the error_if_invalid argument. - Combined. - src/postgres/src/backend/commands/extension.c: - includes, script_error_callback_arg/ExtensionLocation structs, ext_sibling_callback signature: - master commit f2875e8a94d3c4114de897061fdceb36ff33fb34 (CVE import, carrying upstream back-branch 3ecc84cce3c521894d86267df552a2d5f891a409 pg_extension.oid syscache) added the EXTENSIONOID syscache usage and ext_sibling cache (int cacheid spelling). - upstream PG commit ee642cccc43ca1a0ff4a4af2a457208b919af017 made SysCacheIdentifier a typedef enum (callback params typed); the pg19 side already carries the same CVE port with that spelling plus upstream's new PG19 structs. - Took pg19's side throughout. - src/postgres/src/backend/commands/indexcmds.c: - copartitioned/colocation block before ReleaseSysCache: - master commits 8ed6e2d16b29ccb4689e66088e738663d658b283 and others added the yb_amiscopartitioned copartitioned-index logic. - upstream PG commit bc6374cd76abb2e6a48c4b57c0b5a7baa5babd67 made IndexAmRoutines statically allocated and removed the pfree(amRoutine) call. - Took master's new block without the trailing pfree(amRoutine). - child index stmt creation in DefineIndex partition recursion: - master commit 7c50c626fa23af268156ae2ca3528ed354204842 added clearing childStmt->split_options. - upstream PG commit 68dfecbef210dc000271553cfcb2342989d4ca0f replaced the manual childStmt field fixups/var remapping with generateClonedIndexStmt. - Kept upstream's generateClonedIndexStmt and added master's split_options clear after it. - YbSyncSplitOptionsAndPresplit call in DefineIndex (auto-merged; build fix): - master commit 7c50c626fa23af268156ae2ca3528ed354204842 syncs stmt->split_options with the yb_presplit reloption, mutating the IndexStmt. - upstream PG19 constified DefineIndex's IndexStmt parameter. - Wrapped the call with unconstify(IndexStmt *, stmt) (PG's sanctioned escape; the mutation is deliberate). - src/postgres/src/backend/commands/operatorcmds.c: - estimator ACL checks: - master commit f2875e8a94d3c4114de897061fdceb36ff33fb34 (CVE import, carrying upstream back-branch b764b26f2e0a0a57057f301ca87e6a604f5e708b superuser-for-estimator check) added the superuser requirement with PG15's pg_proc_aclcheck. - upstream PG commit c727f511bd7bf3c58063737bcf7a8f331346f253 replaced per-catalog aclcheck functions with object_aclcheck. - Took pg19's changes. - src/postgres/src/backend/commands/sequence.c: - YBCInsertSequenceTuple / YBCReadSequenceTuple calls: - master commit 664b88d2b2bb49103dcf137bdceaf01e08803161 removed the YBIsDBCatalogVersionMode() argument (shared catalog version mode removed). - YB pg19 uses last_value and is_called to match upstream PG commit ba3d93b2e806a877f26922e0f9e1845d0ef1511b. - Took master's argument list with the pg19 variable names. - src/postgres/src/backend/commands/tablecmds.c: - RI_Initial_Check comment: - master rephrased the YB note ("YB Note:"). - upstream PG added the temporal-FK LEFT JOIN sentence. - Kept both. - src/postgres/src/backend/utils/cache/relcache.c: - helper functions before RelationIdGetRelation: - master commit 1964aea1b9fb1bc40ce65cf36319749f4d8464e4 added YbResetRelationCacheAfterPreloadFailure; its side also re-presented PG15-base RelationCloseSmgrByOid. - upstream PG commit 21d9c3ee4ef74e2229341d39811c97f85071c90a removed RelationCloseSmgrByOid. - Took master's YB function; did not resurrect RelationCloseSmgrByOid. - src/postgres/src/backend/utils/cache/inval.c: - yb_refresh_cache_in_progress and YbMaybeRefreshCache: - master commit da9dc9a0b0d7b5088c836ed433e38aaad6de5910 made yb_refresh_cache_in_progress non-static (used from tcop/postgres.c), refactored the refresh logic into YbMaybeRefreshCache, and removed yb_accept_inval_failed_version. - upstream PG commit 3abe9dc18892b9f69bb48a2eb21fbe5cf348a489 added the relsync callback machinery adjacent to the YB statics. - Combined: upstream's relsync block + master's non-static flag and YbMaybeRefreshCache; dropped the removed yb_accept_inval_failed_version. - src/postgres/src/backend/utils/misc/pg_yb_utils.c / src/postgres/src/include/pg_yb_utils.h: - includes and declarations: combined both sides' additions; skipped rusagestub.h, removed by upstream PG commit 36b3d52459aecd4f8bc39a4604e42186c48aa9d2. - HandleYBStatusAtErrorLevel: - master commit c0612878517eac174a77d6591f970a1c8b4584fe reimplemented the macro body as HandleYBStatusAtErrorLevelImpl. - YB pg19 (initial merge) carried the old macro body adapted to PG19 errstart/errfinish. - Took master's macro; the Impl (auto-merged into pg_yb_utils.c) already uses signatures valid on PG19. Build fix within the same macro: master passes PG_FUNCNAME_MACRO to HandleYBStatusAtErrorLevelImpl, but upstream PG commit 320f92b744b44f961e5d56f5f21de003e8027a7f removed PG_FUNCNAME_MACRO (PG19 elog macros pass __func__ directly); changed the macro to pass __func__. - kept both yb_enable_alter_table_rewrite (YB pg19) and yb_enable_replication_origin_shared (master) externs. - (auto-merged; build fix) RelationGetPrimaryKeyIndex callers gain deferrable_ok=false (upstream PG commit 14e87ffa5c543b5f30ead7413084c25f7735039f); rd_newRelfilenodeSubid renamed to rd_newRelfilelocatorSubid (upstream PG commit b0a55e43299c4ea2a9a8c757f9c26352407d0ccc). - src/postgres/src/backend/utils/misc/yb_qpm.c: - yb_qpm_ExecutorStart: - master commit 066e5b0f2b1570dddb88bad7f2d5020a0b5a5e70 added the hook allocating queryDesc->totaltime via InstrAlloc when QPM is enabled. - upstream PG commit 2c16deee2f7d52d6567dcbad046f74a8e880ee52 removed QueryDesc.totaltime, and upstream PG commit 3eea7a0c97e94f9570af87317ce3f6a41eb62768 dropped ExecutorRun's execute_once argument (yb_qpm_ExecutorRun is now 3-argument). - Kept master's hook with its totaltime/InstrAlloc body parked under #if 0 YB_TODO_PG19MERGE. - yb_qpm SRF output block: took master's Datum/tuplestore-style values over the old cstring style; the pg19 side's only change here was psprintf "%ld" -> INT64_FORMAT for useCount (initial-merge format fix), subsumed by master's Int64GetDatum. - (auto-merged; build fix) master's queryDesc->totaltime read parked under #if 0 YB_TODO_PG19MERGE. - yb_pg_stat_plans_get_all_entries tuple descriptor: YB pg19 commit 987eb84c5aa48d22ce330cf115ab634c24288cb8 added TupleDescFinalize before the descriptor is used (PG19 BlessTupleDesc asserts a finalized descriptor); master's SRF rework passes the hand-built descriptor to BlessTupleDesc. Combined: TupleDescFinalize before master's BlessTupleDesc. - src/postgres/src/backend/access/common/reloptions.c: - relopt_real table: master commit 5b50c9b589ad5d4fa38d5418ea8c25a23e7175db added yb_auto_analyze_scale_factor/yb_auto_analyze_cooldown_scale_factor; upstream PG added vacuum_max_eager_freeze_failure_rate at the same spot. Kept all three. - src/postgres/src/backend/access/yb_access/yb_lsm.c: - master commit be86adfd15c7c5c0e5c70955a99aee05a743ae18 added the ybcinparallel_partition_keys helper; upstream PG commit 92fe23d93aa3bbbc40fca669cabc4a4d7975e327 and others changed the amestimateparallelscan signature and renamed ps_offset to ps_offset_am (YB pg19 initial merge added the yb_estimate_parallel_size_am wrapper for it). Kept both; the helper uses ps_offset_am. - src/postgres/src/backend/utils/adt/pg_locale.c: - pg_newlocale_from_collation: - master commit 172b5d73f32281d49d8dc7e72063f99c980705d0 added yb_force_locale_lookup (yb_test_collation && conn-mgr) forcing re-lookup, with a deliberate-leak comment. - upstream PG commits 3aa2373c114124f62e80016d8939331fcb4d5586 (extracted create_pg_locale out of lookup_collation_cache) and 1ba0782ce90cb4261098de59b49ae5cb2326566b (split it into per-provider init functions), plus 005c6b833f7866b71b50a5382e30d6c3f695306e (simplehash collation cache: collation_cache_insert) and 66ac94cdc79216e85f4c9e2d445c6f025653668e (libc code moved to pg_locale_libc.c), rewrote locale construction. - Ported master's force-lookup into upstream's shape: if (!found || yb_force_locale_lookup) invalidates the entry so create_pg_locale re-creates it. The YbCheckUnsupportedLibcLocale port into create_pg_locale_libc remains an existing YB_TODO_PG19MERGE. - src/postgres/src/backend/utils/activity/backend_status.c: - master commits 8b8126b8f20dbbeed3e73ccc6abadf2ac5659120 and b090a837cf893d4917ea900733fbc5d358a8d484 extended the st_userid backend-type condition (YB_YSQL_CONN_MGR_CTRL, YbIsInternalConnBackendType). - upstream PG commit c76db55c9085d0b7984ea337576e45a7d1268b97 split pgstat_bestart into initial/final phases; on pg19 the dbname/userid capture lives in pgstat_bestart_final. - Took upstream's structure and ported master's condition into pgstat_bestart_final (replacing the explicit YB_AUTO_ANALYZE_BACKEND, which YbIsInternalConnBackendType covers). - src/postgres/src/backend/utils/adt/ri_triggers.c: - RI_ConstraintInfo, forward decls, YbFillPKFromFKSlot caller, comments: - master commits 24816e0605138d73d3438e3ab92a2f886e9640e4, ffbccc5fd6fb435f65ff9cf85708eebcce20dbc9 and 659f0bcfa14186443f8ca06122d7044587414745 added the FK->PK coercion cache (yb_cast_paths_valid, yb_fk_to_pk_castfinfo, Yb*Coercion* helpers, YbInvalidateRIFkPkCastCacheCallBack). - YB pg19 (initial merge + 924d8b7d065b5527c347333dc0d00484166b602f) carries the FastPath FK design (FastPathMeta, ri_FastPath* helpers) (upstream PG commit b7b27eb41a5cc0b45a1a9ce5c1cde5883d7bc358); upstream PG commits ee642cccc43ca1a0ff4a4af2a457208b919af017 (SysCacheIdentifier type) and 086c84b23d99c2ad268f97508cd840efc1fdfd79 (is_restrict argument to ri_ReportViolation) changed the surrounding signatures. - Combined: the pg19 struct fields/decls plus master's coercion fields/decls; master's callback def ported to SysCacheIdentifier; master hoisted the pkslot creation out of YbFindReferencedPartition into its caller YBCBuildYBTupleIdDescriptor (the slot is now passed in as a parameter); kept master's location and checked-return usage, with the pg19 3-argument MakeTupleTableSlot signature (upstream PG commit c456e39113809376f6604e720910ccd24e18e034 added the flags argument). - (auto-merged; build fix) dclist_foreach in YbInvalidateRIFkPkCastCacheCallBack (upstream PG commit 7c335b7a20278079e796d62122ef4b821c7fcdf5 made ri_constraint_cache_valid_list a dclist). - src/postgres/src/backend/utils/cache/syscache.c / src/postgres/src/include/utils/syscache.h: - cacheinfo and SysCacheIdentifier: - master commit c037840501f189fb64f0a942a5ad2435b5aecdf8 refactored the YB arrays behind a single YB_CATCACHE_LIST macro (single source of truth generating yb_cache_index_name_table, SysCacheName, yb_catalog_cache_tables, YbCheckSysCacheNames, YbCheckCatalogCacheIds, and per-entry cache-id static_asserts). - master commit f2875e8a94d3c4114de897061fdceb36ff33fb34 (CVE import) hand-added the EXTENSIONOID cache to the PG15 arrays: #include catalog/pg_extension.h, an out-of-alphabetical-order cacheinfo entry (kept out of order to avoid an ABI break), the name-table entry, and YB_CHECK_CATALOG_CACHE_ID(EXTENSIONOID, 85). - upstream PG commit 9b1a6f50b91dca6610932650c8c81a3c924259f9 generates cacheinfo and the SysCacheIdentifier enum from catalog files (catalog/syscache_info.h, catalog/syscache_ids.h, alphabetical); the YB pg19 side maintained the YB arrays in that order (SysCacheSize == 95, including the propgraph caches and EXTENSIONOID). - Adopted master's macro machinery with the list regenerated from the pg19 arrays (95 entries in PG19 enum order; EXTENSIONOID lands at its generated position, 28, not master's appended 85); kept upstream's generated cacheinfo, which subsumes master's include and hand entry; syscache.h keeps the generated enum (dropped master's PG15 enum). - src/postgres/src/backend/utils/misc/guc.c / guc_parameters.dat / guc_tables.c / guc_hooks.h: - includes (guc.c): kept both. - ConfigureNames arrays vs guc_parameters.dat: - master added 26 new YB GUCs, removed yb_enable_concurrent_ddl (b8f0648a398a3ebf964559b77b4401ba5a9a1b64), yb_enable_index_backfill_column_projection, yb_test_stay_in_global_catalog_version_mode (664b88d2b2bb49103dcf137bdceaf01e08803161), and flipped the yb_cdcsdk_stream_tables_without_primary_key default to true (1cbc75d9f0cec24394608319719cfb8e073431f0). - upstream PG commit 63599896545c7869f7dd28cd593e8b548983d613 generates the GUC tables from guc_parameters.dat; YB pg19 commit 11876241ac840ae4cef033971a1082399ff687bd migrated all YB GUCs into that .dat. - Ported: 25 new entries added to guc_parameters.dat in sorted position (bool/int/string/enum forms preserved incl. hooks and GUC_LIST_INPUT/GUC_EXPLAIN flags); 3 dead entries removed; the default flip applied. yb_force_tablespace_locality_oid (PGC_OID) remains in the pre-existing #if 0 YB_TODO_PG19MERGE block pending OID-GUC support in the generated tables. The backing variables for the two new master GUCs whose definitions lived in master's guc.c (yb_enable_advanced_index_cond_fold, yb_enable_pg_subscription) are defined in pg_yb_utils.c per the pg19 convention (caught at link time). - existing .dat registrations updated to master's entry changes: defaults yb_enable_derived_equalities/yb_enable_derived_saops/yb_enable_pg_stat_statements_docdb_metrics false->true, yb_max_merge_scan_streams 0->64, yb_max_num_invalidation_messages 4096->8192, yb_parallel_range_size 1MB->16MB, yb_test_fail_next_ddl max 4->5 (new 5=conflict mode); GUC_NO_SHOW_ALL added to yb_query_diagnostics_disable_database_connection_bgworker; description updates on yb_enable_upsert_mode, yb_invalidation_message_expiration_secs, yb_user_ddls_preempt_auto_analyze. - hook functions: master declared its new hooks static in guc.c; YB pg19 commit 11876241ac840ae4cef033971a1082399ff687bd deleted guc.c's static hook decl block (hooks are non-static, declared in guc_hooks.h). Took master's yb_parse_command_tag_list, yb_check/assign_extra_commands_to_retry[_in_proc] and check_yb_enable_new_relation_fastpath_write[_in_txn_blocks] in the pg19 convention (non-static, declared in guc_hooks.h); dropped master's #include "guc-file.c" (removed on PG19 by the guc file split). - yb_test_force_parallel_options enum array added to guc_tables.c beside the other YB enum-options arrays. - GUC_REPORT gating: took master's yb_should_report_guc(record) call, replacing the initial-merge YB_TODO_PG19MERGE placeholder, and kept upstream PG19's !(record->status & GUC_NEEDS_REPORT) guard. - yb_silence_advisory_locks_not_supported_error: took master's deprecation rework (assign hook converted to a check hook that warns unconditionally, 'Deprecated. This is no-op.' description), ported to the pg19 registration: check_hook field in guc_parameters.dat, non-static hook declared in guc_hooks.h. - YbDbAdminVariables: kept the pg19 pg_attribute_unused() qualifier and took master's new backtrace_functions entry. - src/postgres/src/backend/postmaster/postmaster.c (+ tcop/backend_startup.c): - ProcessStartupPacket / BackendInitialize: - master commits reworked the startup packet: d8cada545783bebf57d57f90fc4c09c9784bfa45 replaced the yb_auto_analyze param with yb_internal_conn_kind; b18920d1f848559ba6bdb50aae0899e40ca6423e added the yb_is_control_conn param and the set-only yb_conn_mgr_is_auth_passthrough_backend flag; 113d63f1b9db4a7ee78d8070932f6c0dfe5dc16a renamed yb_has_auth_passthrough_failed to _finished; e538571b85d5ae7e8a0c20b90c0254390cae5d67 fixed the salen handling. - upstream PG commit 05c3980e7f473ac2061dad9bbb7a9f0ede0279d9 moved ProcessStartupPacket/BackendInitialize into tcop/backend_startup.c; the pg19 live copies carried the old yb_auto_analyze handling. - Took the pg19 postmaster.c (no dead copies); ported all of master's startup-packet changes into backend_startup.c's live copies (added yb_internal_conn.h include, removed it from postmaster.c). - src/postgres/src/backend/libpq/hba.c: - load_hba / load_ident file open and tokenize: - master commit 8156be614b1380a766c5c998b5b23c231320e145 added validation mode (yb_validate_conf_file: open the supplied file, escalate errors to ERROR, only validate). - upstream PG commit a54b658ce77b6705eb1f997b416c2e820a77946c replaced AllocateFile/linecxt with open_auth_file/tokenize_auth_file(+depth) for auth-file inclusions. - Ported master's yb_filename/yb_elevel selection into upstream's open_auth_file idiom for both functions; kept upstream's cleanup comments plus master's validation-mode note. - src/postgres/src/interfaces/libpq/fe-connect.c: - PQconninfoOptions and freePGconn: - master commit 8b8126b8f20dbbeed3e73ccc6abadf2ac5659120 added the yb_internal_conn_kind option (yb_internal_conn registry framework); master commit d8cada545783bebf57d57f90fc4c09c9784bfa45 then removed the yb_auto_analyze option by migrating auto-analyze onto it. - upstream PG19 added new libpq options (OAuth, scram key export, sslkeylogfile) and changed the freePGconn shape (several upstream commits; took the pg19 file's shapes wholesale). - Took the upstream shapes with the yb option renamed/replaced per master. - src/postgres/src/backend/utils/init/postinit.c: - master commit 9daac48b6ea598067b8234fb635e664e527e4fcf added the pg_authid catalog-cache preload before authentication. Took it; dropped the adjacent GetTransactionSnapshot() call that upstream PG commit 952365cded635e54c4177399c0280cb7a5e34c11 removed. - (auto-merged; build fix) in YbCheckMyDatabase, master's pre-auth collation-version check read Anum_pg_database_daticulocale; upstream PG commit f696c0cd5f299f1b51e214efc55a22a782cc175d renamed it to datlocale. Changed it to datlocale. - src/postgres/src/backend/parser/parser.c / src/postgres/src/include/parser/parser.h: - master commit 7c50c626fa23af268156ae2ca3528ed354204842 added RAW_PARSE_YB_SPLIT_CLAUSE / MODE_YB_SPLIT_CLAUSE (yb_presplit reloption parsing); the pg19 side carries upstream PG19's designated-initializer mode table. Added the enum member and a designated-initializer entry. - src/postgres/src/backend/tcop/postgres.c: - includes, exec_bind/execute/describe tails, extended-protocol message cases: - master commit df6f6f02ddb047b36c3baf820d2eb029908c6233 added the per-message distributed-tracing spans (yb_maybe_start_trace_root_span, YB_DIST_TRACE_START/END_SPAN); master commit 103f54edc9b5c2b86c5e55079c0ad6f9f89b9d2a added YbDistTraceEndOpenNodeSpans on top. - master commit e55ee8774a5ff7780ff783d65184c84c062c27a6 added the YbAsh query-plan-pair set/reset per message (ASH query_id leakage fix). - master commit ffdecaabc93d0d88e2e2a843a4232034eb9c59af added the conn-mgr Sync-ack ('Y', YB_BE_SYNC_ACK) message in exec_sync. - upstream PG commits f4b54e1ed9853ab9aff524494866823f951b1e7f (PqMsg_* protocol macros), a8a00124f1ddb9c8506ff75317f6dd64cc480e61 (valgrind_report_error_query), and d0eb4297cc5bbfb02b64fdaaf5510415e92e5312 (EndImplicitTransactionBlock in Sync for pipelining) changed the surrounding code. - Combined throughout: upstream constants/calls plus master's tracing and Sync-ack additions; exec_parse_message keeps master's char yb_firstchar argument. - src/postgres/src/bin/pg_dump/pg_dump.c: - long options / handlers / checks / help / dumpable-extension filter: - master commit 95306ec58b4b56c50b822f7b4d6576ef96af3b12 added --rename-database/--rename-owner (case values 25/26 on master); master commits also excluded postgres_fdw and pg_stat_statements from selectDumpableExtension (both auto-created under global views by default). - upstream PG commits 71ea0d6795438f95f4ee6e35867058c44b270d51 (--restrict-key, psql meta-command restriction) and 7c8280eeb5872f5c2663b562a9c6fcf8ec8a4b82 (check_mut_excl_opts refactor of conflicting options) occupy case 25 and the checks region on pg19. - Took both: rename options renumbered to 27/28; master's requires-create checks kept beside upstream's check_mut_excl_opts; help text merged; extension filter takes master's list. - src/postgres/src/bin/pg_dump/pg_backup_archiver.c: - _printTocEntry owner block: - master commit 95306ec58b4b56c50b822f7b4d6576ef96af3b12 added yb_effective_owner() (--rename-owner rewrites owners equal to the source DB owner) at the PG15 OWNER TO emission sites. - the pg19 side has upstream's _getObjectDescription-based emission block with the YB use_roles/--dump-role-checks framing (YB pg19 commit 9a7d4ae905a808a640e80aba5ade7489c83742ff), plus a stale #if 0 YB_TODO_PG19MERGE copy of the old code. - Applied yb_effective_owner() in the live block and the BLOB METADATA psprintf; removed the stale #if 0 copy. Tested by PgCloneTest.UserIsSet. - java/yb-pgsql/src/test/resources/TestYsqlDump/* (7 conflicted goldens): - master changed dump content (yb_presplit='' reloptions from 7c50c626fa23af268156ae2ca3528ed354204842, yb_global_views_user role/grants, dump-role-checks blocks, grantor changes). - YB pg19 commit 6c748cb711bdc90defe31a83a74fed5644dd7465 reblessed for upstream PG19's dump format: \restrict/\unrestrict from upstream PG commit 71ea0d6795438f95f4ee6e35867058c44b270d51; 19devel version headers; \du without Member of; role-membership GRANTs carry INHERIT TRUE and GRANTED BY postgres instead of GRANTED BY yugabyte_test (upstream PG commit ce6b672e4455820a0348214be0da1a024c3f619f: check_role_grantor attributes a superuser grant without explicit GRANTED BY to the bootstrap superuser, not the executing role). - Combined: the pg19 format everywhere, master's new content (GV role row, GV grant, role-check block, yb_presplit WITH clauses via auto-merge) formatted pg19-style, then regenerated from TestYsqlDump runs (only \unrestrict/\restrict re-bracketing around \if blocks and the pg_stat_statements_reset minmax_only signature changed). The corresponding TestYsqlDump methods pass. - src/postgres/src/test/regress/expected/yb.orig.distinct_pushdown_{base,join,pred}.out: - master commit 0bc300132cf78d88e43790d58bde4690c1ed1de5 (echo query text in parameterized_query tests) and master commit 3114bc8029381bb8189e756c6ee7b36f13832bc4 (auto-infer iteration in parameterized_query tests) changed the test harness and re-blessed the goldens. - YB pg19 commit aa3c6c66658da2caf18951052ba9bb388e97e289 reblessed the old-harness goldens for PG19 EXPLAIN format (actual rows=N.00 decimals). - Took master's goldens (harness echo/auto-infer content) with PG19's rows=N.00 EXPLAIN ANALYZE conversion applied (block rulers recomputed), matching the format state of YB pg19 commit aa3c6c66658da2caf18951052ba9bb388e97e289. - src/postgres/third-party-extensions/pgaudit/{sql,expected}/yb.port.pgaudit.*: - master commit 61e8975d220a7d3a833c359ad5aa72c682760859 changed the pg_stat_statements drop to DROP ... CASCADE (global views by default create gv$ dependents), with the PG15 pgss version string 1.10-yb-2.1. - YB pg19 commit ceeb2ea789a6704e7de4d4895329bae6ec692640 (pgaudit enablement) updated the test to the PG19 pgss version 1.13-yb-1.0 and the new pgaudit's statement logging (no trailing semicolon in the logged statement). - Combined: pg19's version string and log format with master's CASCADE drop and its NOTICE. - src/postgres/src/backend/nodes/{copyfuncs,outfuncs,readfuncs}.c: - upstream PG commit 964d01ae90c314eb31132c2e7712d5d9fc237331 generates node support functions (gen_node_support.pl); master's side re-presented the PG15 handwritten functions carrying the YB node fields (new fields added by master commit 2af35282ad6c7f3ea8fa494b517d9e2de78d0a30). Took the generated includes; the fields live in plannodes.h/pathnodes.h and are picked up by the generator. - src/postgres/src/backend/catalog/pg_depend.c: - alignment-only conflict at the markers; took the pg19 side. - src/postgres/src/backend/catalog/system_functions.sql (+ pg_auth_members.dat, pg_proc.dat): - upstream PG commit f95d73ed433207c4323802dc96e52f3e5553a86c (built-in function ACLs in pg_proc.dat) and upstream PG commit 759b03b24ce96f0ba6d734b570d1a6f4a0fb1177 (built-in function default args in pg_proc.dat) removed the GRANT/REVOKE and CREATE OR REPLACE blocks from system_functions.sql. Took the pg19 side; re-ported each master addition to its PG19 location: - master commit 56ba33eab25349d426268bb4ee9e17f5540a656a (GRANT pg_read_all_stats TO yb_global_views_user): added a pg_auth_members.dat membership entry (inheriting membership, matching GRANT semantics). - master commit 9b08b0010cf1c911b2e25367c955c7af5babd0e2 (REVOKE EXECUTE ... FROM public on yb_replication_origin_session_setup_shared/reset_shared): added proacl => '{POSTGRES=X}' to their pg_proc.dat entries (8894, 8895), matching upstream's pg_replication_origin_session_setup entry. - src/postgres/src/backend/catalog/Makefile: - master commit 61e8975d220a7d3a833c359ad5aa72c682760859 added yb_global_views.sql to install/uninstall; the pg19 side carries upstream PG19's target layout. Took the pg19 uninstall line with yb_global_views.sql appended. - src/postgres/src/Makefile.global.in: - master commit 24a338fbab36a7934fa0988ebe307fd6f46a7d51 removed the YB-specific -I$(YB_SRC_ROOT)/src from CPPFLAGS (PGXS extensions build against installed headers); upstream PG19 added LIBNUMA/LIBURING flags to the same line. With master's YB addition gone, no YB delta remains: the resolved line is upstream PG19's verbatim. - src/postgres/src/include/Makefile: - master commit 38489ebccf39dad70a8f8d66822f2d551d275721 added the YB_PGGATE_HEADERS install; upstream PG19 delegates catalog header install to the catalog subdirectory. Kept both. - src/postgres/src/include/commands/explain.h (+ explain_state.h): - upstream PG commit c65bc2e1d14a2d4daed7c1921ac518f2c5ac3d17 split ExplainState into explain_state.h (extensible EXPLAIN); master added ybMaskConstants to the struct. Took upstream's split and added ybMaskConstants in explain_state.h. - src/postgres/src/include/catalog/pg_proc.dat: - yb_fix_catalog_version_table: master commit 664b88d2b2bb49103dcf137bdceaf01e08803161 rewrote descr/prosrc (shared catalog version mode removed); YB pg19 (initial merge) renumbered the oid 8060 -> 9815 (collision with an upstream PG19 oid). Took the pg19 oid with master's descr. - src/postgres/src/backend/replication/slot.c: - ReplicationSlotValidateName: - master commit b462225c7a9664f6761f873b9cb766be371f6d07 added the YbNotificationsSlotPrefix reservation (ereport style). - upstream PG commit 228c3708685542d34e6f02c74240656327a5c622 and others added the pg_conflict_detection reserved-slot check and converted validation to err_code/err_msg/err_hint out-params. - Ported master's check to the out-param style after upstream's check. - YbReplicationSlotCreateForDB: - master commit b462225c7a9664f6761f873b9cb766be371f6d07 removed Assert(ReplicationSlotValidateName(...)). Took master's removal. - (auto-merged; build fix) drop-path ReplicationSlotAcquire gets error_if_invalid=false (upstream PG commit f41d8468ddea34170fe19fdc17b5a247e7d3ac78), matching upstream's drop path. - src/postgres/src/backend/replication/logical/logicalfuncs.c: - pg_logical_slot_get_changes_guts loop: - master commit 8710fe7f3d0b7b8ba61627ef7d291ca577f7c06f added the yb_record/GetConsistentChanges retry logic (query API pull mode), rewriting the PG15-shaped decode loop. - upstream PG commit 7dcea51c2a4dcf7c512bbd4f618d1d3620f9d3d7 added old_resowner tracking with Assert(CurrentResourceOwner == old_resowner) after LogicalDecodingProcessRecord. - Combined: master's loop structure with upstream's Assert inside the process-record branch. - src/postgres/yb-extensions/yb_pg_metrics/yb_pg_metrics.c: - YbStatementType per-cache enum members (CatCacheIdMisses_N / CatCacheTableMisses_N): - master commit 16d281b6a6e7fa9458c2db8c74c9ff1243348b76 removed them, replacing per-cache ybpgm_table slots with the per-DB YbpgmCatalogCacheMetricsEntry array (sized by SysCacheSize / YbNumCatalogCacheTables at compile time). - the pg19 side had expanded the same members to the PG19 cache set (SysCacheSize == 95). - Took master's removal. - src/postgres/src/tools/pgindent/typedefs.list: kept the pg19 side (ExtensionLocation, from upstream PG commit f3c9e341cdf167ae3378e74e770558e81f9aa48e, extension_control_path work); master's additions auto-merged. - src/postgres/src/test/regress/expected/yb.orig.create_table_like.out: - \d output: YB pg19 rebless carries upstream PG19's "Not-null constraints:" \d section; master commit 7c50c626fa23af268156ae2ca3528ed354204842 added the yb_presplit Options lines. Kept both sections per block. - src/postgres/src/include/nodes/pathnodes.h (auto-merged; build fix): - PlannerInfo.yb_tserver_uuids: - master commit 98418b7bc5149e5a2253b30968e2efca05f33343 added const char **yb_tserver_uuids (per-tserver federated child UUIDs); fine under PG15's handwritten node functions. - upstream PG commit 964d01ae90c314eb31132c2e7712d5d9fc237331 generates node support from pathnodes.h; gen_node_support.pl has no handler for char ** ("could not handle type char** in struct PlannerInfo"). - Marked the field pg_node_attr(read_write_ignore), matching upstream's treatment of PlannerInfo fields the generator cannot serialize (boundParams, partition_directory); PlannerInfo is already no_copy_equal/no_read so only outfuncs is affected. - src/postgres/src/backend/parser/gram.y (auto-merged; build fix): - yb_presplit_value production: - master commit 7c50c626fa23af268156ae2ca3528ed354204842 added the production using the YB filter token _YB_SPLIT_P (base-era spelling). - upstream PG commit 4b3d173629f4cd7ab6cd700d1053af5d5c7c9e37 (ALTER TABLE ... SPLIT PARTITION) made SPLIT a real keyword; the initial merge replaced _YB_SPLIT_P with SPLIT throughout (YbOptSplit already uses it). - Renamed the tokens in the merged-in production to SPLIT. - src/postgres/src/include/access/reloptions.h + src/postgres/src/backend/access/common/reloptions.c (auto-merged; build fix): - YbAddColocationIdReloption signature: - master commit a20345c4c428f7ebc0502a3e98e13f5e557d7003 added YbAddColocationIdReloption(bits32 kinds). - upstream PG commit bab2f27eaaad77f799ecc224f9e11b09adb07d5a removed the bits8/bits16/bits32 typedefs; PG19 reloption kind parameters are plain uint32. - Changed the parameter to uint32 in both the declaration and definition. - src/postgres/src/backend/catalog/partition.c (auto-merged; build fix): - yb_has_ancestor_partition_attrs: - master commit a7c0df145723cc64a6ae4c5d0d5194ee03d0238e added the function using the PG15-era 3-argument build_attrmap_by_name and destructive bms_first_member iteration. - upstream PG commit ad86d159b6ab90b195b06fb5c7b593900a7f9cd8 added the missing_ok argument to build_attrmap_by_name; upstream PG commit 462bb7f12851c215dfc21a88ae0ed4bf7fcb36a3 removed bms_first_member. - Kept both PG and YB parameters (added false missing_ok) and converted the loop to non-destructive bms_next_member iteration. - src/postgres/src/backend/commands/{explain.c, propgraphcmds.c}, src/postgres/src/backend/partitioning/partbounds.c (auto-merged; build fix): - deparse_expression callers: - master commit a1e9cfada4d9c95fd965ef6085ea38dc04ba5021 widened deparse_expression to six arguments (yb_pretty, yb_maskconstants). - upstream PG19-only call sites master could not know about still passed four arguments: show_window_keys in explain.c, the pg_propgraph expression deparse in propgraphcmds.c, and the SPLIT PARTITION error path in partbounds.c. - Extended those callers: es->ybMaskConstants in explain.c (matching master's idiom in the same file), false/false for the catalog and error-message sites. - src/postgres/src/backend/utils/adt/uuid.c (auto-merged; build fix): restored the utils/builtins.h include master's yb_convert_uuid_to_yb_uuid_string_repr needs (hex_encode) in a /* YB includes */ section. - src/postgres/src/include/tcop/cmdtag.h + src/postgres/src/backend/tcop/cmdtag.c + src/postgres/src/backend/utils/misc/guc.c (auto-merged; build fix): master commit 6b94be9cbed701a8422ce3193aca5ea8362c013d sizes the yb_extra_commands_to_retry bool arrays with the COMMAND_TAG_NEXTTAG enum sentinel, which upstream PG commit 1f7452fa598ee5f1ed32f1fcce101c63c6f1933f removed (tag_behavior[] is now implicitly sized). Added YbGetCommandTagCount() in cmdtag.c returning lengthof(tag_behavior), used it for the array sizing in guc.c's yb_parse_command_tag_list. - src/postgres/src/include/executor/execdesc.h + src/postgres/src/backend/executor/execMain.c (auto-merged; build fix): yb_query_stats typed as NodeInstrumentation (the struct carrying yb_instr after upstream PG commit 5a79e78501f46bd3ac7fbd0ff84cf1e20dbafd19 split node vs query instrumentation), allocated via InstrAllocNode. - src/postgres/src/backend/utils/misc/yb_ash.c (auto-merged; build fix): post_parse_analyze hook takes const JumbleState *. - src/postgres/src/backend/utils/misc/yb_dist_trace.c (auto-merged; build fix): ExecutorRun hook loses execute_once (upstream PG commit 3eea7a0c97e94f9570af87317ce3f6a41eb62768). - src/postgres/src/backend/replication/logical/origin.c (auto-merged; build fix): master's yb_replication_origin_session_setup_shared / yb_replication_origin_session_reset_shared changed as per upstream PG commits 1fdbca159e0055fefc0b391ec09520d0b3bd9248 (RepOriginId -> ReplOriginId) and 8f1e2dfe033e9a3236265c3b9f61bd226f4a8f54 (session globals consolidated into replorigin_xact_state). - src/postgres/src/backend/utils/activity/{pgstat_backend.c, pgstat_io.c} (auto-merged; build fix): - pgstat_tracks_backend_bktype / pgstat_tracks_io_bktype switches: - master added BackendTypes YB_YSQL_CONN_MGR_CTRL, YB_RELCACHE_INIT_BACKEND, YB_GLOBAL_VIEW_BACKEND respectively. - upstream PG19's exhaustive -Wswitch BackendType switches (pgstat_backend.c is new upstream) did not know them. - Added the three to the existing YB_TODO_PG19MERGE opt-out arms (return false, matching the other YB backend types). - src/yb/yql/pgwrapper/pg_catalog_perf-test.cc (auto-merged; build fix): - master commit 109d0ed265f39391b634a61465dbcf9dd8ce6539 added auth-prefetch cache test configs using kResponseCacheSize5MB. - YB pg19 commit ef2c04d3fad65a680c67ca97a5c3c83bd0976a0a renamed the constant to kResponseCacheSize6MB (PG19 catalogs need a larger response cache). - Pointed the new configs at the 6MB constant. Test Plan: - Jenkins: yugabyte-db-phabricator run on this revision (clang21 axis) for the full suite; known residuals are listed at the end of the Summary. - Confirmed locally (macOS clang21 release): - ./yb_build.sh release green; yb-ctl single-node cluster: initdb, connect (TCP + unix socket + bare ysqlsh), CREATE TABLE / INSERT / SELECT / ALTER SEQUENCE / DROP smoke tests pass. - TestPgRegressContribYbPostgresFdw passes. - TestPgRegressContribYbGlobalViews passes. - TestYsqlDump: ysqlDumpAllWithoutYbMetadata and ysqlDumpColocatedTablesWithTablespaces pass; ysqlDumpAllWithYbMetadata, ysqlDumpAllWithDumpRoleChecks, ysqlDumpColocatedDB and ysqlDumpLegacyColocatedDB show no golden diffs (their local runs end in a macOS-only tserver teardown abort -- waitpid EINTR -- that does not occur on Linux CI). Reviewers: #yba-api-review!, anijhawan, nbhatia, nsingh Subscribers: yugaware, nikhil Differential Revision: https://phorge.dev.yugabyte.com/D56481

[BACKPORT 2026.1][#30831] CDC: Align the WAL sort window for GetChanges across peers to prevent data loss Summary: #### Backport Description No merge conflicts were encountered. #### Original Description ##### Code changes summary There exists a possible case where CDC can miss out on sending some records when the leadership of a tablet changes. This issue is hit when CDC is streaming WAL records which have been applied out of order and the tablet's leadership changes during this. Two things to remember: # When streaming Ops which have been applied out of order, CDC does not move its checkpoint forward. Instead the `wal_segment_index` field is used to indicate how many out-of-order records beyond the checkpoint (from_op_id) have been streamed from the sorted WAL Ops list. The subsequent call skips over these many records from the start of the sorted array of WAL Ops. # Another thing to note here is that each peer of the tablet has same committed WAL Ops but the segment boundaries can differ on each replica. These two factors combined result into a CDC data loss. Following example illustrates how this CDC data loss manifests: ``` GetChanges 1 was served by peer "a". The following WAL Op Ids were read Before sorting: 1.1, 1.2, 1.3, 1.4, 1.5, 1.6, 1.7, 1.8, 1.9, 1.10 After sorting : 1.1, 1.2, 1.3, 1.6, 1.9, 1.8, 1.5, 1.4, 1.7, 1.10 CDC reads one segment of WAL at a time. Sorting is done by CDC according to the commit time. Assume that GetChanges response size limit reached after decoding 1.8. Response: {1.1, 1.2, 1.3, 1.6, 1.9, 1.8}; checkpoint = 1.3, wal_segment_index = 3 ------------------------------------------------------------------------------------------ GetChanes 2 was served by peer "b". But its wal segment was rolled over earlier: Before sorting : 1.1, 1.2, 1.3, 1.4, 1.5, 1.6, 1.7 After sorting : 1.1, 1.2, 1.3, 1.6, 1.5, 1.4, 1.7 GetChanges read: 1.6, 1.5, 1.4, 1.7 --> Because checkpoint was 1.3 Since wal_segment_index was 3, we skipped the first three wal ops read and directly processed index 1.7. As a result, ops with index 1.4 and 1.5 were never streamed. ``` Overall, following conditions are needed to hit this issue: # CDC should be lagging. # The leadership of a tablet should change between two successive GetChanges calls. # The batch of records being streamed by CDC in the first GetChanges call should have out-of-order apply operations. # Some of these out-of-order ops lie in a different WAL segment, in one of the peers. This peer now becomes the new leader. # Next GetChanges call to this new leader will miss out on sending data. Now to fix this issue, a new field called `max_index_in_sort_window` in the GetChangesRequestPB and in GetChangesResponsePB. A GetChanges call populates this in the response with the max index it read from the WAL. We ensure that max_index_in_sort_window never moves backward. The subsequent GetChanges call receives this value in its request. It will read the WAL such that maximum index read from the WAL is >= `max_index_in_sort_window` value received in the request. If for any reason (ex: shutdown / WAL read failure etc) we are not able to read the WAL till max_index_in_sort_window, we do not ship any records in this GetChanges call as we cannot be sure of its correctness. The safe time, checkpoint and max_index_in_sort_window are kept unchanged in this case. **The Invariant**: In steady state (no restarts), if GetChanges on peer "A" of a tablet reads WAL upto point x, then the subsequent GetChanges calls on any peer of the same tablet (not necessarily A), will read WAL at least upto the point x. In the example above, GetChanges 1 would send max_index_in_sort_window = 10. GetChanges 2 would now read an additional segment to get all the ops till index 10. The WAL read after sorting would become {6, 9, 8, 5, 4, 7, 10}. We would correctly skip over {6, 9, 8} since wal_segment_index is 3 and start streaming from 5. This way no data would be missed by CDC. ##### Considerations for connector No changes required in the logical replication connector as the forwarding of max_index_in_sort_window and wal_segment_index from previous response to the next request is done by the virtual WAL. However, in gRPC model this work is owned by the connector. Hence corresponding change would be needed there. ##### Upgrade / Rollback safety This diff introduces an optional field `optional int64 max_index_in_sort_window` (default 0) in GetChangesRequestPB and in GetChangesResponsePB. Since it is optional, if the incoming request does not set max_index_in_sort_window, then we will proceed with the value 0, which is as good as disabling this mechanism. Hence the change is upgrade and rollback safe. ##### Considerations for colocated tables NA Original commit: f3a4c239e7a1e24e8c305e8ee5d885caf37d5b5a / D56990 Test Plan: Existing CDC unit tests and stress tests. An exact UT to deterministically replicate the above mentioned CDC data loss failure could not be written since it needs reliable out of order applies. Reviewers: skumar, asrinivasan, devansh.singhal Reviewed By: devansh.singhal Subscribers: ycdcxcluster, kannan Tags: #jenkins-ready Differential Revision: https://phorge.dev.yugabyte.com/D57063

2026-08-20

[BACKPORT 2025.2][#30831] CDC: Align the WAL sort window for GetChanges across peers to prevent data loss Summary: #### Backport Description No merge conflicts were encountered. #### Original Description ##### Code changes summary There exists a possible case where CDC can miss out on sending some records when the leadership of a tablet changes. This issue is hit when CDC is streaming WAL records which have been applied out of order and the tablet's leadership changes during this. Two things to remember: # When streaming Ops which have been applied out of order, CDC does not move its checkpoint forward. Instead the `wal_segment_index` field is used to indicate how many out-of-order records beyond the checkpoint (from_op_id) have been streamed from the sorted WAL Ops list. The subsequent call skips over these many records from the start of the sorted array of WAL Ops. # Another thing to note here is that each peer of the tablet has same committed WAL Ops but the segment boundaries can differ on each replica. These two factors combined result into a CDC data loss. Following example illustrates how this CDC data loss manifests: ``` GetChanges 1 was served by peer "a". The following WAL Op Ids were read Before sorting: 1.1, 1.2, 1.3, 1.4, 1.5, 1.6, 1.7, 1.8, 1.9, 1.10 After sorting : 1.1, 1.2, 1.3, 1.6, 1.9, 1.8, 1.5, 1.4, 1.7, 1.10 CDC reads one segment of WAL at a time. Sorting is done by CDC according to the commit time. Assume that GetChanges response size limit reached after decoding 1.8. Response: {1.1, 1.2, 1.3, 1.6, 1.9, 1.8}; checkpoint = 1.3, wal_segment_index = 3 ------------------------------------------------------------------------------------------ GetChanes 2 was served by peer "b". But its wal segment was rolled over earlier: Before sorting : 1.1, 1.2, 1.3, 1.4, 1.5, 1.6, 1.7 After sorting : 1.1, 1.2, 1.3, 1.6, 1.5, 1.4, 1.7 GetChanges read: 1.6, 1.5, 1.4, 1.7 --> Because checkpoint was 1.3 Since wal_segment_index was 3, we skipped the first three wal ops read and directly processed index 1.7. As a result, ops with index 1.4 and 1.5 were never streamed. ``` Overall, following conditions are needed to hit this issue: # CDC should be lagging. # The leadership of a tablet should change between two successive GetChanges calls. # The batch of records being streamed by CDC in the first GetChanges call should have out-of-order apply operations. # Some of these out-of-order ops lie in a different WAL segment, in one of the peers. This peer now becomes the new leader. # Next GetChanges call to this new leader will miss out on sending data. Now to fix this issue, a new field called `max_index_in_sort_window` in the GetChangesRequestPB and in GetChangesResponsePB. A GetChanges call populates this in the response with the max index it read from the WAL. We ensure that max_index_in_sort_window never moves backward. The subsequent GetChanges call receives this value in its request. It will read the WAL such that maximum index read from the WAL is >= `max_index_in_sort_window` value received in the request. If for any reason (ex: shutdown / WAL read failure etc) we are not able to read the WAL till max_index_in_sort_window, we do not ship any records in this GetChanges call as we cannot be sure of its correctness. The safe time, checkpoint and max_index_in_sort_window are kept unchanged in this case. **The Invariant**: In steady state (no restarts), if GetChanges on peer "A" of a tablet reads WAL upto point x, then the subsequent GetChanges calls on any peer of the same tablet (not necessarily A), will read WAL at least upto the point x. In the example above, GetChanges 1 would send max_index_in_sort_window = 10. GetChanges 2 would now read an additional segment to get all the ops till index 10. The WAL read after sorting would become {6, 9, 8, 5, 4, 7, 10}. We would correctly skip over {6, 9, 8} since wal_segment_index is 3 and start streaming from 5. This way no data would be missed by CDC. ##### Considerations for connector No changes required in the logical replication connector as the forwarding of max_index_in_sort_window and wal_segment_index from previous response to the next request is done by the virtual WAL. However, in gRPC model this work is owned by the connector. Hence corresponding change would be needed there. ##### Upgrade / Rollback safety This diff introduces an optional field `optional int64 max_index_in_sort_window` (default 0) in GetChangesRequestPB and in GetChangesResponsePB. Since it is optional, if the incoming request does not set max_index_in_sort_window, then we will proceed with the value 0, which is as good as disabling this mechanism. Hence the change is upgrade and rollback safe. ##### Considerations for colocated tables NA Original commit: f3a4c239e7a1e24e8c305e8ee5d885caf37d5b5a / D56990 Test Plan: Existing CDC unit tests and stress tests. An exact UT to deterministically replicate the above mentioned CDC data loss failure could not be written since it needs reliable out of order applies. Reviewers: skumar, asrinivasan, devansh.singhal Reviewed By: devansh.singhal Subscribers: kannan, ycdcxcluster Tags: #jenkins-ready Differential Revision: https://phorge.dev.yugabyte.com/D57064

[BACKPORT 2024.2][#30831] CDC: Align the WAL sort window for GetChanges across peers to prevent data loss Summary: #### Backport Description No merge conflicts were encountered. #### Original Description ##### Code changes summary There exists a possible case where CDC can miss out on sending some records when the leadership of a tablet changes. This issue is hit when CDC is streaming WAL records which have been applied out of order and the tablet's leadership changes during this. Two things to remember: # When streaming Ops which have been applied out of order, CDC does not move its checkpoint forward. Instead the `wal_segment_index` field is used to indicate how many out-of-order records beyond the checkpoint (from_op_id) have been streamed from the sorted WAL Ops list. The subsequent call skips over these many records from the start of the sorted array of WAL Ops. # Another thing to note here is that each peer of the tablet has same committed WAL Ops but the segment boundaries can differ on each replica. These two factors combined result into a CDC data loss. Following example illustrates how this CDC data loss manifests: ``` GetChanges 1 was served by peer "a". The following WAL Op Ids were read Before sorting: 1.1, 1.2, 1.3, 1.4, 1.5, 1.6, 1.7, 1.8, 1.9, 1.10 After sorting : 1.1, 1.2, 1.3, 1.6, 1.9, 1.8, 1.5, 1.4, 1.7, 1.10 CDC reads one segment of WAL at a time. Sorting is done by CDC according to the commit time. Assume that GetChanges response size limit reached after decoding 1.8. Response: {1.1, 1.2, 1.3, 1.6, 1.9, 1.8}; checkpoint = 1.3, wal_segment_index = 3 ------------------------------------------------------------------------------------------ GetChanes 2 was served by peer "b". But its wal segment was rolled over earlier: Before sorting : 1.1, 1.2, 1.3, 1.4, 1.5, 1.6, 1.7 After sorting : 1.1, 1.2, 1.3, 1.6, 1.5, 1.4, 1.7 GetChanges read: 1.6, 1.5, 1.4, 1.7 --> Because checkpoint was 1.3 Since wal_segment_index was 3, we skipped the first three wal ops read and directly processed index 1.7. As a result, ops with index 1.4 and 1.5 were never streamed. ``` Overall, following conditions are needed to hit this issue: # CDC should be lagging. # The leadership of a tablet should change between two successive GetChanges calls. # The batch of records being streamed by CDC in the first GetChanges call should have out-of-order apply operations. # Some of these out-of-order ops lie in a different WAL segment, in one of the peers. This peer now becomes the new leader. # Next GetChanges call to this new leader will miss out on sending data. Now to fix this issue, a new field called `max_index_in_sort_window` in the GetChangesRequestPB and in GetChangesResponsePB. A GetChanges call populates this in the response with the max index it read from the WAL. We ensure that max_index_in_sort_window never moves backward. The subsequent GetChanges call receives this value in its request. It will read the WAL such that maximum index read from the WAL is >= `max_index_in_sort_window` value received in the request. If for any reason (ex: shutdown / WAL read failure etc) we are not able to read the WAL till max_index_in_sort_window, we do not ship any records in this GetChanges call as we cannot be sure of its correctness. The safe time, checkpoint and max_index_in_sort_window are kept unchanged in this case. **The Invariant**: In steady state (no restarts), if GetChanges on peer "A" of a tablet reads WAL upto point x, then the subsequent GetChanges calls on any peer of the same tablet (not necessarily A), will read WAL at least upto the point x. In the example above, GetChanges 1 would send max_index_in_sort_window = 10. GetChanges 2 would now read an additional segment to get all the ops till index 10. The WAL read after sorting would become {6, 9, 8, 5, 4, 7, 10}. We would correctly skip over {6, 9, 8} since wal_segment_index is 3 and start streaming from 5. This way no data would be missed by CDC. ##### Considerations for connector No changes required in the logical replication connector as the forwarding of max_index_in_sort_window and wal_segment_index from previous response to the next request is done by the virtual WAL. However, in gRPC model this work is owned by the connector. Hence corresponding change would be needed there. ##### Upgrade / Rollback safety This diff introduces an optional field `optional int64 max_index_in_sort_window` (default 0) in GetChangesRequestPB and in GetChangesResponsePB. Since it is optional, if the incoming request does not set max_index_in_sort_window, then we will proceed with the value 0, which is as good as disabling this mechanism. Hence the change is upgrade and rollback safe. ##### Considerations for colocated tables NA Original commit: f3a4c239e7a1e24e8c305e8ee5d885caf37d5b5a / D56990 Test Plan: Existing CDC unit tests and stress tests. An exact UT to deterministically replicate the above mentioned CDC data loss failure could not be written since it needs reliable out of order applies. Reviewers: skumar, asrinivasan, devansh.singhal Reviewed By: devansh.singhal Subscribers: ycdcxcluster, kannan Tags: #jenkins-ready Differential Revision: https://phorge.dev.yugabyte.com/D57065

2026-08-19

[PLAT-21967] Use a separate private key to sign node agent JWT to remove dependency on server key Summary: This is in preparation to allow custom certs for Node Agent. The current issue is that server.key needs to be copied to YBA as it is used in signing the JWT and our cert storage for encryption in transit (EIT) which is going to be used for NA does not store it. Morever, customers are not willing to share it to YBA. This change creates a signer public and private key which are independently managed by YBA and is not tied to the custom cert. Test Plan: Manually tested. 1. Upgrade NA of nodes in a universe before this change. 2. New universe creation 3. Previous itests passed too. Reviewers: amalyshev, skhilar Reviewed By: amalyshev Subscribers: yugaware Differential Revision: https://phorge.dev.yugabyte.com/D57011

The documentation is generated from this commit.

dist-trace: propagate trace context across the RPC boundary Carry a distributed trace from an RPC caller to its callee: define the wire type, write it into the request header, decode it on the far side, and open a server span parented under the caller's client span. All of it lands together because none of the pieces is observable on its own -- a wire field nobody reads, or a reader with nothing on the wire, cannot be tested. Wire type (rpc_header.proto): - TraceContextPB (W3C trace-context) and RequestHeader.trace_context. This is the type shared by the RPC-header and shared-memory transports; the latter starts using it in a later commit. Writer (outbound_call.cc): - SetRequestParam serializes the active client span's SpanContext into the trace_context submessage while it is sizing and writing the header, so the header is still written in a single pass. ~30 bytes on the wire, and only when a trace is active; an older peer skips the unknown field. Reader (serialization.{cc,h}): - ParseTraceContext(Slice) decodes the RequestHeader.trace_context wire field; ToSpanContext(TraceContextPB) is the shared-memory equivalent; both go through BuildSpanContext and fail on a zero/invalid context. - ParseHeader captures RequestHeader.trace_context into ParsedRequestHeader. Server / inbound span (yb_rpc.{cc,h}): - YBInboundCall::ParseFrom parses the header trace_context into parent_span_context_ (best-effort: a bad context is logged, never fails the RPC). - CreateServerSpan starts a StartServerSpanWithScope child of that parent (remote) or of an explicitly passed context (local); RespondSuccess/Failure set status and end it; DropServerSpanScope releases the scope on the handler thread while the span ends later. Client local-call handoff (local_call.{cc,h}, rpc_context.cc): - LocalOutboundCall tags rpc.local_call and drops its client-span scope in the ctor (a local call never hops threads to drop it later). - otel_span_context() exposes the outbound span's context so the local inbound span can parent under it (no wire header exists locally). - RpcContext creates the server span: from the wire header for remote calls, from the outbound call's context for local calls. Process init (tserver/db_server_base.cc): - DbServerBase is the shared base of Master and TabletServer, so DbServerBase:: Init is the single place that initializes the process-wide tracer (service name = server name, node id = permanent uuid), with Shutdown tearing it down. Gated by IsDistTraceEnabled, so it is a no-op when tracing is off. Without this the server spans above are never exported, which is why the 10 lines ship here rather than as a commit of their own. Test: TestRpcSpanReachesTabletServer runs a SELECT under a known traceparent and asserts that the TabletServer's inbound "rpc yb.tserver.PgClientService.Perform" span lands in that trace as a child of the ysql backend's outbound span, with the expected rpc.service/rpc.method attributes. The new WaitForRemoteChildSpan collector helper does the caller/callee pairing check.

2026-08-18

[#33092] DocDB: Honor the requested read time in dump_tablet_data Summary: **Problem** dump_tablet_data scanned at the requested read_ht without checking whether the replica had caught up to it. A replica that was behind returned only the rows it had applied and presented them as the state at read_ht. A consistency check comparing XOR hashes across replicas then saw divergence that did not exist, and had no way to tell that apart from real corruption. **Fix** The tablet now waits for safe time to reach an explicit read_ht before scanning. If it does not get there in time, the scan fails with a retryable READ_TIME_NOT_REACHED instead of answering with partial data. The message reports the requested time, the replica's safe time, the gap, and how long it waited, so a caller can tell a replica that is merely behind from one that has diverged. Requests that carry no read_ht read at their own safe time, exactly as before. **Bounding the wait** The wait is bounded by the request's max_wait_ms. When unset, the server falls back to the dump_tablet_data_max_read_time_wait_ms gflag, default 5s. Zero fails immediately. The wait also stops dump_tablet_data_deadline_margin_ms, default 1s, before the RPC deadline. Without that margin the server gave up at the same instant the client stopped listening, so the caller saw a generic RPC timeout rather than READ_TIME_NOT_REACHED. The margin is what makes the real error observable. yb-ts-cli dump_tablet_data and yb-admin get_table_hash expose the bound as --read_time_wait_ms, default 5000, 0 to fail fast. The CLIs and the server share one constant for that default so they cannot drift apart. **Who is affected** yb-admin get_table_hash is the only pre-existing caller that reaches the new path, because it always sends an explicit read_ht. It now waits briefly instead of accepting a partially applied answer. Every other caller sends no read_ht and never waits. **Upgrade/Rollback safety** Both proto changes are additive, and these files are proto2. max_wait_ms is a new optional field in DumpTabletDataRequestPB. READ_TIME_NOT_REACHED is a new value in the existing TabletServerErrorPB.Code enum. Both are wire-compatible in either direction. Old server, new client: the server ignores max_wait_ms and behaves as it does today. The caller does not get the fix, and nothing breaks. New server, old client: the client sends no max_wait_ms, so the server's gflag applies. An old client asking for a read time its replica has not reached now gets an error instead of partial data, which is the fix working as intended. An old client that receives READ_TIME_NOT_REACHED keeps the unknown enum value in unknown fields and reads the default code, but the status message still carries the full explanation, so the failure stays legible. No AutoFlag is needed. This changes only the request and response of a diagnostic RPC. Nothing on disk changes. **Rejecting a read time that cannot be legitimate** A read time far ahead of this server's clock is a caller mistake, not lag: a units typo, or a clock that is not cluster-synced. Waiting for it burns the whole timeout and then reports lag, and a script that retries on a retryable error retries forever. So a read_ht more than dump_tablet_data_max_read_time_ahead_ms, default 60s, ahead of this server's clock fails immediately with InvalidArgument, which is not retryable. A read_ht closer than that is treated as lag and takes the normal bounded wait, returning READ_TIME_NOT_REACHED if it never arrives. The threshold is absolute, so which of the two errors a caller gets does not depend on its --timeout_ms or on its max_wait_ms. Test Plan: Twelve new cases: ten in YBTsCliDumpReadTimeTest and two in AdminCliGetTableHashReadTimeTest. They cover the caught-up fast path, fail-fast, waits bounded by the request and by the gflag, clamping to the RPC deadline, the unset-read_ht path, the structured error contract, and both sides of the far-future threshold: a read time past it is InvalidArgument, one under it waits. ./yb_build.sh --cxx-test tools_yb-ts-cli-test --gtest_filter='YBTsCliTest.DumpTabletData*' ./yb_build.sh --cxx-test tools_yb-admin-test --gtest_filter='AdminCliGetTableHashReadTimeTest.*' DumpTabletDataReadTimeTooFarInFuture sets the threshold to 10s and sweeps four combinations of max_wait_ms and --timeout_ms, asserting InvalidArgument in all four. That is what pins the classification to the threshold rather than to the caller's timeout. DumpTabletDataReadTimeWaitClampedToDeadline asserts READ_TIME_NOT_REACHED through both the CLI message and the RPC error code, and observes "waited 4.001s" under a 5s timeout. That assertion fails without the deadline margin. Regression check on the pre-existing callers, 14 cases, all passing: pg_libpq-test DumpTabletData and TestGetTableXorHash, four async_writes-test cases, the seven yb-admin get_table_hash cases, and YBTsCliTest.TestDumpTabletData. All passing in release; the new cases also pass under ASAN. Reviewers: hideaki.kimura Reviewed By: hideaki.kimura Subscribers: ybase Differential Revision: https://phorge.dev.yugabyte.com/D56620

[#30831] CDC: Align the WAL sort window for GetChanges across peers to prevent data loss Summary: ##### Code changes summary There exists a possible case where CDC can miss out on sending some records when the leadership of a tablet changes. This issue is hit when CDC is streaming WAL records which have been applied out of order and the tablet's leadership changes during this. Two things to remember: # When streaming Ops which have been applied out of order, CDC does not move its checkpoint forward. Instead the `wal_segment_index` field is used to indicate how many out-of-order records beyond the checkpoint (from_op_id) have been streamed from the sorted WAL Ops list. The subsequent call skips over these many records from the start of the sorted array of WAL Ops. # Another thing to note here is that each peer of the tablet has same committed WAL Ops but the segment boundaries can differ on each replica. These two factors combined result into a CDC data loss. Following example illustrates how this CDC data loss manifests: ``` GetChanges 1 was served by peer "a". The following WAL Op Ids were read Before sorting: 1.1, 1.2, 1.3, 1.4, 1.5, 1.6, 1.7, 1.8, 1.9, 1.10 After sorting : 1.1, 1.2, 1.3, 1.6, 1.9, 1.8, 1.5, 1.4, 1.7, 1.10 CDC reads one segment of WAL at a time. Sorting is done by CDC according to the commit time. Assume that GetChanges response size limit reached after decoding 1.8. Response: {1.1, 1.2, 1.3, 1.6, 1.9, 1.8}; checkpoint = 1.3, wal_segment_index = 3 ------------------------------------------------------------------------------------------ GetChanes 2 was served by peer "b". But its wal segment was rolled over earlier: Before sorting : 1.1, 1.2, 1.3, 1.4, 1.5, 1.6, 1.7 After sorting : 1.1, 1.2, 1.3, 1.6, 1.5, 1.4, 1.7 GetChanges read: 1.6, 1.5, 1.4, 1.7 --> Because checkpoint was 1.3 Since wal_segment_index was 3, we skipped the first three wal ops read and directly processed index 1.7. As a result, ops with index 1.4 and 1.5 were never streamed. ``` Overall, following conditions are needed to hit this issue: # CDC should be lagging. # The leadership of a tablet should change between two successive GetChanges calls. # The batch of records being streamed by CDC in the first GetChanges call should have out-of-order apply operations. # Some of these out-of-order ops lie in a different WAL segment, in one of the peers. This peer now becomes the new leader. # Next GetChanges call to this new leader will miss out on sending data. Now to fix this issue, a new field called `max_index_in_sort_window` in the GetChangesRequestPB and in GetChangesResponsePB. A GetChanges call populates this in the response with the max index it read from the WAL. We ensure that max_index_in_sort_window never moves backward. The subsequent GetChanges call receives this value in its request. It will read the WAL such that maximum index read from the WAL is >= `max_index_in_sort_window` value received in the request. If for any reason (ex: shutdown / WAL read failure etc) we are not able to read the WAL till max_index_in_sort_window, we do not ship any records in this GetChanges call as we cannot be sure of its correctness. The safe time, checkpoint and max_index_in_sort_window are kept unchanged in this case. **The Invariant**: In steady state (no restarts), if GetChanges on peer "A" of a tablet reads WAL upto point x, then the subsequent GetChanges calls on any peer of the same tablet (not necessarily A), will read WAL at least upto the point x. In the example above, GetChanges 1 would send max_index_in_sort_window = 10. GetChanges 2 would now read an additional segment to get all the ops till index 10. The WAL read after sorting would become {6, 9, 8, 5, 4, 7, 10}. We would correctly skip over {6, 9, 8} since wal_segment_index is 3 and start streaming from 5. This way no data would be missed by CDC. ##### Considerations for connector No changes required in the logical replication connector as the forwarding of max_index_in_sort_window and wal_segment_index from previous response to the next request is done by the virtual WAL. However, in gRPC model this work is owned by the connector. Hence corresponding change would be needed there. ##### Upgrade / Rollback safety This diff introduces an optional field `optional int64 max_index_in_sort_window` (default 0) in GetChangesRequestPB and in GetChangesResponsePB. Since it is optional, if the incoming request does not set max_index_in_sort_window, then we will proceed with the value 0, which is as good as disabling this mechanism. Hence the change is upgrade and rollback safe. ##### Considerations for colocated tables NA Test Plan: Existing CDC unit tests and stress tests. An exact UT to deterministically replicate the above mentioned CDC data loss failure could not be written since it needs reliable out of order applies. Reviewers: skumar, asrinivasan Reviewed By: asrinivasan Subscribers: kannan, ycdcxcluster Tags: #jenkins-ready Differential Revision: https://phorge.dev.yugabyte.com/D56990

2026-08-17

dist-trace: propagate trace context across the RPC boundary Carry a distributed trace from an RPC caller to its callee: define the wire type, write it into the request header, decode it on the far side, and open a server span parented under the caller's client span. All of it lands together because none of the pieces is observable on its own -- a wire field nobody reads, or a reader with nothing on the wire, cannot be tested. Wire type (rpc_header.proto): - TraceContextPB (W3C trace-context) and RequestHeader.trace_context. This is the type shared by the RPC-header and shared-memory transports; the latter starts using it in a later commit. Writer (outbound_call.cc): - SetRequestParam serializes the active client span's SpanContext into the trace_context submessage while it is sizing and writing the header, so the header is still written in a single pass. ~30 bytes on the wire, and only when a trace is active; an older peer skips the unknown field. Reader (serialization.{cc,h}): - ParseTraceContext(Slice) decodes the RequestHeader.trace_context wire field; ToSpanContext(TraceContextPB) is the shared-memory equivalent; both go through BuildSpanContext and fail on a zero/invalid context. - ParseHeader captures RequestHeader.trace_context into ParsedRequestHeader. Server / inbound span (yb_rpc.{cc,h}): - YBInboundCall::ParseFrom parses the header trace_context into parent_span_context_ (best-effort: a bad context is logged, never fails the RPC). - CreateServerSpan starts a StartServerSpanWithScope child of that parent (remote) or of an explicitly passed context (local); RespondSuccess/Failure set status and end it; DropServerSpanScope releases the scope on the handler thread while the span ends later. Client local-call handoff (local_call.{cc,h}, rpc_context.cc): - LocalOutboundCall tags rpc.local_call and drops its client-span scope in the ctor (a local call never hops threads to drop it later). - otel_span_context() exposes the outbound span's context so the local inbound span can parent under it (no wire header exists locally). - RpcContext creates the server span: from the wire header for remote calls, from the outbound call's context for local calls. Process init (tserver/db_server_base.cc): - DbServerBase is the shared base of Master and TabletServer, so DbServerBase:: Init is the single place that initializes the process-wide tracer (service name = server name, node id = permanent uuid), with Shutdown tearing it down. Gated by IsDistTraceEnabled, so it is a no-op when tracing is off. Without this the server spans above are never exported, which is why the 10 lines ship here rather than as a commit of their own. Test: TestRpcSpanReachesTabletServer runs a SELECT under a known traceparent and asserts that the TabletServer's inbound "rpc yb.tserver.PgClientService.Perform" span lands in that trace as a child of the ysql backend's outbound span, with the expected rpc.service/rpc.method attributes. The new WaitForRemoteChildSpan collector helper does the caller/callee pairing check.

[BACKPORT 2026.1][#32679] CDC: Decouple wal/intent and history barriers movement Summary: #### Backport summary No merge conflicts. #### Original summary ##### Code changes summary **Problem** Let us consider 2 scenarios pertaining to `CDCMasterBgTask`: 1. Stream creation of first logical replication stream which polls sys_catalog: The stream creation workflow first populates the sys_catalog_tablet-stream entry in cdc_state table with invalid checkpoint. In later part of workflow it overwrites it to a valid value. However if CDCMasterBgTask runs in between, then as part of revising the barriers, it will request for release of intent and wal barriers (since as per its computation it didn't see a valid index and checkpoint for sys_catalog wal and intent respectively). - The conclusion from this point is that even if `CDCMasterBgTask` from its computation sees that WAL and Intent barriers can be released, it shouldn't do so in such case. - This is the race being seen by test `CDCSDKConsumptionConsistentChangesTest.TestVWALConsumptionWithMultipleAlter` ([#32679](https://github.com/yugabyte/yugabyte-db/issues/32679)) where `CDCMasterBgTask` releases the intent barriers and on next GetChanges() call we see `CDCSDK Trying to fetch already GCed intents` error. 2. Only logical replication stream (which polls sys_catalog) gets deleted: In such a case, no other stream requires wal and intents of sys_catalog. So `CDCMasterBgTask` computes and finds that WAL and intent barriers can be released. So it do so. - Here, `CDCMasterBgTask` decision was valid. So, we currently don't have a proper invariant for `CDCMasterBgTask` which can define when to release the retention barriers of WAL and intents. **Fix** We make a strict invariant for `CDCMasterBgTask` wrt release of WAL and Intent barriers by: - selectively not refreshing WAL and intent barriers when `CDCMasterBgTask` computes the `cdc_sdk_op_id` to be OpId::Max(). - selectively releasing the stale retention barriers via `ResetStaleRetentionBarriersOp`. Selective here means that while refreshing/releasing barriers, it can be chosen which resource's barrier to refresh and which to not. Wrt to current use-cases, we refresh or release WAL and Intent barriers together and independent to History barrier. We can see that this approach solves both scenarios mentioned above. In both cases, `CDCMasterBgTask` won't refresh the WAL and Intent barriers (based on condition `computed cdc_sdk_op_id == OpId::Max()`). Thus once these resources' barriers go stale, `ResetStaleRetentionBarriersOp` will selectively release barriers only for such resources. Implementation-wise: - Introduced `CDCRetentionBarrierMoveSelector` which contains three bools, each corresponding to decision to move/refresh WAL, Intent and History barrier respectively. All 3 bools have default value as `true` (signifying to "move all" barriers defaultly). It is threaded through the entire barriers revision path. Thus each barrier is now guarded by its selector bit, so any subset can be moved. - When the sys_catalog checkpoint is OpId::Max(), `CDCMasterBgTask` moves only the history barrier (i.e `CDCRetentionBarrierMoveSelector({false, false, true})`), leaving WAL/Intents for `ResetStaleRetentionBarriersOp` to release once stale. - `ResetStaleRetentionBarriersOp` also uses `CDCRetentionBarrierMoveSelector` for selective barriers release. It divides the releasing of stale barriers in 2 groups: WAL+Intents & History. 1. For WAL+Intent barriers staleness check, it uses `cdc_min_replicated_index_refresh_time_` and for history barrier staleness, it used a newly introduced variable `cdc_sdk_safe_time_refresh_time_`. 2. It now also maintain the last time the op ran per group, via `wal_intent_barrier_last_reset_time_` & `history_barrier_last_reset_time_`. ##### Upgrade/Rollback safety The change adds 3 new optional bools in `UpdateCdcReplicatedIndexRequestPB` which are skip_moving_wal_retention_barrier`, `skip_moving_intents_retention_barrier`& `skip_moving_history_retention_barrier`. These are used by leader master's `CDCMasterBgTask` to relay the information about which all resources' barriers need to be refreshed on follower masters. Since during upgrade some follower masters may not have upgraded yet, thus running old code. The upgraded master leader sending refresh info for selective barriers (via these newly fields in proto) will be discarded by such followers since such fields are absent on its old code. Thus, such followers can move all resources' retention barriers. To mitigate this, `CDCMasterBgTask` only emits a selective (subset) request once the newly introduced **`cdcsdk_enable_selective_retention_barrier_revision`** auto-flag is promoted. ##### Considerations for colocated tables NA ##### Compatibility with logical and gRPC streams NA ##### Considerations for connector NA Original commit: 6ffde0e8323ad250383b41baf138d29ca5c7974f / D55740 Test Plan: ./yb_build.sh asan --clang21 --cxx-test cdcsdk_consumption_consistent_changes-test --gtest-filter CDCSDKConsumptionConsistentChangesTest.TestVWALConsumptionWithMultipleAlter -n 100 --stop-at-failure -- -p 4 Reviewers: sumukh.phalgaonkar, asrinivasan, skumar, sergei, ssharma Reviewed By: ssharma Subscribers: ycdcxcluster Tags: #jenkins-ready Differential Revision: https://phorge.dev.yugabyte.com/D56998

[#32679] CDC: Decouple wal/intent and history barriers movement Summary: ##### Code changes summary **Problem** Let us consider 2 scenarios pertaining to `CDCMasterBgTask`: 1. Stream creation of first logical replication stream which polls sys_catalog: The stream creation workflow first populates the sys_catalog_tablet-stream entry in cdc_state table with invalid checkpoint. In later part of workflow it overwrites it to a valid value. However if CDCMasterBgTask runs in between, then as part of revising the barriers, it will request for release of intent and wal barriers (since as per its computation it didn't see a valid index and checkpoint for sys_catalog wal and intent respectively). - The conclusion from this point is that even if `CDCMasterBgTask` from its computation sees that WAL and Intent barriers can be released, it shouldn't do so in such case. - This is the race being seen by test `CDCSDKConsumptionConsistentChangesTest.TestVWALConsumptionWithMultipleAlter` ([#32679](https://github.com/yugabyte/yugabyte-db/issues/32679)) where `CDCMasterBgTask` releases the intent barriers and on next GetChanges() call we see `CDCSDK Trying to fetch already GCed intents` error. 2. Only logical replication stream (which polls sys_catalog) gets deleted: In such a case, no other stream requires wal and intents of sys_catalog. So `CDCMasterBgTask` computes and finds that WAL and intent barriers can be released. So it do so. - Here, `CDCMasterBgTask` decision was valid. So, we currently don't have a proper invariant for `CDCMasterBgTask` which can define when to release the retention barriers of WAL and intents. **Fix** We make a strict invariant for `CDCMasterBgTask` wrt release of WAL and Intent barriers by: - selectively not refreshing WAL and intent barriers when `CDCMasterBgTask` computes the `cdc_sdk_op_id` to be OpId::Max(). - selectively releasing the stale retention barriers via `ResetStaleRetentionBarriersOp`. Selective here means that while refreshing/releasing barriers, it can be chosen which resource's barrier to refresh and which to not. Wrt to current use-cases, we refresh or release WAL and Intent barriers together and independent to History barrier. We can see that this approach solves both scenarios mentioned above. In both cases, `CDCMasterBgTask` won't refresh the WAL and Intent barriers (based on condition `computed cdc_sdk_op_id == OpId::Max()`). Thus once these resources' barriers go stale, `ResetStaleRetentionBarriersOp` will selectively release barriers only for such resources. Implementation-wise: - Introduced `CDCRetentionBarrierMoveSelector` which contains three bools, each corresponding to decision to move/refresh WAL, Intent and History barrier respectively. All 3 bools have default value as `true` (signifying to "move all" barriers defaultly). It is threaded through the entire barriers revision path. Thus each barrier is now guarded by its selector bit, so any subset can be moved. - When the sys_catalog checkpoint is OpId::Max(), `CDCMasterBgTask` moves only the history barrier (i.e `CDCRetentionBarrierMoveSelector({false, false, true})`), leaving WAL/Intents for `ResetStaleRetentionBarriersOp` to release once stale. - `ResetStaleRetentionBarriersOp` also uses `CDCRetentionBarrierMoveSelector` for selective barriers release. It divides the releasing of stale barriers in 2 groups: WAL+Intents & History. 1. For WAL+Intent barriers staleness check, it uses `cdc_min_replicated_index_refresh_time_` and for history barrier staleness, it used a newly introduced variable `cdc_sdk_safe_time_refresh_time_`. 2. It now also maintain the last time the op ran per group, via `wal_intent_barrier_last_reset_time_` & `history_barrier_last_reset_time_`. ##### Upgrade/Rollback safety The change adds 3 new optional bools in `UpdateCdcReplicatedIndexRequestPB` which are skip_moving_wal_retention_barrier`, `skip_moving_intents_retention_barrier`& `skip_moving_history_retention_barrier`. These are used by leader master's `CDCMasterBgTask` to relay the information about which all resources' barriers need to be refreshed on follower masters. Since during upgrade some follower masters may not have upgraded yet, thus running old code. The upgraded master leader sending refresh info for selective barriers (via these newly fields in proto) will be discarded by such followers since such fields are absent on its old code. Thus, such followers can move all resources' retention barriers. To mitigate this, `CDCMasterBgTask` only emits a selective (subset) request once the newly introduced **`cdcsdk_enable_selective_retention_barrier_revision`** auto-flag is promoted. ##### Considerations for colocated tables NA ##### Compatibility with logical and gRPC streams NA ##### Considerations for connector NA Test Plan: ./yb_build.sh asan --clang21 --cxx-test cdcsdk_consumption_consistent_changes-test --gtest-filter CDCSDKConsumptionConsistentChangesTest.TestVWALConsumptionWithMultipleAlter -n 100 --stop-at-failure -- -p 4 Reviewers: sumukh.phalgaonkar, asrinivasan, skumar, sergei Reviewed By: asrinivasan Subscribers: ycdcxcluster Tags: #jenkins-ready Differential Revision: https://phorge.dev.yugabyte.com/D55740

2026-08-14

[#33296] DocDB: Table locks: Fix CleanupExpiredLeaseEpochs logic at the master's object lock manager Summary: Expired-lease lock cleanup was writing the release against the wrong lease epoch in sys catalog. It stamped `max_lease_epoch_to_release + 1` on the request so tservers would fence out old acquires, but that same field is also the key used to delete locks from `lease_epochs`. The delete often became a no-op, so old locks stayed on disk after the in-memory release. This wasn't a problem prior to commit https://github.com/yugabyte/yugabyte-db/commit/c670b00d9da57a4af8b0be8850bf9b98391947da / D56278 as we were explicitly ignoring locks correspinding to expired ysql leases while exporting the object locks bootstrap payload. The commit changed the behavior for master bootstrap payload to load up all locks since it wanted to establish the semantic - Master's persistent lock state of active locks should process the release only after all tservers have released the lock. Despite the releases being relaunched, and released in the master's in-memory lock manager, the persisted state still remains (The space amplification problem existed prior to the commit). This could result in a false lock conflict on master failover when the new master tries replaying the expired locks as well as the active conflicting locks if any. This revision fixes the issue in the following manner - changing `lease_epoch` to reflect the actual epoch with which is the lock is associated to. - introduce `ignore_lease_epochs_before` to notify the tservers that they can drop any subsequent acquire requests from the source tserver with lease epoch lesser than `ignore_lease_epochs_before`. Note that this was the original intention for setting lease_epoch on the release request to `max_lease_epoch_to_release + 1`. The normal release operations don't populate `ignore_lease_epochs_before`. The tservers prefer `ignore_lease_epochs_before` when set, and fallback to use `lease_epoch` when tracking the max seen lease epoch for other tservers. **Upgrade/Downgrade safety** Table locking hasn't been shipped in any release yet, so there shouldn't be any implications. Test Plan: Jenkins ``` ./yb_build.sh --cxx-test object_lock-test --gtest_filter ObjectLockTest.ExpiredLeaseReleaseClearsPersistedLocks ``` Reviewers: amitanand, zdrudi Reviewed By: amitanand Subscribers: ybase Differential Revision: https://phorge.dev.yugabyte.com/D56930

2026-08-13

dist-trace: propagate trace context across the RPC boundary Carry a distributed trace from an RPC caller to its callee: define the wire type, write it into the request header, decode it on the far side, and open a server span parented under the caller's client span. All of it lands together because none of the pieces is observable on its own -- a wire field nobody reads, or a reader with nothing on the wire, cannot be tested. Wire type (rpc_header.proto): - TraceContextPB (W3C trace-context) and RequestHeader.trace_context. This is the type shared by the RPC-header and shared-memory transports; the latter starts using it in a later commit. Writer (outbound_call.cc): - SetRequestParam serializes the active client span's SpanContext into the trace_context submessage while it is sizing and writing the header, so the header is still written in a single pass. ~30 bytes on the wire, and only when a trace is active; an older peer skips the unknown field. Reader (serialization.{cc,h}): - ParseTraceContext(Slice) decodes the RequestHeader.trace_context wire field; ToSpanContext(TraceContextPB) is the shared-memory equivalent; both go through BuildSpanContext and fail on a zero/invalid context. - ParseHeader captures RequestHeader.trace_context into ParsedRequestHeader. Server / inbound span (yb_rpc.{cc,h}): - YBInboundCall::ParseFrom parses the header trace_context into parent_span_context_ (best-effort: a bad context is logged, never fails the RPC). - CreateServerSpan starts a StartServerSpanWithScope child of that parent (remote) or of an explicitly passed context (local); RespondSuccess/Failure set status and end it; DropServerSpanScope releases the scope on the handler thread while the span ends later. Client local-call handoff (local_call.{cc,h}, rpc_context.cc): - LocalOutboundCall tags rpc.local_call and drops its client-span scope in the ctor (a local call never hops threads to drop it later). - otel_span_context() exposes the outbound span's context so the local inbound span can parent under it (no wire header exists locally). - RpcContext creates the server span: from the wire header for remote calls, from the outbound call's context for local calls. Process init (tserver/db_server_base.cc): - DbServerBase is the shared base of Master and TabletServer, so DbServerBase:: Init is the single place that initializes the process-wide tracer (service name = server name, node id = permanent uuid), with Shutdown tearing it down. Gated by IsDistTraceEnabled, so it is a no-op when tracing is off. Without this the server spans above are never exported, which is why the 10 lines ship here rather than as a commit of their own. Test: TestRpcSpanReachesTabletServer runs a SELECT under a known traceparent and asserts that the TabletServer's inbound "rpc yb.tserver.PgClientService.Perform" span lands in that trace as a child of the ysql backend's outbound span, with the expected rpc.service/rpc.method attributes. The new WaitForRemoteChildSpan collector helper does the caller/callee pairing check.

2026-08-12

[#31589] DocDB: Serve YSQL lease RPCs on the master high-priority thread pool Summary: YSQL lease heartbeats from tservers to the master were handled by the regular-priority RPC thread pool via the MasterDdl service. Under an overload of regular-priority master RPCs (e.g. catalog cache reads or heartbeats or GetTableSchema or GetTabletLocations) the lease refresh could be backed up, and a lease failure kills all PG sessions on that tserver. This moves RefreshYsqlLease and RelinquishYsqlLease into a dedicated MasterYsqlLease service and registers it on the master's high-priority RPC thread pool (rpc::ServicePriority::kHigh), isolating lease refreshes from regular master RPC load. (Consensus is the other user of the high-priority pool.) **Upgrade/Rollback safety:** The new service keeps custom_service_name = "yb.master.MasterService" and the same method names, so the on-the-wire remote method is identical to when these RPCs lived in MasterDdl. This makes the change compatible with any master/tserver upgrade order: an old tserver's MasterDdlProxy and a new tserver's MasterYsqlLeaseProxy produce byte-identical requests, resolved by whichever service the master has registered. The only visible change is the RPC's metrics group name (yb.master.MasterDdl -> yb.master.MasterYsqlLease). Test Plan: Existing tests exercise the lease RPCs through the new service/proxy: ./yb_build.sh fastdebug --clang21 --cxx-test object_lock-test ./yb_build.sh fastdebug --clang21 --cxx-test master-test Reviewers: bkolagani, zdrudi, #db-approvers Reviewed By: zdrudi, #db-approvers Subscribers: ybase Differential Revision: https://phorge.dev.yugabyte.com/D56573

2026-08-10

[BACKPORT 2026.1][#28482] YSQL: implement new-relation fastpath write optimization (skip intents) Summary: This diff introduces the *new-relation fastpath write* optimization (informally "skip intents"). In a standard YB distributed transaction, all writes first go through the intents DB and are only moved to the regular DB on commit. This diff allows writes to a relation that was newly created (or had its physical storage swapped) in the same transaction to bypass the intents DB and write directly to the regular DB, and lets reads on those relations skip the intents merge as well. The result is significantly less I/O for CTAS, table rewrites, MV refresh, and bulk loads in DDL-only transactions. **Core decision logic — `YbCanSkipIntents(Relation rel, bool is_write)` in `pg_yb_utils.c`** Returns true (i.e., the optimization is safe) when all of the following hold: 1. The relation was created in the current transaction (`rd_createSubid != InvalidSubTransactionId`) or had its storage swapped in the current transaction (`rd_newRelfilenodeSubid != InvalidSubTransactionId`). The second case extends the optimization to `ALTER TABLE` rewrites, `REINDEX`, and `REFRESH MATERIALIZED VIEW` (non-concurrent). 2. The relation is not a system catalog (`rd_id >= FirstNormalObjectId`), not temporary, and not colocated. 3. The optimization has not been disabled earlier in this transaction. 4. The transaction has no non-read-committed *named* savepoint above the current sub-txn (`ROLLBACK TO SAVEPOINT` requires real intents). 5. The statement is top-level *unless* the in-txn-blocks preview GUC is on and the isolation level is READ COMMITTED — in which case non-top-level statements are allowed only when DDL transaction blocks (`ysql_yb_ddl_transaction_block_enabled`) are also enabled. For reads, an additional Halloween-problem guard runs in `YbMaybeDisableSkipIntentsForCurrentTxn()`: if a same-txn-created relation is read inside a function/trigger, from SPI, from a non-SELECT op, or from a SELECT with a modifying CTE, we permanently disable the optimization for the rest of the transaction so the next write does not corrupt isolation (e.g., `INSERT INTO t SELECT id+100 FROM t`). **GUCs** Two PGC_SUSET booleans (both refuse to change inside a transaction block, and both make the connection sticky in YSQL Conn-Mgr when set off-default): - `yb_enable_new_relation_fastpath_write` (default `true`): master switch. Top-level statements only. - `yb_enable_new_relation_fastpath_write_in_txn_blocks` (default `false`, preview-flag gated): allows fastpath inside `BEGIN`/transaction blocks. Only effective under READ COMMITTED + `ysql_yb_ddl_transaction_block_enabled`. **xCluster / CDC interaction** xCluster is fully compatible with the skip-intents optimization, as its table-level streams correctly replicate the direct fastpath writes. However, logical replication and legacy CDCSDK require intents, so we must not use fastpath when the database participates in CDCSDK. YbMaybeDisableSkipIntentsForCDCSDK() is called from the table-creation path and: 1. Checks whether `pg_publication` has any rows (publication-based CDCSDK). The result is session-cached and invalidated through a `PUBLICATIONOID` syscache callback so the scan only happens once per publication change. 2. If `ysql_cdcsdk_enable_old_namespace_streams` is true, falls back to a master RPC for old-style namespace-level streams (no slot). Failures are conservatively treated as "namespace is part of CDCSDK". The fallback is plumbed through a new `IsNamespacePartOfCDCSDK` RPC on both the master replication service and the PG client service: - New master RPC `master.IsNamespacePartOfCDCSDK(namespace_id)` implemented in `xrepl_catalog_manager.cc` by scanning `cdc_stream_map_`. - New tserver RPC `pg_client.IsNamespacePartOfCDCSDK(database_oid)` implemented in `pg_client_service.cc`, which translates oid → namespace id and forwards to master. - New ASH wait state `kIsNamespacePartOfCDCSDK` and `YBClient::IsNamespacePartOfCDCSDK` helper. **Wire protocol changes** - `pgsql_protocol.proto`: adds optional `bool skip_intents_write` to `PgsqlWriteRequestPB` and optional `bool skip_intents_read` to `PgsqlReadRequestPB`. - `pg_client.proto`: adds the new `IsNamespacePartOfCDCSDK` RPC + request/response messages. - `master_replication.proto`: adds the new `IsNamespacePartOfCDCSDK` RPC + request/response messages. All wire changes are additive optional fields / new RPCs and only flow on PG <-> tserver and tserver <-> master paths, so they are upgrade-safe. **YBClient batcher / async RPC** `Batcher` records whether all ops in a batch carry the skip-intents flag (writes and reads both). When set: - `Batcher::transaction()` returns `nullptr` so the op bypasses transaction metadata propagation and goes straight to the tablet as a non-transactional write/read. - `force_consistent_read` is set unconditionally, and any explicit `read_time` is cleared in `AsyncRpc::ProcessResponseFromTserver` to avoid "Restart read required" errors. - A new tablet-server counter `skip_intents_writes` is incremented per fastpath write batch. **PG executor / planner integration** - All `YbNewInsert/Update/Delete/InsertBlock` callers now pass `YbCanSkipIntentsWrite(rel)`; `YbNewSelect/Sample` pass `YbCanSkipIntentsRead(rel)`. The flag is plumbed through the entire pggate stack (`PgInsert/Update/Delete/Select/SelectIndex/Sample/SamplePicker`). - `pg_session.cc` flushes the buffered op queue when the skip-intents mode of the next op differs from the previous one (since fastpath and normal ops can't share a batch). - `nodeLockRows.c`: when fastpath is in effect we skip `YBCLockTuple` entirely — there's nothing in intents to lock. - `postgres.c::yb_is_retry_possible`: query-layer retry is disabled once any fastpath write has been issued in this transaction (data is already in regular DB; replaying would duplicate). - `xact.c`: `YbEnableSkipIntentsForNewTransaction()` resets per-txn state at every transaction start. - `Assert(!YbCanSkipIntentsWrite(rel))` was added to the catalog DELETE path in `ybModifyTable.c` as a safety net — catalogs must never use fastpath. - `YbGetSPIStackDepth()` (new in `spi.c`) is used to detect "we're inside a function/procedure". **Savepoint helper split** `YBTransactionContainsNonReadCommittedSavepoint()` (used by the skip-intents check) is now a separate helper that returns true for any non-RC subtransaction (named or anonymous). The original named-savepoint semantics are preserved under a new name, `YBTransactionContainsNonReadCommittedNamedSavepoint()`, which is what the existing "interleaving SAVEPOINT & DDL" check now uses. **API rename** `YbGetTableDistribution(Oid)` → `YbGetTableDistributionById(Oid)`; a new `YbGetTableDistribution(Relation)` overload is the preferred form when the caller already holds a `Relation` (used by the fastpath check itself). Callers in `costsize.c`, `allpaths.c`, and `pg_hint_plan/core.c` updated. **Upgrade / rollback safety** The src/yb/tserver/pg_client.proto and src/yb/common/pgsql_protocol.proto changes are only used in PG -> tserver communication which is upgrade safe. All new wire fields are additive optional fields, and the new RPCs follow the standard add-and-rolling-upgrade pattern: during a rolling upgrade, `IsNamespacePartOfCDCSDK` may not yet exist on the master — `YbMaybeDisableSkipIntentsForCDCSDK()` treats the RPC failure conservatively and disables the optimization for that transaction. The optimization is also gated by GUCs that default to off (in-txn-blocks variant) or to a behavior equivalent to today (top-level only path is on by default but produces identical visible behavior). Original commit: 48fc88e231 / D48866 NOTE: There were merge conflicts that were resolved. Test Plan: - New `pg_skip_intents_metrics-test.cc` exercises the metric across CTAS, multiple ALTER TABLE rewrite shapes (type change, ADD/DROP PRIMARY KEY, volatile defaults, multi-index tables), `MATERIALIZED VIEW` refresh (concurrent vs non-concurrent), publication on/off, isolation-level matrix (RC / RR / SERIALIZABLE), the same-txn-created-read Halloween guards, and PITR over fastpath-written data. - New `xcluster/xcluster_ysql_skip_intents-test.cc` validates xCluster + automatic DDL replication when both clusters have skip-intents enabled (CTAS, ALTER rewrite, chained CTAS). - `pg_ddl_transaction-test::PgDdlTransactionTest.TestNoSkipIntentsWriteOnSavepoint` covers the named-savepoint disable path end-to-end. - `TestDdlSavepoints` and `TestDdlTransactionBlocks` randomly toggle the in-txn-block GUC to broaden coverage. `TestDdlTransactionBlocks` also runs the `yb_ddl_txn_block_schedule` regress schedule, which includes the scaled-up `yb.orig.ddl_txn_visibility` regress test that exercises the Halloween-problem patterns. Test commands: ``` ./yb_build.sh release --cxx-test pg_skip_intents_metrics-test ./yb_build.sh release --cxx-test xcluster_ysql_skip_intents-test ./yb_build.sh release --cxx-test pg_ddl_transaction-test --gtest_filter PgDdlTransactionTest.TestNoSkipIntentsWriteOnSavepoint ./yb_build.sh release --java-test 'org.yb.pgsql.TestDdlSavepoints' ./yb_build.sh release --java-test 'org.yb.pgsql.TestDdlTransactionBlocks' ``` TPC-C perf run: no regression observed with the GUC on or off. Reviewers: sanketh, pjain, sergei, timur, bkolagani, xCluster, patnaik.balivada, jhe, hsunder Reviewed By: pjain Subscribers: yql, ybase, hsunder, jason, dmitry, smishra Differential Revision: https://phorge.dev.yugabyte.com/D56741

[#31758] YSQL: Replace WaitForYsqlBackends with WaitForLockers on CREATE INDEX CONCURRENTLY path Summary: **Problem** Concurrent CREATE INDEX used `WaitForYsqlBackendsCatalogVersion` between phases which waits for any backend still operating on older catalog versions to advance the version. And since this happens only on transaction boundaries, the wait can be long in some cases and the create index cmd might timeout and fail. The reason for the above wait seems to be for the backends to observe the relevant index state/permission changes. With object locking, we can do better since acquisition of a lock on the base/index table would lead to replay of invalidation messages, and hence we would see the latest state/permissions of the base/index table (similar to what PG does). There is one other problem where create index can deadlock with other DDL when object locking is enabled (with the `WaitForYsqlBackendsCatalogVersion` route). CREATE INDEX holds a session level SHARE lock on the base relation across the phases of index creation, and waits for other backends to catch up. Meanwhile the other backends could be stuck waiting to acquire a conflicting lockmode on the base relation (for instance, a truncate requires ACCESS EXCLUSIVE which conflicts with SHARE), and hence it could lead to a deadlock and is resolved only when either times out. **Solution** 1. When object locking is enabled, `CREATE INDEX CONCURRENTLY` now waits for conflicting lockers as opposed to waiting for all backends at the following places a. before index permission is switched to indisready=true (phase 2), and b. before backfill starts (phase 3) 2. When object locking is enabled, `CREATE INDEX CONCURRENTLY` now has an additional 4th phase which does a timed wait for all backends with older catalog versions. This is similar to PG’s `WaitForOlderSnapshots` during index creation. Note that switching of index permission to `indisvalid=true` is now pushed to this phase. 3. To prevent an index from serving a read whose snapshot predates it, backfill safe time is persisted as `birth_time` on the index table in docdb. This `birth_time` is set async by master post backfilling the index as part of the task to update index table to allow compactions (resets the `retain_delete_markers`). On index read, docdb rejects reads with `read_time` < `birth_time` (SnapshotTooOld) in case `birth_time` exists and is non-zero. In case the `birth_time` hasn't been set, we return a retry-able error so that the read is retried from the query layer (host tserver/pg_client_session node that sent the read op). The motivation of the additional timed wait being introduced in the 4th phase is to give older backends more time to finish as they might operate on the table/index and error out due to checks being introduced in step 3 above. Without object locking, there is no need for such a wait as we do an exhaustive wait for backends in phases 2 & 3 of index creation. Note: In case of follower reads, the pg catalog is read as of now. so the reads ops with read time in the past can end up hitting the index (which might not have all data as of the read time), and are rightly failed with the current fix. **Note for xCluster automatic target**: For current non legacy case, the backfill writes are streamed from the source as opposed to doing a local backfill. On ddl commit at the source, the commit time is sent to the target, at which point the index is made valid. [[ https://github.com/yugabyte/yugabyte-db/issues/33155 | GH #33155 ]] tracks the requirement to send the index birth time from the source to target. Until then, the above WaitForLockers optimization on the create index path is disabled for xCluster automatic target, so create index would wait for all lagging backends. On the target, the index birth time would be 0, and hence reads wouldn't be rejected. Existing tests `XClusterDDLReplicationTest.IndexCreationImmediatelyAfterInsert`, `XClusterDDLReplicationIndexBackfillTest.TargetWaitsForReplicatedBackfill`, `XClusterDDLReplicationIndexBackfillTest.ReplicatedFromSource`, and `XClusterDDLReplicationIndexBackfillTest.LegacyLocalBackfill` exercise this path and assert that the reads to the index don't fail (`BackfillTable::read_time_for_backfill_` is `kInvalidHybridTimeValue` on target). **Upgrade / Rollback safety** - Index birth time is checked for during ysql index reads at docdb only if birth time is non-zero. So the check isn't executed for older indexes created prior to the upgrade as the birth time doesn't exist both at the master and docdb. - Indexes created post upgrade would have birth time both at docdb and master - Indexes created on upgraded master and mixed mode tservers should never see the need to trigger the birth time check as object locking feature would be off in mixed mode, and create index would instead take the waitforbackends path. Though master would have the birth time for this index, docdb might or might not have it depending on where the tablet RBS'ed from. And if the tservers were on older version while creating the index, none of them would have the birth time after the upgrade as well. This isn't a problem since the check at the read layer is only triggered when birth time is non zero. Added a kill switch to disable the check if it is creating any issues - `TEST_disable_index_birth_time_check` can be set to true to disable the check at the read layer. Note that the test flag wouldn't disable the above optimization of taking waitforlockers path (+ one timed waitforbackends) as opposed to long/unbounded waitforbackends path. Test Plan: Jenkins ``` ./yb_build.sh --cxx-test pg_object_locks-test --gtest_filter PgObjectLocksTest.CreateIndexDoesntDeadlockWithTruncate ``` Prior to the fix, create index can deadlock with truncate as truncate might be waiting on the session level lock held by create index, and create index might be waiting for truncate's backend to observe the latest catalog version. ``` ./yb_build.sh --cxx-test pg_object_locks-test --gtest_filter *WaitForBackendsBeforeMarkingIndexValid* ``` Test asserts that reads to index with read times < index birth time fail with read restart. test also asserts that the read is retried when the birth time hasn't been persisted yet (decided by whether the tablet still has retain delete markers set i.e. garbage collection disabled). Reviewers: pjain, sanketh, patnaik.balivada, amitanand, myang, jason, jhe Reviewed By: patnaik.balivada Subscribers: svc_phabricator, jason, myang, yql, ybase Differential Revision: https://phorge.dev.yugabyte.com/D53646

2026-08-07

dist-trace: propagate trace context across the RPC boundary Carry a distributed trace from an RPC caller to its callee: define the wire type, write it into the request header, decode it on the far side, and open a server span parented under the caller's client span. All of it lands together because none of the pieces is observable on its own -- a wire field nobody reads, or a reader with nothing on the wire, cannot be tested. Wire type (rpc_header.proto): - TraceContextPB (W3C trace-context) and RequestHeader.trace_context. This is the type shared by the RPC-header and shared-memory transports; the latter starts using it in a later commit. Writer (outbound_call.cc): - SetRequestParam serializes the active client span's SpanContext into the trace_context submessage while it is sizing and writing the header, so the header is still written in a single pass. ~30 bytes on the wire, and only when a trace is active; an older peer skips the unknown field. Reader (serialization.{cc,h}): - ParseTraceContext(Slice) decodes the RequestHeader.trace_context wire field; ToSpanContext(TraceContextPB) is the shared-memory equivalent; both go through BuildSpanContext and fail on a zero/invalid context. - ParseHeader captures RequestHeader.trace_context into ParsedRequestHeader. Server / inbound span (yb_rpc.{cc,h}): - YBInboundCall::ParseFrom parses the header trace_context into parent_span_context_ (best-effort: a bad context is logged, never fails the RPC). - CreateServerSpan starts a StartServerSpanWithScope child of that parent (remote) or of an explicitly passed context (local); RespondSuccess/Failure set status and end it; DropServerSpanScope releases the scope on the handler thread while the span ends later. Client local-call handoff (local_call.{cc,h}, rpc_context.cc): - LocalOutboundCall tags rpc.local_call and drops its client-span scope in the ctor (a local call never hops threads to drop it later). - otel_span_context() exposes the outbound span's context so the local inbound span can parent under it (no wire header exists locally). - RpcContext creates the server span: from the wire header for remote calls, from the outbound call's context for local calls. Process init (tserver/db_server_base.cc): - DbServerBase is the shared base of Master and TabletServer, so DbServerBase:: Init is the single place that initializes the process-wide tracer (service name = server name, node id = permanent uuid), with Shutdown tearing it down. Gated by IsDistTraceEnabled, so it is a no-op when tracing is off. Without this the server spans above are never exported, which is why the 10 lines ship here rather than as a commit of their own. Test: TestRpcSpanReachesTabletServer runs a SELECT under a known traceparent and asserts that the TabletServer's inbound "rpc yb.tserver.PgClientService.Perform" span lands in that trace as a child of the ysql backend's outbound span, with the expected rpc.service/rpc.method attributes. The new WaitForRemoteChildSpan collector helper does the caller/callee pairing check.

[BACKPORT 2025.2][#32332] DocDB: Add new ASH events for Write Pipelining Summary: Adding in two new WaitStateCodes: - `kRaft_WaitingForPipelinedReplication` waits on the tserver in `TabletPeer::RegisterAsyncWriteCompletion` until the op is Raft-committed. - `kYBClient_WaitingForPipelinedWrites` waits on the client for the pipelined writes to drain before the transaction can commit. Making a few changes to WaitForAsyncWrite for `kRaft_WaitingForPipelinedReplication`: - Adding `yb.rpc.send_metadata` to the RPC - WaitForAsyncWriteRpc copies the metadata of the issuing statement's WaitStateInfo. Since this RPC outlives the issuing statement, we need to do this to keep this state around. For kYBClient_WaitingForPipelinedWrites`: `YBTransaction::Impl::Commit` defers the commit into `async_write_commit_waiter_` while writes are still in flight, and that callback runs on the thread that completes the last write, which carries no wait state. Capturing the current wait state at deferral time and adopting it in the waiter, otherwise the COMMITTED record and the APPLYING fan-out it triggers are left unattributed. **Upgrade/Rollback safety:** Only adding `yb.rpc.send_metadata` to `WaitForAsyncWrite`. This just makes the client populate the existing optional metadata field in the request header. If omitted then we have current behaviour. Original commit: d65c8a9f050c985d0e462847ae3825c18ae2328f / D56271 Test Plan: Added `WritePipeliningWaitAttributedToStatement` and a new `AshTestVerifyOccurrence` test for the new wait states. Without the fix, the test shows query_id 0 samples, with the fix all samples have proper ids. Reviewers: asaha Reviewed By: asaha Subscribers: yql, ybase Differential Revision: https://phorge.dev.yugabyte.com/D56662

[BACKPORT 2026.1][#32332] DocDB: Add new ASH events for Write Pipelining Summary: Adding in two new WaitStateCodes: - `kRaft_WaitingForPipelinedReplication` waits on the tserver in `TabletPeer::RegisterAsyncWriteCompletion` until the op is Raft-committed. - `kYBClient_WaitingForPipelinedWrites` waits on the client for the pipelined writes to drain before the transaction can commit. Making a few changes to WaitForAsyncWrite for `kRaft_WaitingForPipelinedReplication`: - Adding `yb.rpc.send_metadata` to the RPC - WaitForAsyncWriteRpc copies the metadata of the issuing statement's WaitStateInfo. Since this RPC outlives the issuing statement, we need to do this to keep this state around. For kYBClient_WaitingForPipelinedWrites`: `YBTransaction::Impl::Commit` defers the commit into `async_write_commit_waiter_` while writes are still in flight, and that callback runs on the thread that completes the last write, which carries no wait state. Capturing the current wait state at deferral time and adopting it in the waiter, otherwise the COMMITTED record and the APPLYING fan-out it triggers are left unattributed. **Upgrade/Rollback safety:** Only adding `yb.rpc.send_metadata` to `WaitForAsyncWrite`. This just makes the client populate the existing optional metadata field in the request header. If omitted then we have current behaviour. Original commit: d65c8a9f050c985d0e462847ae3825c18ae2328f / D56271 Test Plan: Added `WritePipeliningWaitAttributedToStatement` and a new `AshTestVerifyOccurrence` test for the new wait states. Without the fix, the test shows query_id 0 samples, with the fix all samples have proper ids. Reviewers: asaha Reviewed By: asaha Subscribers: ybase, yql Differential Revision: https://phorge.dev.yugabyte.com/D56661

2026-08-06

[#32332] DocDB: Add new ASH events for Write Pipelining Summary: Adding in two new WaitStateCodes: - `kRaft_WaitingForPipelinedReplication` waits on the tserver in `TabletPeer::RegisterAsyncWriteCompletion` until the op is Raft-committed. - `kYBClient_WaitingForPipelinedWrites` waits on the client for the pipelined writes to drain before the transaction can commit. Making a few changes to WaitForAsyncWrite for `kRaft_WaitingForPipelinedReplication`: - Adding `yb.rpc.send_metadata` to the RPC - WaitForAsyncWriteRpc copies the metadata of the issuing statement's WaitStateInfo. Since this RPC outlives the issuing statement, we need to do this to keep this state around. For kYBClient_WaitingForPipelinedWrites`: `YBTransaction::Impl::Commit` defers the commit into `async_write_commit_waiter_` while writes are still in flight, and that callback runs on the thread that completes the last write, which carries no wait state. Capturing the current wait state at deferral time and adopting it in the waiter, otherwise the COMMITTED record and the APPLYING fan-out it triggers are left unattributed. **Upgrade/Rollback safety:** Only adding `yb.rpc.send_metadata` to `WaitForAsyncWrite`. This just makes the client populate the existing optional metadata field in the request header. If omitted then we have current behaviour. Test Plan: Added `WritePipeliningWaitAttributedToStatement` and a new `AshTestVerifyOccurrence` test for the new wait states. Without the fix, the test shows query_id 0 samples, with the fix all samples have proper ids. Reviewers: asaha Reviewed By: asaha Subscribers: yql, ybase Differential Revision: https://phorge.dev.yugabyte.com/D56271

[#32314] DocDB: Dynamic DocDB History Retention for Long-Running Transactions (Phase 1, Database level) Summary: Currently, any snapshot older than `timestamp_history_retention_interval_sec` (defaulted to 15 minutes) can be deleted by compaction, even if the snapshot is still being read by an ongoing transaction. Once the snapshot is deleted, any further read attempt will fail with Snapshot Too Old error and immediately aborts, blocking all long running transactions/DDLs. The only workaround right now is to increase the history retention time across the cluster, which results in unnecessary data retention and database bloat. We wish to keep track of all the long running transactions, and prevent the deletion of the snapshots that they are currently reading from. **Overview:** On every tserver, each YSQL session registers/saves the read time of its current transaction and the database id that it's reading from. In the tserver's heartbeat to master RPC (defaulted to once per second), tserver iterates through all sessions, finds the earliest read time(the pin) of each database, and sends them to master if it has lived for longer than `db_history_retention_pin_min_txn_age_sec` (defaulted to 5 minutes). The master stores every tserver's pins, finds the earliest pin of each database globally, and sends them back to the tserver in the master heartbeat RPC response. Tserver then saves the global database pins, and forbids compaction from deleting any database snapshot younger or equal to the pin. A hard cap `db_history_retention_pin_max_txn_age_sec` (defaulted to 1 day) has also been added to ensure that a pin cannot block compaction indefinitely, and any snapshot older than it will be marked available for compaction even if it is held by pins. **Implementation details/choices:** The session's registered read time is an atomic `uint64` that lives on each `PgClientSession` object instead of a centralized map/tracker to reduce lock contention. When tserver's `TryHeartbeat` is called, it only acquires a shared lock to the tserver's `sessions_` container (always mandatory since sessions can be destroyed while heartbeat runs and may result in nullptr dereference without locking), and reads each pin atomically without locking at the session level. This may lead to slightly stale pins, but it is negligible given that it will be immediately corrected in the next heartbeat and the generous history retention window. The master keeps a copy of each tserver's local database pins that lives on the `TSDescriptor`. On receiving a tserver heartbeat RPC, that server's previous pins is replaced by current pins in the RPC payload to ensure that there are no stale pins. Given the majority use case where a master will be assigned less than 3000 tservers and the possibility of a tserver dropping at any moment, iterating through all live tservers (obtained from `TabletServer::GetLiveTServers`) upon receiving a heartbeat is the simplest and should be sufficient. Each database's pin can only increases monotonically once master has received at least 1 heartbeat from every tserver. Given the 15 minute history retention window, it is assumed that by the time compaction runs, every tserver would have either already sent multiple heartbeats and received the correct global database pins, or marked as failure and kicked out of the cluster. In the event where a tserver's heartbeat gets lost, the master will retain the previous set of local pins of that server, and the tserver will retain the previous set of global pins received from master. These pins will be stale, but given the assumption, this will never result in the tserver compacting snapshots that are being read. This also cannot lead to a tserver blocking compaction indefinitely as master will mark any tserver UNRESPONSIVE if a heartbeat has not received for longer than `tserver_unresponsive_timeout_ms` (defaulted to 1 minute), and only the pins of live tservers are used when computing the global pins. The earliest active PG snapshot serial number is determined through an asynchronous producer/consumer model: On any ActiveSnapshot stack or RegisteredSnapshots heap change, PG’s snapshot manager (`snapmgr.c`) determines the oldest live read-point serial by iterating through both structures and immediately publishes it to session shared memory. Then, tserver consumes (reads) the published serial # and maps it to a YB HybridTime. Since the mapping is done by reading the `pg_client_sessions`'s `read_point_` map which requires acquiring the session lock, it is done in 2 places: `DoPerform`, a guaranteed mapping since the session lock is always held when the Perform RPC is made, and a best-effort mapping on `GetDbHistoryRetentionPin` called by `TryHeartbeat`, which attempts to acquire lock and maps if succeeded, and skips otherwise since Heartbeat cannot afford to busy wait. This preserves the earliest snapshot across nested function calls within a statement (i.e. `SELECT * FROM table WHERE key = some_func()` and for the lifetime of a cursor, while allowing independent Read Committed statements to use newer snapshots. As functions cannot be invoked on their own, they will always be a part of a statement, and the pins will not advance as long as the outside statement is still executing. Procedures, on the other hand, must be invoked with `CALL`. Statements inside effectively act as standalone statements, and the pin will be advanced after each statement finishes executing. The `SAVEPOINT` statement acts as a subtransaction that is allowed within in an ongoing uncommitted transaction and does not create an actual snapshots. PG marks all statements executed inside the SAVEPOINT block with a SubXID and invalidate all statement if the savepoint was reloaded, and this cannot cause a snapshot too old error. When inserting/updating rows with Foreign key constraints, the lastest snapshot should be used regardless of transaction isolation level (even repeatable read). This diff does not interfere with the existing behavior, where a foreign key check on the latest snapshot is ran when an insert/update executes, and a `Serialization Error` is returned if the foreign key involved is modified/removed by another pg process. **Upgrade/Rollback safety:** This diff introduces `ts_ysql_db_oldest_pinned_read_times` in the TSHeartbeat RPC's request, and `cluster_ysql_db_oldest_pinned_read_times` in TSHeartbeat's response. In the case of a Tserver/Master version mismatch, if Tserver's version is newer, master will ignore the local pin map provided by Tserver in the heartbeat, and Tserver will treat the global cluster pin map as empty since master will not provide one. If master's version is newer, Tserver retains the previous compaction logic and compact solely based on the history retention window. In both cases, compaction falls back to the previous behaviour, and the cluster functions as normal with no additional handling needed. This feature is also protected by the gFlag `enable_db_history_retention_pins` (set to true by default), and the GUC `yb_db_history_retention_pin_mode` (can be toggled between all, DDLs only, or none, set to DDLs only by default). Test Plan: **Test cases:** Tserver: `./yb_build.sh release --cxx-test ts_tablet_manager-test --gtest_filter 'ComputeDbHistoryRetentionPinCutoffTest.*'` Master: `./yb_build.sh release --cxx-test master-test --gtest_filter 'MasterTest.*YsqlDbOldestPinnedReadTimes*'` **Manual testing (correctness):** Start a local cluster (`bin/yb-ctl create`) and open 2 sessions (s1, s2) connected to the same cluster in 2 separate terminals (`bin/ysqlsh`). Execute, in order: s1: ``` DROP TABLE IF EXISTS compaction_test; CREATE TABLE compaction_test (k INT PRIMARY KEY, v INT) SPLIT INTO 1 TABLETS; INSERT INTO compaction_test VALUES (1, 100); ``` s2: ``` BEGIN TRANSACTION ISOLATION LEVEL REPEATABLE READ; SELECT * FROM compaction_test WHERE k = 1; ``` s1: ``` UPDATE compaction_test SET v = v + 1 WHERE k = 1; UPDATE compaction_test SET v = v + 1 WHERE k = 1; ``` Wait 15 minutes (the default history retention window), then in another terminal, manually trigger compaction: `yb-admin -master_addresses 127.0.0.1:7100 compact_table ysql.yugabyte compaction_test 60` Finally, execute on s2: ``` SELECT * FROM compaction_test WHERE k = 1; ``` This should result in a snapshot too old error on previous builds, but will execute successfully on this diff's build. **Performance testing:** https://perf.dev.yugabyte.com/report/view/W3siaXNCYXNlbGluZSI6ZmFsc2UsIm5hbWUiOiJTZWxlY3RlZCBUZXN0LWlkIiwidGVzdF9pZCI6MTY2NTg3MjR9LHsiaXNCYXNlbGluZSI6dHJ1ZSwibmFtZSI6IkJhc2VsaW5lIiwidGVzdF9pZCI6IjE2NjU4NzgzIn1d?metric=latency This change does not introduce significant performance overhead compared to the baseline. Reviewers: kfranz, timur, dmitry, zdrudi Reviewed By: kfranz, timur, zdrudi Subscribers: sergei, timur, patnaik.balivada, jason, yql, ybase Differential Revision: https://phorge.dev.yugabyte.com/D54598

2026-08-05

dist-trace: propagate trace context across the RPC boundary Carry a distributed trace from an RPC caller to its callee: define the wire type, write it into the request header, decode it on the far side, and open a server span parented under the caller's client span. All of it lands together because none of the pieces is observable on its own -- a wire field nobody reads, or a reader with nothing on the wire, cannot be tested. Wire type (rpc_header.proto): - TraceContextPB (W3C trace-context) and RequestHeader.trace_context. This is the type shared by the RPC-header and shared-memory transports; the latter starts using it in a later commit. Writer (outbound_call.cc): - SetRequestParam serializes the active client span's SpanContext into the trace_context submessage while it is sizing and writing the header, so the header is still written in a single pass. ~30 bytes on the wire, and only when a trace is active; an older peer skips the unknown field. Reader (serialization.{cc,h}): - ParseTraceContext(Slice) decodes the RequestHeader.trace_context wire field; ToSpanContext(TraceContextPB) is the shared-memory equivalent; both go through BuildSpanContext and fail on a zero/invalid context. - ParseHeader captures RequestHeader.trace_context into ParsedRequestHeader. Server / inbound span (yb_rpc.{cc,h}): - YBInboundCall::ParseFrom parses the header trace_context into parent_span_context_ (best-effort: a bad context is logged, never fails the RPC). - CreateServerSpan starts a StartServerSpanWithScope child of that parent (remote) or of an explicitly passed context (local); RespondSuccess/Failure set status and end it; DropServerSpanScope releases the scope on the handler thread while the span ends later. Client local-call handoff (local_call.{cc,h}, rpc_context.cc): - LocalOutboundCall tags rpc.local_call and drops its client-span scope in the ctor (a local call never hops threads to drop it later). - otel_span_context() exposes the outbound span's context so the local inbound span can parent under it (no wire header exists locally). - RpcContext creates the server span: from the wire header for remote calls, from the outbound call's context for local calls. Process init (tserver/db_server_base.cc): - DbServerBase is the shared base of Master and TabletServer, so DbServerBase:: Init is the single place that initializes the process-wide tracer (service name = server name, node id = permanent uuid), with Shutdown tearing it down. Gated by IsDistTraceEnabled, so it is a no-op when tracing is off. Without this the server spans above are never exported, which is why the 10 lines ship here rather than as a commit of their own. Test: TestRpcSpanReachesTabletServer runs a SELECT under a known traceparent and asserts that the TabletServer's inbound "rpc yb.tserver.PgClientService.Perform" span lands in that trace as a child of the ysql backend's outbound span, with the expected rpc.service/rpc.method attributes. The new WaitForRemoteChildSpan collector helper does the caller/callee pairing check.

Revert "[#32314] DocDB: Dynamic DocDB History Retention for Long-Running Transactions (Phase 1, Database level)" Summary: This reverts commit 7b100f3d47063801a8fc9185f8cf9ca500d247b1. It introduced a TSAN data race on PgClientSession::Impl::database_oid_: the new lock-free heartbeat path (TryHeartbeat -> GetDatabasePins -> GetDbHistoryRetentionPin) reads database_oid_ without the session lock, racing with its lazy initialization in EnsureClientSessionCgroup, which runs under the session lock. Detected by TSAN in CDCSDKConsistentSnapshotTest.InsertBeforeDuringAfterSnapshot: https://csiweb.dev.yugabyte.com/csi/file/text/22763789.txt _automated · Claude Code (Fable 5)_ **Upgrade/Rollback safety:** Revert of not released changes Test Plan: jenkins Reviewers: arybochkin, amitanand Reviewed By: amitanand Subscribers: timur, yql, ybase Tags: #jenkins-ready Differential Revision: https://phorge.dev.yugabyte.com/D56579

2026-08-04

[#32784] xCluster: Tag xCluster internal PG connections with YbInternalConnKind Summary: A tserver opens internal (yb-tserver-key authenticated) libpq connections to its own local postgres for several purposes. The `YbInternalConnKind` registry gives each kind its own `BackendType`, makes it show up distinctly in `pg_stat_activity.backend_type`, and lets it receive per-kind behavior (e.g. regarding catalog cache preloading). xCluster opens such internal connections in two lifecycles that were previously untagged (lumped into the generic internal-connection bucket): - steady-state DDL replay by `XClusterDDLQueueHandler` (long-lived, holds the ddl_queue advisory lock, replays arbitrary DDL), and - master-driven setup/teardown (`CREATE`/`DROP EXTENSION` and `ddl_queue` table cleanup), routed through the `AdminExecutePgsql` RPC. Register two new `YbInternalConnKind` values -- `xcluster_ddl_queue` and `xcluster_setup` -- so each gets its own BackendType and shows up distinctly in `pg_stat_activity.backend_type`, and can be given per-kind behavior later (right now this is mainly used to customize the catalog cache preloading). **Upgrade/Rollback safety:** This diff adds an optional string field, `yb_internal_conn_kind`, to `AdminExecutePgsqlRequestPB`, which is sent over the wire from the master to the closest tserver. This is safe during upgrade/downgrade in every scenario: | Master | Tserver | Result | | --- | --- | --- | | new | old | Old tserver doesn't know field 3 and ignores it; connection stays untagged → pre-change behavior | | old | new | Field absent, so the tserver reads an empty string and leaves the connection untagged → pre-change behavior | | new | new | Connection tagged `xcluster_setup` | Test Plan: ``` ./yb_build.sh release --cxx-test pg_relcache_init_internal_conn-test ``` Reviewers: mlillibridge, xCluster, hsunder, jhe Reviewed By: jhe Subscribers: yql, ybase, svc_phabricator Differential Revision: https://phorge.dev.yugabyte.com/D55930

dist-trace: propagate trace context across the RPC boundary Carry a distributed trace from an RPC caller to its callee: define the wire type, write it into the request header, decode it on the far side, and open a server span parented under the caller's client span. All of it lands together because none of the pieces is observable on its own -- a wire field nobody reads, or a reader with nothing on the wire, cannot be tested. Wire type (rpc_header.proto): - TraceContextPB (W3C trace-context) and RequestHeader.trace_context. This is the type shared by the RPC-header and shared-memory transports; the latter starts using it in a later commit. Writer (outbound_call.cc): - SetRequestParam serializes the active client span's SpanContext into the trace_context submessage while it is sizing and writing the header, so the header is still written in a single pass. ~30 bytes on the wire, and only when a trace is active; an older peer skips the unknown field. Reader (serialization.{cc,h}): - ParseTraceContext(Slice) decodes the RequestHeader.trace_context wire field; ToSpanContext(TraceContextPB) is the shared-memory equivalent; both go through BuildSpanContext and fail on a zero/invalid context. - ParseHeader captures RequestHeader.trace_context into ParsedRequestHeader. Server / inbound span (yb_rpc.{cc,h}): - YBInboundCall::ParseFrom parses the header trace_context into parent_span_context_ (best-effort: a bad context is logged, never fails the RPC). - CreateServerSpan starts a StartServerSpanWithScope child of that parent (remote) or of an explicitly passed context (local); RespondSuccess/Failure set status and end it; DropServerSpanScope releases the scope on the handler thread while the span ends later. Client local-call handoff (local_call.{cc,h}, rpc_context.cc): - LocalOutboundCall tags rpc.local_call and drops its client-span scope in the ctor (a local call never hops threads to drop it later). - otel_span_context() exposes the outbound span's context so the local inbound span can parent under it (no wire header exists locally). - RpcContext creates the server span: from the wire header for remote calls, from the outbound call's context for local calls. Process init (tserver/db_server_base.cc): - DbServerBase is the shared base of Master and TabletServer, so DbServerBase:: Init is the single place that initializes the process-wide tracer (service name = server name, node id = permanent uuid), with Shutdown tearing it down. Gated by IsDistTraceEnabled, so it is a no-op when tracing is off. Without this the server spans above are never exported, which is why the 10 lines ship here rather than as a commit of their own. Test: TestRpcSpanReachesTabletServer runs a SELECT under a known traceparent and asserts that the TabletServer's inbound "rpc yb.tserver.PgClientService.Perform" span lands in that trace as a child of the ysql backend's outbound span, with the expected rpc.service/rpc.method attributes. The new WaitForRemoteChildSpan collector helper does the caller/callee pairing check.

[#32314] DocDB: Dynamic DocDB History Retention for Long-Running Transactions (Phase 1, Database level) Summary: Currently, any snapshot older than `timestamp_history_retention_interval_sec` (defaulted to 15 minutes) can be deleted by compaction, even if the snapshot is still being read by an ongoing transaction. Once the snapshot is deleted, any further read attempt will fail with Snapshot Too Old error and immediately aborts, blocking all long running transactions/DDLs. The only workaround right now is to increase the history retention time across the cluster, which results in unnecessary data retention and database bloat. We wish to keep track of all the long running transactions, and prevent the deletion of the snapshots that they are currently reading from. **Overview:** On every tserver, each YSQL session registers/saves the read time of its current transaction and the database id that it's reading from. In the tserver's heartbeat to master RPC (defaulted to once per second), tserver iterates through all sessions, finds the earliest read time(the pin) of each database, and sends them to master if it has lived for longer than `db_history_retention_pin_min_txn_age_sec` (defaulted to 5 minutes). The master stores every tserver's pins, finds the earliest pin of each database globally, and sends them back to the tserver in the master heartbeat RPC response. Tserver then saves the global database pins, and forbids compaction from deleting any database snapshot younger or equal to the pin. A hard cap `db_history_retention_pin_max_txn_age_sec` (defaulted to 1 day) has also been added to ensure that a pin cannot block compaction indefinitely, and any snapshot older than it will be marked available for compaction even if it is held by pins. **Implementation details/choices:** The session's registered read time is an atomic `uint64` that lives on each `PgClientSession` object instead of a centralized map/tracker to reduce lock contention. When tserver's `TryHeartbeat` is called, it only acquires a shared lock to the tserver's `sessions_` container (always mandatory since sessions can be destroyed while heartbeat runs and may result in nullptr dereference without locking), and reads each pin atomically without locking at the session level. This may lead to slightly stale pins, but it is negligible given that it will be immediately corrected in the next heartbeat and the generous history retention window. The master keeps a copy of each tserver's local database pins that lives on the `TSDescriptor`. On receiving a tserver heartbeat RPC, that server's previous pins is replaced by current pins in the RPC payload to ensure that there are no stale pins. Given the majority use case where a master will be assigned less than 3000 tservers and the possibility of a tserver dropping at any moment, iterating through all live tservers (obtained from `TabletServer::GetLiveTServers`) upon receiving a heartbeat is the simplest and should be sufficient. Each database's pin can only increases monotonically once master has received at least 1 heartbeat from every tserver. Given the 15 minute history retention window, it is assumed that by the time compaction runs, every tserver would have either already sent multiple heartbeats and received the correct global database pins, or marked as failure and kicked out of the cluster. In the event where a tserver's heartbeat gets lost, the master will retain the previous set of local pins of that server, and the tserver will retain the previous set of global pins received from master. These pins will be stale, but given the assumption, this will never result in the tserver compacting snapshots that are being read. This also cannot lead to a tserver blocking compaction indefinitely as master will mark any tserver UNRESPONSIVE if a heartbeat has not received for longer than `tserver_unresponsive_timeout_ms` (defaulted to 1 minute), and only the pins of live tservers are used when computing the global pins. The earliest active PG snapshot serial number is determined through an asynchronous producer/consumer model: On any ActiveSnapshot stack or RegisteredSnapshots heap change, PG’s snapshot manager (`snapmgr.c`) determines the oldest live read-point serial by iterating through both structures and immediately publishes it to session shared memory. Then, tserver consumes (reads) the published serial # and maps it to a YB HybridTime. Since the mapping is done by reading the `pg_client_sessions`'s `read_point_` map which requires acquiring the session lock, it is done in 2 places: `DoPerform`, a guaranteed mapping since the session lock is always held when the Perform RPC is made, and a best-effort mapping on `GetDbHistoryRetentionPin` called by `TryHeartbeat`, which attempts to acquire lock and maps if succeeded, and skips otherwise since Heartbeat cannot afford to busy wait. This preserves the earliest snapshot across nested function calls within a statement (i.e. `SELECT * FROM table WHERE key = some_func()` and for the lifetime of a cursor, while allowing independent Read Committed statements to use newer snapshots. As functions cannot be invoked on their own, they will always be a part of a statement, and the pins will not advance as long as the outside statement is still executing. Procedures, on the other hand, must be invoked with `CALL`. Statements inside effectively act as standalone statements, and the pin will be advanced after each statement finishes executing. The `SAVEPOINT` statement acts as a subtransaction that is allowed within in an ongoing uncommitted transaction and does not create an actual snapshots. PG marks all statements executed inside the SAVEPOINT block with a SubXID and invalidate all statement if the savepoint was reloaded, and this cannot cause a snapshot too old error. When inserting/updating rows with Foreign key constraints, the lastest snapshot should be used regardless of transaction isolation level (even repeatable read). This diff does not interfere with the existing behavior, where a foreign key check on the latest snapshot is ran when an insert/update executes, and a `Serialization Error` is returned if the foreign key involved is modified/removed by another pg process. **Upgrade/Rollback safety:** This diff introduces `ts_ysql_db_oldest_pinned_read_times` in the TSHeartbeat RPC's request, and `cluster_ysql_db_oldest_pinned_read_times` in TSHeartbeat's response. In the case of a Tserver/Master version mismatch, if Tserver's version is newer, master will ignore the local pin map provided by Tserver in the heartbeat, and Tserver will treat the global cluster pin map as empty since master will not provide one. If master's version is newer, Tserver retains the previous compaction logic and compact solely based on the history retention window. In both cases, compaction falls back to the previous behaviour, and the cluster functions as normal with no additional handling needed. This feature is also protected by the gFlag `enable_db_history_retention_pins` (set to true by default), and the GUC `yb_db_history_retention_pin_mode` (can be toggled between all, DDLs only, or none, set to DDLs only by default). Test Plan: **Test cases:** Tserver: `./yb_build.sh release --cxx-test ts_tablet_manager-test --gtest_filter 'ComputeDbHistoryRetentionPinCutoffTest.*'` Master: `./yb_build.sh release --cxx-test master-test --gtest_filter 'MasterTest.*YsqlDbOldestPinnedReadTimes*'` **Manual testing (correctness):** Start a local cluster (`bin/yb-ctl create`) and open 2 sessions (s1, s2) connected to the same cluster in 2 separate terminals (`bin/ysqlsh`). Execute, in order: s1: ``` DROP TABLE IF EXISTS compaction_test; CREATE TABLE compaction_test (k INT PRIMARY KEY, v INT) SPLIT INTO 1 TABLETS; INSERT INTO compaction_test VALUES (1, 100); ``` s2: ``` BEGIN TRANSACTION ISOLATION LEVEL REPEATABLE READ; SELECT * FROM compaction_test WHERE k = 1; ``` s1: ``` UPDATE compaction_test SET v = v + 1 WHERE k = 1; UPDATE compaction_test SET v = v + 1 WHERE k = 1; ``` Wait 15 minutes (the default history retention window), then in another terminal, manually trigger compaction: `yb-admin -master_addresses 127.0.0.1:7100 compact_table ysql.yugabyte compaction_test 60` Finally, execute on s2: ``` SELECT * FROM compaction_test WHERE k = 1; ``` This should result in a snapshot too old error on previous builds, but will execute successfully on this diff's build. **Performance testing:** https://perf.dev.yugabyte.com/report/view/W3siaXNCYXNlbGluZSI6ZmFsc2UsIm5hbWUiOiJTZWxlY3RlZCBUZXN0LWlkIiwidGVzdF9pZCI6MTY2NTg3MjR9LHsiaXNCYXNlbGluZSI6dHJ1ZSwibmFtZSI6IkJhc2VsaW5lIiwidGVzdF9pZCI6IjE2NjU4NzgzIn1d?metric=latency This change does not introduce significant performance overhead compared to the baseline. Reviewers: kfranz, timur, dmitry, zdrudi Reviewed By: kfranz, timur, zdrudi Subscribers: sergei, timur, patnaik.balivada, jason, yql, ybase Differential Revision: https://phorge.dev.yugabyte.com/D54598

2026-08-03

dist-trace: add TraceContextPB and RequestHeader.trace_context Define TraceContextPB (W3C trace-context) and the trace_context field on the RPC RequestHeader, both in rpc_header.proto. This is the wire type shared by the RPC-header and shared-memory trace-propagation transports; no code references it yet.

[BACKPORT 2025.2][#31886,#32360,#32498,#32389,#32310,#31890] DocDB: Vector index: Indexed table should own reverse mapping Summary: This is a single change which contains all not yet backported subtasks of https://github.com/yugabyte/yugabyte-db/issues/31886. **Backported from parent #31886 (this revision)** Cherry-pick order: `#32360 => #32498 => #32389 => #32310 => #31890`. [[https://github.com/yugabyte/yugabyte-db/issues/32360 | #32360: Vector index: Add a new gflag and table parameter for vector reverse mapping ownership]] Original commit: b61cc9f8c154273e2b0c62f739018a6ae4c11a56 / D54966 [[https://github.com/yugabyte/yugabyte-db/issues/32498 | #32498: Vector Index: Add new value type for vector value]] Original commit: 7e23f2df4aa203c46cfb7055e40d06a0b789bfda / D55267 [[https://github.com/yugabyte/yugabyte-db/issues/32389 | #32389: Vector index: instrument reverse mapping ownership into insert and update code paths]] Original commit: c4371d826a57e6e54fdf9469f66124521dd00c60 / D55268 [[https://github.com/yugabyte/yugabyte-db/issues/32310 | #32310: Vector index: Reverse mapping is not added for YSQL single-shard fast-path writes]] Original commit: e0d459503f69b723e307f92acc0337a4da0846ae / D55383 [[https://github.com/yugabyte/yugabyte-db/issues/31890 | #31890: Vector index: reverse mapping entries are not cleaned up on drop]] Original commit: 3a369403f2a2b1b8b1ebcb1473bdfa5d7d3530fe / D55788 **Already on 2025.2 (not part of this diff)** [[https://github.com/yugabyte/yugabyte-db/issues/31958 | #31958: Vector Index: Skip reverse mapping insertion during backfill]] [[https://github.com/yugabyte/yugabyte-db/issues/32342 | #32342: Vector Index: Search should skip tombstoned reverse mappings]] **External dependencies (partial / adapted)** These are **not** full backports of the listed issues; only the pieces required to land the parent-task commits on `2025.2`. [[https://github.com/yugabyte/yugabyte-db/issues/32582 | #32582]] / D55491: **Partial.** `EncodedDocVectorValue::FromSlice` trailing size as `uint8_t`; `EncodeVectorSchemaMissingValue` / `DecodeVectorSchemaMissingValueForPgRow` (adapted to `QLValuePB`). Needed by `#31890` / `doc_vector_id-test`. **Not taken:** DDL encode in `pg_client_session.cc`, packed-row path, or `PgVectorTest.ReadVectorColumnAfterAddColumnDefault`. [[https://github.com/yugabyte/yugabyte-db/issues/26679 | #26679]] / D50522: **Partial (via `#32389` writer files).** `#32389` needs `VectorIndexesUpdater` / `ShouldMaintainVectorIndexes` / `TombstoneVectorReverseMappingIds`, brought in via post-`#32389` `rocksdb_writer.{h,cc}`. Restored 2025.2-only `TransactionMetadataUpdateWriter`: fixed `IntentsWriter::Apply`'s metadata classification to match 2025.2's byte layout (master's size-only check misclassified update records), and dropped the unrelated `DumpIntentsContext` debug-dump helper ([[https://github.com/yugabyte/yugabyte-db/issues/14641 | #14641]], out of scope). Full xCluster DDL / vector-index replication from `#26679` was **not** backported. [[https://github.com/yugabyte/yugabyte-db/issues/29705 | #29705]] / D49017 | **Not backported.** Master uses `QLValueMsg`; 2025.2 still uses `QLValuePB`. Call sites from `#32498` / `#31890` / `#32310` tests were adapted to `QLValuePB` instead of landing the lightweight-protobuf typedef migration. Also adapted `#32389` / related writer code that referenced master-only `yb::storage::` frontiers to `rocksdb::` (2025.2 still uses the RocksDB frontier types). **2025.2 adaptation notes** - Kept 2025.2 `ysql_use_packed_row_v2` / related flag definitions when `#32360` conflicted with newer master AutoFlag plumbing. - Dropped `#31899`'s `ExternalApplyGatesVectorIndexFeed` / `CountingVectorIndex` test double that rode in with the `#32310` conflict resolution: `#31899` is already on 2025.2 (`c6346fda8c2`), and that backport intentionally omitted this helper (it depends on master `storage::` frontiers). Kept only `#32310`'s `FastPathVectorReverseMapping` coverage. - Fixed merge damage in `pg_vector_index-test.cc` (missing closing brace after `StatusResolutionDuringBootstrapBackfill`). **Upgrade/Rollback safety** Guarded by master runtime auto-flag `enable_table_owned_vector_reverse_mapping` (`kExternal`, `false => true`). Test Plan: Jenkins Reviewers: sergei, hsunder, zdrudi Reviewed By: sergei, hsunder Subscribers: ybase Differential Revision: https://phorge.dev.yugabyte.com/D56410

2026-07-31

[#23954] xCluster: DDL Replication - Replicate backfill of create index Summary: **Background** xCluster's loop-prevention filter on the source drops any WAL record that has external_hybrid_time set. The intent was to drop writes that originated from xCluster itself so they don't echo back. The side effect is that index backfill writes also carry external_hybrid_time (the producer set it with the backfill read time), so they get dropped too. Because of this, the target runs its own full local backfill at the same hybrid time. the index goes through a window where source and target have it built independently from different snapshots. **Solution** The loop-prevention rule is changing: the consumer now sets a dedicated field, xcluster_target_applied, on every write it applies, and the producer filters on that instead of external_hybrid_time. With that change, source backfill writes are no longer caught by the filter and will replicate to the target. **New auto flag**: xcluster_filter_target_applied Each xCluster stream carries an immutable boolean` xcluster_use_target_applied_filter` recording whether the stream uses the new filter. This new field is set to true when new introduced auto flag is true and source is on automatic ddl mode. On the target, when BackfillTable decides what to do for a CREATE INDEX, it checks the stream's `xcluster_use_target_applied_filter`. If the bit is set, the target knows the source's backfill writes are coming over replication, so it skips its own local backfill and just waits until the namespace's xCluster safe time reaches the source's backfill hybrid time, then marks the index ready. If the bit is not set, the target falls back to the legacy local-backfill path. As part of create index, we make sure the safe time is create than the commit time from source, which is the source backfill hybrid time. So once index is created, all the backfill writes will be already replicated, and we can just mark backfill as done. **Upgrade/Rollback safety:** Streams created before the AutoFlag is promoted will keep the old behavior permanently, so partially upgraded clusters won’t hit a mid-stream behavior change. The required upgrade order is target first, then source. Test Plan: ./yb_build.sh --cxx-test xcluster_ddl_replication-test --gtest_filter XClusterDDLReplicationIndexBackfillTest.ReplicatedFromSource ./yb_build.sh --cxx-test xcluster_ddl_replication-test --gtest_filter XClusterDDLReplicationIndexBackfillTest.LegacyLocalBackfill ./yb_build.sh --cxx-test xcluster_ddl_replication-test --gtest_filter XClusterDDLReplicationIndexBackfillTest.TargetWaitsForReplicatedBackfill Reviewers: jhe, xCluster, hsunder Reviewed By: jhe Subscribers: ybase Differential Revision: https://phorge.dev.yugabyte.com/D52769

[BACKPORT 2025.2][#31542] docdb: Vector Index: Chunked compaction implementation Summary: The change introduces a chunked compaction. It allows to not build one in-memory merged index for all input vectors but to output into multiple chunk files, each bounded by a configurable memory budget, to reduce OOM risk. If the limit is very small, compaction still produces at least one vector per output chunk. New gflag `vector_index_compaction_chunk_max_mem_store_size_mb` (runtime, default 0) is introduced and should be greater than 0 to enable chunked compaction. Original commit: 2fd12083996acf2f8a481729d25ac8bcf1f76fb8 / D54283 **Upgrade/Rollback safety:** Just a comment update in .proto file. Test Plan: ./yb_build.sh --cxx-test ann_methods_vector_lsm-test --gtest_filter VectorLSMTest.ChunkedCompactionRespectsMemStoreLimit/kHnswlib ./yb_build.sh --cxx-test ann_methods_vector_lsm-test --gtest_filter VectorLSMTest.ChunkedCompactionRespectsMemStoreLimit/kUsearch ./yb_build.sh --cxx-test ann_methods_vector_lsm-test --gtest_filter VectorLSMTest.OpenAfterChunkedCompaction/kHnswlib ./yb_build.sh --cxx-test ann_methods_vector_lsm-test --gtest_filter VectorLSMTest.OpenAfterChunkedCompaction/kUsearch ./yb_build.sh --cxx-test ann_methods_vector_lsm-test --gtest_filter VectorLSMTest.DefaultCompactionMergesMultipleChunks/kHnswlib ./yb_build.sh --cxx-test ann_methods_vector_lsm-test --gtest_filter VectorLSMTest.DefaultCompactionMergesMultipleChunks/kUsearch Reviewers: sergei, zdrudi Reviewed By: zdrudi Subscribers: ybase Differential Revision: https://phorge.dev.yugabyte.com/D55386

2026-07-30

[BACKPORT 2025.2][#32321] DocDB: Postpone tablet split until vector index backfill completes Summary: During vector index creation the reverse mapping records are written into the tablet's regular RocksDB as part of the tserver-side backfill. This growth can push the tablet past the split threshold, and because the vector index backfill runs on the tserver (the master skips its backfill orchestration) the master's existing `table->IsBackfilling()` gate does not cover it. As a result an automatic tablet split could start while the backfill was still running. Report a new `has_active_vector_index_backfill` flag from the tserver to the master, mirroring `may_have_orphaned_post_split_data`. `TabletVectorIndexes::HasActiveBackfill` returns true while any vector index on the tablet has not finished backfilling. `CatalogManager::ShouldSplitValidCandidate` now rejects a tablet whose leader reports an active vector index backfill, so the split is postponed until the backfill completes. Manual splits are unaffected, consistent with the existing checks in that method. **Upgrade/Rollback safety:** Added new field for new functionality. --- _automated · Claude Code (Opus 4.8)_ Original commit: 5adc7e22a8ca215c9c71c913a2feafa77d5a4345 / D55282 Test Plan: ./yb_build.sh debug --cxx-test pg_vector_index-test --gtest_filter 'PgDistributedVectorIndexTest.AutoSplitDuringBackfill/None' Reviewers: arybochkin, hsunder Reviewed By: arybochkin Subscribers: ybase, yql Tags: #jenkins-ready Differential Revision: https://phorge.dev.yugabyte.com/D55447

2026-07-29

[BACKPORT 2025.2][#30720] DocDB: Block based hnswlib Summary: We already use hnswlib as one of the vector index backends. However, unlike usearch, we currently do not support generating a block-based vector index representation from an hnswlib index. At the moment, block-based index construction is only implemented for the usearch backend. To make the system consistent across supported backends, we should extend this functionality to hnswlib as well. This diff adds support for using an existing hnswlib index as a source and converting it into the block-based vector index representation, in the same way we currently do for usearch. **Upgrade/Rollback safety:** Added entry to enum and renamed existing entry. Both changes are backward compatible. Original commit: 362f867bb6f824658a579e6353ca5a983f930408 / D51066 Test Plan: Jenkins Reviewers: arybochkin, hsunder Reviewed By: arybochkin Subscribers: ybase, yql Tags: #jenkins-ready Differential Revision: https://phorge.dev.yugabyte.com/D56232

2026-07-28

[BACKPORT 2026.1][#32335] YSQL: Pass single read time option from pggate to pg client session Summary: Original commit: 5db3d9a1d1a5b0f1a8f7e78cfde62dec30881ef8 / D54921 #### Problem Prior to this revision, multiple read time options may be set on PerformOptionsPB. It is unclear which read time option should be preferred. Example: - Parallel query may set ENSURE_READ_TIME_IS_SET read_time_manipulation. - GUC yb_read_after_commit_visibility=relaxed sets clamp_uncertainty_window option. Both options are set on the proto and the clamp option should be preferred. This is not obvious. #### Approach 1. Decide the preference in pggate itself. 2. Send only a single read time option in the perform RPC. This simplifies the contract between pggate and pg client session. Additionally: 1. Collect read time options logic into PgTxnManager::SetReadTimeOptions 2. Add appropriate validation checks for incompatible read time options. 3. Document this in pggate/README in section named Read point selection section. 4. Rename catalog session to legacy catalog session since plain sessions are used for catalog reads with newer concurrent DDL feature. 5. On similar lines, rename kDDL sessions to kAutonomousDDL. #### Some notes: 1. Follower reads are only applicable to read only transactions. And serializable never occurs in read only transactions. Therefore, follower reads and serializable do not co-occur. 2. Previously, parallel scans overwrote RESTART flag to ENSURE_READ_TIME_IS_SET. Instead, RESTART should take precedence because RESTART picks a read time on pg client session as well. 3. Previously, clamp and deferred options were ignored if any of the writes within the txn are non transactional. Now, the perform RPC must have a non transactional write to ignore clamp and deferred options. #### Upgrade/Rollback safety Perform RPC is intra node and therefore the client and the server are co-versioned. #### Merge conflicts Resolved one conflict in src/yb/tserver/pg_client_session.cc. 2026.1 does not have the master-only refactor that moved GetSessionKindBasedOnDDLOptions out of PgClientSession::Impl into free functions in the anonymous namespace; on 2026.1 it is still a private member, and the anonymous namespace holds the Status Commit() helper at that location instead. - pg_client_session.cc:1233 - kept 2026.1's Status Commit() helper and dropped the cherry-pick's GetSessionKindBasedOnDDLOptions / IsDDLUseRegularTxnBlock free functions, which belong to that unrelated refactor rather than to this revision. - pg_client_session.cc:2170 - applied this revision's kDdl -> kAutonomousDdl rename to 2026.1's member-function equivalent, which is where the enum value is used on this branch. Test Plan: Jenkins ./yb_build.sh release --cxx-test pg_read_time-test Reviewers: pjain, smishra, sanketh, bkolagani Reviewed By: pjain Subscribers: ybase, yql Differential Revision: https://phorge.dev.yugabyte.com/D56115

[BACKPORT 2025.2][#28911] DocDB: Remove DummyANN from DocDB Summary: DummyANN was created to provide ability to create postgres side part of vector index stuff, while DocDB lack vector index support. Now DummyANN is not used, and just consumes time in case of refactoring or API changes. This diff removes DummyANN from DocDB. **Upgrade/rollback safety:** Marked test only enum entry as deprecated. Jira: DB-18634 Original commit: 990827155d566705a67e2f5e0605ac17b176980c / D47396 Test Plan: Jenkins Reviewers: arybochkin Reviewed By: arybochkin Subscribers: yql, ybase Tags: #jenkins-ready Differential Revision: https://phorge.dev.yugabyte.com/D56203

[BACKPORT 2026.1.1][PLAT-20725][PLAT-21159][PLAT-21758] Log retention logic enhancements Summary: Allow keeping audit logs on the DB nodes for compliancy This is a requirement from customers, who don't want to export audit logs to external systems, but want to still be compliant with security requirements. Basically, what it does is - before gzipping postgres (YSQL) or tserser (YCQl) logs we're copying over all the audit line logs from the file to ./audit/[ysql or ycql]/......audit.log file. Now, we zip this new audit log file as well as the original log file. Original log file will be cleaned up, while audit log gzipped file will be kept on the node until the configured amount of days pass - and deleted after that. Number of days are controlled via additional autid logs setting. If the setting is not configured (or is configured to 0) - we keep the old behaviour. Test Plan: Installed older YBA release Created universe. Configured YSQL audit logs via the UI. Upgraded YBA. Re-configured YSQL audit logs via the UI to set the retention inetrval to 1 day. Make sure log rotation script was updated + otel-collector/log_cleanup_env file contains the new setting. Wait for postgres log file to be gzipped. Make sure audit log file gzip was created as well. Wait for 1 day to pass. Make sure the audit log file was deleted after the script run. Reviewers: #yba-api-review!, vbansal Reviewed By: vbansal Subscribers: yugaware Differential Revision: https://phorge.dev.yugabyte.com/D56179

[BACKPORT 2026.1][PLAT-20725][PLAT-21159][PLAT-21758] Log retention logic enhancements Summary: Allow keeping audit logs on the DB nodes for compliancy This is a requirement from customers, who don't want to export audit logs to external systems, but want to still be compliant with security requirements. Basically, what it does is - before gzipping postgres (YSQL) or tserser (YCQl) logs we're copying over all the audit line logs from the file to ./audit/[ysql or ycql]/......audit.log file. Now, we zip this new audit log file as well as the original log file. Original log file will be cleaned up, while audit log gzipped file will be kept on the node until the configured amount of days pass - and deleted after that. Number of days are controlled via additional autid logs setting. If the setting is not configured (or is configured to 0) - we keep the old behaviour. Test Plan: Installed older YBA release Created universe. Configured YSQL audit logs via the UI. Upgraded YBA. Re-configured YSQL audit logs via the UI to set the retention inetrval to 1 day. Make sure log rotation script was updated + otel-collector/log_cleanup_env file contains the new setting. Wait for postgres log file to be gzipped. Make sure audit log file gzip was created as well. Wait for 1 day to pass. Make sure the audit log file was deleted after the script run. Reviewers: vbansal, #yba-api-review! Reviewed By: vbansal Subscribers: yugaware Differential Revision: https://phorge.dev.yugabyte.com/D56178

2026-07-27

[PLAT-21758] Fix issues in audit log retention logic Summary: This fixes few issues with audit log retention logic and log cleanup in general: 1. If we failed to extract audit log entries to a separate file - leave the original file alone to avoid losing audit logs. 2. Make sure overall PG logs cleanup does not also clean audit logs - we have separate logic for that. 3. Make sure the script returns non-0 exit code in case something fails. This makes sure alert is raised and we can address that in time. 4. Update log cleanup script on each audit log config change to always pick up the latest fixes. Test Plan: Change audit log retention time Make sure script is updated. Disable writing to audit logs dir. Make sure script fails, returns non-0 exit code and leaves the original files untouched. Enable writing back. Run the script again with --postgres_max_log_size 0 - make sure it successfully retrieves audit logs and removes the original files. Run the script again with --postgres_max_log_size 0 - make surre audit log files stay untouched. Reviewers: vbansal Reviewed By: vbansal Subscribers: yugaware Differential Revision: https://phorge.dev.yugabyte.com/D56090

dist-trace: add TraceContextPB and RequestHeader.trace_context Define TraceContextPB (W3C trace-context) and the trace_context field on the RPC RequestHeader, both in rpc_header.proto. This is the wire type shared by the RPC-header and shared-memory trace-propagation transports; no code references it yet.

2026-07-25

[BACKPORT 2026.1][#31149] CDC: Allow gRPC stream creation via PG syntax Summary: #### Backport Description No merge conflicts were encountered. #### Original Description ##### Code Changes summary This change lets gRPC CDC streams be created and managed through the standard PostgreSQL replication-slot interface. Such streams gets a `slot_name`, `yb_grpc` plugin_name. These streams are also populated with `replica_identity_map` in their metadata and a `slot entry` in cdc_state table. This change is gated behind an auto flag `FLAGS_cdc_pg_create_grpc_stream` that turns on only after the cluster is finalised on a supporting version. The current behaviour of creating gRPC streams via `yb-admin` cmd is still available but is discouraged to use via a warning. gRPC streams created via such method also get the `slot_name` and `yb_grpc` plugin name in their metadata and `slot entry` in cdc_state table. However, such streams still will rely on `record_type` instead of `replica_identity_map`. Thus, in general we can have 3 types of gRPC streams once the upgrade of a universe is finalized. 1. **Old streams**: - These are streams which were either already present in universe before upgrade or the streams which got created after upgrade but before upgrade-finalization. - They don't have `slot_name`, `plugin_name`, `replica_identity_map` in their metadata and `slot_entry` in cdc_state table. 2. **New streams created via yb-admin**: - Streams being created post upgrade-finalization comes under this category. - Such streams will have `slot_name`, `plugin_name` in their metadata and `slot_entry` in cdc_state table. They won't have `replica_identity_map` but would rely on `record_type` passed in yb-admin cmd. 3. **New streams created via PG syntax** - These are also streams which are being created post upgrade-finalization. If a gRPC stream is attempted for creation via PG syntax before upgrade, then it is errored out. - Such streams will have `slot_name`, `plugin_name` and `replica_identity_map` in their metadata and `slot_entry` in cdc_state table. They behave analogous to logical_replication_streams. In addition to changes to stream creation, this revision also adds a mechanism to backfill old gRPC streams with: - auto-generated `slot_name` (formatted as `grpc_<stream_id>`), - `yb_grpc` plugin_name, and - a `slot entry` in cdc_state table. Thus once the backfill completes, the `type 1` streams mentioned above become analogous to `type 2` streams. This revision also changes the plugin name from "" to `yboutput` for `yb_system` notifications streams (i.e for LISTEN/NOTIFY slots) since we are using below as invariant for gRPC streams: - if there's no plugin_name field set in create stream request or in persisted data. - if present, plugin_name should be empty or should be `yb_grpc`. Along with this, it also backfill the plugin name to `yboutput` for existing notification streams. > The original revision was raised by **Bakul Gupta**. This is being commandeered by me after his internship completion. ##### Upgrade/rollback safety considerations: Feature is gated by the AutoFlag `FLAGS_cdc_pg_create_grpc_stream`. Creation via PG syntax and the legacy-gRPC backfill take effect only after promotion at upgrade-finalization, so the change is rollback-safe while the flag is unpromoted. **Proto change:** the only wire-format change is a new field `optional bool is_notification_slot` in `CDCStreamInfoPB` (the `GetCDCStreamResponsePB.stream()` message) which is present in master. It is upgrade/rollback safe because even if the nodes which are not upgraded yet won't have any discrepancy (wrt to their existing code) to differentiate between the notification slots and others. The existing code used to rely on replication slot name for checking logical replication streams. This old logic will clearly classify the notification streams as logical replication. ##### Considerations for colocated tables: NA ##### Considerations for connector: Streams created via PG syntax do not carry the `record_type` option (they use `replica_identity_map`). The implications of it for the gRPC (Debezium) connector are: 1. The connector's `LogicalDecoder` enum must recognize `yb_grpc` so that `plugin.name=yb_grpc` is accepted and routed to the yb-proto decoder. The [latest released connector](https://github.com/yugabyte/debezium-connector-yugabytedb/releases/tag/vdz.1.9.5.yb.grpc.2026.1) **will not work** with such streams as its `LogicalDecoder` enum has no `yb_grpc` value, so config validation rejects `plugin.name=yb_grpc` and the connector fails to start. - We have an ongoing [PR](https://github.com/yugabyte/debezium-connector-yugabytedb/pull/409) for it. 2. Consuming a stream that omits `record_type` also requires treating such missing `record_type`. The [PR](https://github.com/yugabyte/debezium-connector-yugabytedb/pull/402) already handles it and is available in [latest released connector](https://github.com/yugabyte/debezium-connector-yugabytedb/releases/tag/vdz.1.9.5.yb.grpc.2026.1). Streams created via `yb-admin` still carry `record_type`, so they keep working with existing connectors. Jira: [DB-21024](https://yugabyte.atlassian.net/browse/DB-21024) Original commit: 1cbc75d9f0cec24394608319719cfb8e073431f0 / D52170 Test Plan: ./yb_build.sh --cxx-test master_xrepl-test --gtest_filter 'MasterTestXRepl.TestCreateCDCGrpcStream' ./yb_build.sh --cxx-test cdcsdk_ysql-test --gtest_filter 'CDCSDKYsqlTest.TestBackfillOfLegacyGrpcStream' ./yb_build.sh --cxx-test cdcsdk_ysql-test --gtest_filter 'CDCSDKYsqlTest.TestRecordTypeOptionPresenceForStreams' ./yb_build.sh --cxx-test cdcsdk_ysql-test --gtest_filter 'CDCSDKYsqlTest.TestBackfillOfNotificationsStreamPluginName' Reviewers: skumar, xCluster, hsunder, stiwary, sumukh.phalgaonkar, bakul.gupta, asrinivasan Reviewed By: sumukh.phalgaonkar Subscribers: ycdcxcluster, ybase Tags: #jenkins-ready Differential Revision: https://phorge.dev.yugabyte.com/D56095

[BACKPORT 2026.1.1][#31149] CDC: Allow gRPC stream creation via PG syntax Summary: #### Backport Description No merge conflicts were encountered. #### Original Description ##### Code Changes summary This change lets gRPC CDC streams be created and managed through the standard PostgreSQL replication-slot interface. Such streams gets a `slot_name`, `yb_grpc` plugin_name. These streams are also populated with `replica_identity_map` in their metadata and a `slot entry` in cdc_state table. This change is gated behind an auto flag `FLAGS_cdc_pg_create_grpc_stream` that turns on only after the cluster is finalised on a supporting version. The current behaviour of creating gRPC streams via `yb-admin` cmd is still available but is discouraged to use via a warning. gRPC streams created via such method also get the `slot_name` and `yb_grpc` plugin name in their metadata and `slot entry` in cdc_state table. However, such streams still will rely on `record_type` instead of `replica_identity_map`. Thus, in general we can have 3 types of gRPC streams once the upgrade of a universe is finalized. 1. **Old streams**: - These are streams which were either already present in universe before upgrade or the streams which got created after upgrade but before upgrade-finalization. - They don't have `slot_name`, `plugin_name`, `replica_identity_map` in their metadata and `slot_entry` in cdc_state table. 2. **New streams created via yb-admin**: - Streams being created post upgrade-finalization comes under this category. - Such streams will have `slot_name`, `plugin_name` in their metadata and `slot_entry` in cdc_state table. They won't have `replica_identity_map` but would rely on `record_type` passed in yb-admin cmd. 3. **New streams created via PG syntax** - These are also streams which are being created post upgrade-finalization. If a gRPC stream is attempted for creation via PG syntax before upgrade, then it is errored out. - Such streams will have `slot_name`, `plugin_name` and `replica_identity_map` in their metadata and `slot_entry` in cdc_state table. They behave analogous to logical_replication_streams. In addition to changes to stream creation, this revision also adds a mechanism to backfill old gRPC streams with: - auto-generated `slot_name` (formatted as `grpc_<stream_id>`), - `yb_grpc` plugin_name, and - a `slot entry` in cdc_state table. Thus once the backfill completes, the `type 1` streams mentioned above become analogous to `type 2` streams. This revision also changes the plugin name from "" to `yboutput` for `yb_system` notifications streams (i.e for LISTEN/NOTIFY slots) since we are using below as invariant for gRPC streams: - if there's no plugin_name field set in create stream request or in persisted data. - if present, plugin_name should be empty or should be `yb_grpc`. Along with this, it also backfill the plugin name to `yboutput` for existing notification streams. > The original revision was raised by **Bakul Gupta**. This is being commandeered by me after his internship completion. ##### Upgrade/rollback safety considerations: Feature is gated by the AutoFlag `FLAGS_cdc_pg_create_grpc_stream`. Creation via PG syntax and the legacy-gRPC backfill take effect only after promotion at upgrade-finalization, so the change is rollback-safe while the flag is unpromoted. **Proto change:** the only wire-format change is a new field `optional bool is_notification_slot` in `CDCStreamInfoPB` (the `GetCDCStreamResponsePB.stream()` message) which is present in master. It is upgrade/rollback safe because even if the nodes which are not upgraded yet won't have any discrepancy (wrt to their existing code) to differentiate between the notification slots and others. The existing code used to rely on replication slot name for checking logical replication streams. This old logic will clearly classify the notification streams as logical replication. ##### Considerations for colocated tables: NA ##### Considerations for connector: Streams created via PG syntax do not carry the `record_type` option (they use `replica_identity_map`). The implications of it for the gRPC (Debezium) connector are: 1. The connector's `LogicalDecoder` enum must recognize `yb_grpc` so that `plugin.name=yb_grpc` is accepted and routed to the yb-proto decoder. The [latest released connector](https://github.com/yugabyte/debezium-connector-yugabytedb/releases/tag/vdz.1.9.5.yb.grpc.2026.1) **will not work** with such streams as its `LogicalDecoder` enum has no `yb_grpc` value, so config validation rejects `plugin.name=yb_grpc` and the connector fails to start. - We have an ongoing [PR](https://github.com/yugabyte/debezium-connector-yugabytedb/pull/409) for it. 2. Consuming a stream that omits `record_type` also requires treating such missing `record_type`. The [PR](https://github.com/yugabyte/debezium-connector-yugabytedb/pull/402) already handles it and is available in [latest released connector](https://github.com/yugabyte/debezium-connector-yugabytedb/releases/tag/vdz.1.9.5.yb.grpc.2026.1). Streams created via `yb-admin` still carry `record_type`, so they keep working with existing connectors. Jira: [DB-21024](https://yugabyte.atlassian.net/browse/DB-21024) Original commit: 1cbc75d9f0cec24394608319719cfb8e073431f0 / D52170 Test Plan: ./yb_build.sh --cxx-test master_xrepl-test --gtest_filter 'MasterTestXRepl.TestCreateCDCGrpcStream' ./yb_build.sh --cxx-test cdcsdk_ysql-test --gtest_filter 'CDCSDKYsqlTest.TestBackfillOfLegacyGrpcStream' ./yb_build.sh --cxx-test cdcsdk_ysql-test --gtest_filter 'CDCSDKYsqlTest.TestRecordTypeOptionPresenceForStreams' ./yb_build.sh --cxx-test cdcsdk_ysql-test --gtest_filter 'CDCSDKYsqlTest.TestBackfillOfNotificationsStreamPluginName' Reviewers: skumar, xCluster, hsunder, stiwary, sumukh.phalgaonkar, bakul.gupta, asrinivasan Reviewed By: sumukh.phalgaonkar Subscribers: ybase, ycdcxcluster Tags: #jenkins-ready Differential Revision: https://phorge.dev.yugabyte.com/D56096

2026-07-24

[#31149] CDC: Allow gRPC stream creation via PG syntax Summary: ##### Code Changes summary This change lets gRPC CDC streams be created and managed through the standard PostgreSQL replication-slot interface. Such streams gets a `slot_name`, `yb_grpc` plugin_name. These streams are also populated with `replica_identity_map` in their metadata and a `slot entry` in cdc_state table. This change is gated behind an auto flag `FLAGS_cdc_pg_create_grpc_stream` that turns on only after the cluster is finalised on a supporting version. The current behaviour of creating gRPC streams via `yb-admin` cmd is still available but is discouraged to use via a warning. gRPC streams created via such method also get the `slot_name` and `yb_grpc` plugin name in their metadata and `slot entry` in cdc_state table. However, such streams still will rely on `record_type` instead of `replica_identity_map`. Thus, in general we can have 3 types of gRPC streams once the upgrade of a universe is finalized. 1. **Old streams**: - These are streams which were either already present in universe before upgrade or the streams which got created after upgrade but before upgrade-finalization. - They don't have `slot_name`, `plugin_name`, `replica_identity_map` in their metadata and `slot_entry` in cdc_state table. 2. **New streams created via yb-admin**: - Streams being created post upgrade-finalization comes under this category. - Such streams will have `slot_name`, `plugin_name` in their metadata and `slot_entry` in cdc_state table. They won't have `replica_identity_map` but would rely on `record_type` passed in yb-admin cmd. 3. **New streams created via PG syntax** - These are also streams which are being created post upgrade-finalization. If a gRPC stream is attempted for creation via PG syntax before upgrade, then it is errored out. - Such streams will have `slot_name`, `plugin_name` and `replica_identity_map` in their metadata and `slot_entry` in cdc_state table. They behave analogous to logical_replication_streams. In addition to changes to stream creation, this revision also adds a mechanism to backfill old gRPC streams with: - auto-generated `slot_name` (formatted as `grpc_<stream_id>`), - `yb_grpc` plugin_name, and - a `slot entry` in cdc_state table. Thus once the backfill completes, the `type 1` streams mentioned above become analogous to `type 2` streams. This revision also changes the plugin name from "" to `yboutput` for `yb_system` notifications streams (i.e for LISTEN/NOTIFY slots) since we are using below as invariant for gRPC streams: - if there's no plugin_name field set in create stream request or in persisted data. - if present, plugin_name should be empty or should be `yb_grpc`. Along with this, it also backfill the plugin name to `yboutput` for existing notification streams. > The original revision was raised by **Bakul Gupta**. This is being commandeered by me after his internship completion. ##### Upgrade/rollback safety considerations: Feature is gated by the AutoFlag `FLAGS_cdc_pg_create_grpc_stream`. Creation via PG syntax and the legacy-gRPC backfill take effect only after promotion at upgrade-finalization, so the change is rollback-safe while the flag is unpromoted. **Proto change:** the only wire-format change is a new field `optional bool is_notification_slot` in `CDCStreamInfoPB` (the `GetCDCStreamResponsePB.stream()` message) which is present in master. It is upgrade/rollback safe because even if the nodes which are not upgraded yet won't have any discrepancy (wrt to their existing code) to differentiate between the notification slots and others. The existing code used to rely on replication slot name for checking logical replication streams. This old logic will clearly classify the notification streams as logical replication. ##### Considerations for colocated tables: NA ##### Considerations for connector: Streams created via PG syntax do not carry the `record_type` option (they use `replica_identity_map`). The implications of it for the gRPC (Debezium) connector are: 1. The connector's `LogicalDecoder` enum must recognize `yb_grpc` so that `plugin.name=yb_grpc` is accepted and routed to the yb-proto decoder. The [latest released connector](https://github.com/yugabyte/debezium-connector-yugabytedb/releases/tag/vdz.1.9.5.yb.grpc.2026.1) **will not work** with such streams as its `LogicalDecoder` enum has no `yb_grpc` value, so config validation rejects `plugin.name=yb_grpc` and the connector fails to start. - We have an ongoing [PR](https://github.com/yugabyte/debezium-connector-yugabytedb/pull/409) for it. 2. Consuming a stream that omits `record_type` also requires treating such missing `record_type`. The [PR](https://github.com/yugabyte/debezium-connector-yugabytedb/pull/402) already handles it and is available in [latest released connector](https://github.com/yugabyte/debezium-connector-yugabytedb/releases/tag/vdz.1.9.5.yb.grpc.2026.1). Streams created via `yb-admin` still carry `record_type`, so they keep working with existing connectors. Jira: [DB-21024](https://yugabyte.atlassian.net/browse/DB-21024) Test Plan: ./yb_build.sh --cxx-test master_xrepl-test --gtest_filter 'MasterTestXRepl.TestCreateCDCGrpcStream' ./yb_build.sh --cxx-test cdcsdk_ysql-test --gtest_filter 'CDCSDKYsqlTest.TestBackfillOfLegacyGrpcStream' ./yb_build.sh --cxx-test cdcsdk_ysql-test --gtest_filter 'CDCSDKYsqlTest.TestRecordTypeOptionPresenceForStreams' ./yb_build.sh --cxx-test cdcsdk_ysql-test --gtest_filter 'CDCSDKYsqlTest.TestBackfillOfNotificationsStreamPluginName' Reviewers: skumar, xCluster, hsunder, stiwary, sumukh.phalgaonkar, bakul.gupta, asrinivasan Reviewed By: sumukh.phalgaonkar Subscribers: ybase, ycdcxcluster Tags: #jenkins-ready Differential Revision: https://phorge.dev.yugabyte.com/D52170

2026-07-23

[#32232] DocDB: Add support for disk selection within a storage tier Summary: Adds tier-aware disk selection on the tablet server so new tablets can land on the correct storage tier (home / `path_id` 0) and future tier migrations can resolve a target `path_id` within a tier. **Tablet creation (CreateTabletRequestPB, TabletServiceAdmin, TSTabletManager::CreateNewTablet)** - Adds optional `target_storage_tier` (e.g. "ssd", "hdd") on `CreateTabletRequestPB`, intended to come from the tablespace’s `storage_tier`. - Plumbs it into `GetAndRegisterDataAndWalDir` so data root selection considers only disks tagged with that tier via `FsManager::GetDataRootDirsForTier`, using the same per-table then per-drive least-loaded policy as today. - WAL directory selection stays tier-agnostic (WAL roots are not part of `tier_paths`). - If the requested tier has no disks on this node, logs a warning and falls back to the default all-disk policy instead of failing tablet creation. **Within-tier placement (SelectPathIdForTier)** - New helper that picks the least-loaded data root in a given tier and maps it back to the tablet’s `path_id` using `RaftGroupMetadata::tier_paths()`. - Intended for follow-up work (e.g. `AlterTabletTier` + `light_weight_compact`) when moving SSTs to another disk/tier. **Upgrade/Rollback safety:** - This diff adds `target_storage_tier` as a new optional field on `CreateTabletRequestPB`. - New binary, old peer: field is unset -> empty tier -> unchanged disk selection as today. - Old binary, new peer: old tservers ignore the new optional field; tablets are still created with legacy placement until the cluster is fully upgraded. What Test/Preview/AutoFlag is used to guard the feature? - No AutoFlag and no preview flag. - Behavior is gated by `target_storage_tier` on tablet create (non-empty -> tier-filtered data root selection) and by `FsManager` tier labels from `--fs_data_dirs`. Empty/unset tier or missing tier on the node preserves today’s selection behavior. Test Plan: Following Unit Tests added: `TsTabletManagerTieredDriveTest.SelectPathIdForTierPicksCorrectTier` — SelectPathIdForTier returns a path_id on the requested tier; unknown tier returns NotFound. ` TsTabletManagerTieredDriveTest.SelectPathIdForTierBalancesWithinTier` — within-tier picker prefers the less-loaded HDD after artificial skew. Also ran: ``` ./yb_build.sh release --cxx-test ts_tablet_manager-test --sj --skip-pg-parquet --no-odyssey --no-ybc 2>&1 | tee /tmp/test-ts-tablet-manager.log ``` Reviewers: mhaddad, timur Reviewed By: mhaddad Subscribers: ybase Differential Revision: https://phorge.dev.yugabyte.com/D55535

2026-07-15

[#32228] DocDB: Add Support for multi paths in rocksdb Summary: Adds groundwork for multi-path tiered storage in DocDB: per-tablet tier directories persisted in the tablet superblock, and RocksDB configured to be aware of all available disks at startup. - **Superblock & Metadata** (`metadata.proto`, `KvStoreInfo`): adds `tier_paths` to persist storage tier pins (path ID, tier name, absolute path); synthesizes a default `"home"` path when loading an older, single-tier superblock. - **RocksDB Initialization** (`Tablet::OpenRegularDB`): populates `DBOptions::db_paths` from the stored tier paths; sets `target_size = UINT64_MAX` for all slots to disable RocksDB's default capacity-based spilling. - **Remote Bootstrap (RBS)**: clears source `tier_paths` on the destination node to avoid copying invalid physical directory paths; destination falls back to a synthesized single home tier (multi-path aware checkpointing is a follow-up task). - **RocksDB Fork**: adds tests validating packing/unpacking of the new 5-bit `path_id` (`kMaxPathId = 31`). **Upgrade/Rollback safety:** Describe how this change handles upgrade and rollback of YugabyteDB. - New format: `tier_paths` is a new optional repeated field on `KvStoreInfoPB`. Old binaries ignore it and keep using `rocksdb_dir` only. - New binary on old superblock: empty `tier_paths` is made into one `home` entry; no migration needed. - Old binary on new superblock: ignores `tier_paths`, opens with legacy single-path behavior. Safe as long as no SSTs have been moved off `home`, this change doesn't move any, so pre-tiered rollback remains safe. - Config / behavior: single-tier or non-tiered `--fs_data_dirs` clusters get one synthesized/one-tier entry, matching today's behavior. Home disk selection is unchanged (tier-blind), tier-aware placement is also a follow up task to be done. What Test/Preview/AutoFlag is used to guard the feature? Feature guard: No AutoFlag. Gated by presence of `tier_paths` on the superblock and a non-empty tier layout from `FsManager`; missing/empty `tier_paths` preserves legacy single-home behavior. Test Plan: Added the following unit test cases: - `NewTabletOnMultiDriveHasAllTierPaths` — Creating a new tablet on a multi-drive tserver (SSD + two HDDs) records one `tier_paths` entry per disk, with home on SSD at `path_id` 0, and persists those paths through reload and on the on-disk superblock. - `TierPathsUpdatedOnDestination` — Remote-bootstrapping a tablet onto a new node rewrites tier_paths to the destination node’s layout (not the source’s), and the persisted superblock matches in-memory metadata after `FetchAll`. - `MigrateOldSuperblockOnMultiDrivePopulatesAllDisks` — Loading a tablet on a multi-drive tserver when the on-disk superblock has no tier_paths (pre–tiered-storage) repopulates an entry for every disk and persists the upgrade. Ran unit tests using the following commands: ``` # Rebuild what you touched (skip if you just built) ./yb_build.sh release --cxx-test tablet-metadata-test --sj --skip-pg-parquet --no-odyssey --no-ybc 2>&1 | tee /tmp/test-tablet-metadata.log ./yb_build.sh release --cxx-test version_edit_test --sj --skip-pg-parquet --no-odyssey --no-ybc 2>&1 | tee /tmp/test-version-edit.log ./yb_build.sh release --cxx-test fs_manager-test --sj --skip-pg-parquet --no-odyssey --no-ybc 2>&1 | tee /tmp/test-fs-manager.log ./yb_build.sh release --cxx-test remote_bootstrap_rocksdb_client-test --sj --skip-pg-parquet --no-odyssey --no-ybc 2>&1 | tee /tmp/test-rbs-client.log ``` Did some manual testing: ``` ./build/latest/bin/yb-master \ --master_addresses=127.0.0.1:7100 --rpc_bind_addresses=127.0.0.1:7100 \ --fs_data_dirs=/tmp/ybtier/master --webserver_port=7000 --replication_factor=1 \ --placement_cloud=cloud1 --placement_region=r1 --placement_zone=z1 \ > /tmp/ybtier/master.out 2>&1 & ``` ``` ./build/latest/bin/yb-tserver \ --tserver_master_addrs=127.0.0.1:7100 --rpc_bind_addresses=127.0.0.1:9100 \ --webserver_port=9000 --pgsql_proxy_bind_address=127.0.0.1:5433 \ --fs_data_dirs=/tmp/ybtier/ssd1:ssd,/tmp/ybtier/ssd2:ssd,/tmp/ybtier/ssd3:ssd,/tmp/ybtier/hdd1:hdd,/tmp/ybtier/hdd2:hdd,/tmp/ybtier/hdd3:hdd \ --fs_wal_dirs=/tmp/ybtier/wal \ --placement_cloud=cloud1 --placement_region=r1 --placement_zone=z1 \ --logtostderr --minloglevel=0 > /tmp/ybtier/tserver.out 2>&1 & ``` ``` ./bin/ysqlsh -h 127.0.0.1 -p 5433 -c \ "CREATE TABLE t (k int primary key, v text); \ INSERT INTO t SELECT g,'x' FROM generate_series(1,1000) g;" ``` Screenshot of output showing rocksdb supporting multiple paths: {F517554} Reviewers: mhaddad, arybochkin, timur, sergei Reviewed By: mhaddad Subscribers: ybase Differential Revision: https://phorge.dev.yugabyte.com/D55203

2026-07-09

[BACKPORT 2026.1][#32321] DocDB: Postpone tablet split until vector index backfill completes Summary: During vector index creation the reverse mapping records are written into the tablet's regular RocksDB as part of the tserver-side backfill. This growth can push the tablet past the split threshold, and because the vector index backfill runs on the tserver (the master skips its backfill orchestration) the master's existing `table->IsBackfilling()` gate does not cover it. As a result an automatic tablet split could start while the backfill was still running. Report a new `has_active_vector_index_backfill` flag from the tserver to the master, mirroring `may_have_orphaned_post_split_data`. `TabletVectorIndexes::HasActiveBackfill` returns true while any vector index on the tablet has not finished backfilling. `CatalogManager::ShouldSplitValidCandidate` now rejects a tablet whose leader reports an active vector index backfill, so the split is postponed until the backfill completes. Manual splits are unaffected, consistent with the existing checks in that method. **Upgrade/Rollback safety:** Added new field for new functionality. --- _automated · Claude Code (Opus 4.8)_ Original commit: 5adc7e22a8ca215c9c71c913a2feafa77d5a4345 / D55282 Test Plan: ./yb_build.sh debug --cxx-test pg_vector_index-test --gtest_filter 'PgDistributedVectorIndexTest.AutoSplitDuringBackfill/None' Reviewers: arybochkin, hsunder Reviewed By: arybochkin Subscribers: yql, ybase Tags: #jenkins-ready Differential Revision: https://phorge.dev.yugabyte.com/D55398

[BACKPORT 2026.1][#32360] DocDB: Vector index: add a new gflag and table parameter for vector reverse mapping ownership Summary: The change adds master-side gflag `enable_table_owned_vector_reverse_mapping` (default false), `owns_vector_reverse_mapping` on Schema::TableProperties, backup/restore fixup for that property, and plumbing of the flag/property into CompactionSchemaInfo and vector index backfill. The new parameter is set only during table creation and cannot be changed later. Additionally, `skip_reverse_mapping_backfill` is removed; backfill now follows the indexed table's `owns_vector_reverse_mapping` instead. Insert/update changes for vector reverse mapping ownership and compaction behavior are deferred to a follow-up revisions. Original commit: b61cc9f8c154273e2b0c62f739018a6ae4c11a56 / D54966 **Upgrade/Rollback safety:** Guarded by master runtime gflag `enable_table_owned_vector_reverse_mapping`. Test Plan: ./yb_build.sh --cxx-test pg_vector_index-test --gtest_filter=PgVectorIndexUtilTest.BackfillSkipsReverseMapping ./yb_build.sh --cxx-test pg_vector_index-test --gtest_filter=PgVectorIndexUtilTest.BackfillWritesReverseMapping ./yb_build.sh --cxx-test pg_vector_index-test --gtest_filter=PgVectorIndexUtilTest.NumTopVectorsToRemoveExceedsResultEntries ./yb_build.sh --cxx-test yb-backup-cross-feature-test --gtest_filter=YBBackupTest.TestYSQLTableOwnedVectorReverseMapping Reviewers: sergei, hsunder Reviewed By: hsunder Subscribers: ybase, yql Differential Revision: https://phorge.dev.yugabyte.com/D55064

[BACKPORT 2026.1][#31542] docdb: Vector Index: Chunked compaction implementation Summary: The change introduces a chunked compaction. It allows to not build one in-memory merged index for all input vectors but to output into multiple chunk files, each bounded by a configurable memory budget, to reduce OOM risk. If the limit is very small, compaction still produces at least one vector per output chunk. New gflag `vector_index_compaction_chunk_max_mem_store_size_mb` (runtime, default 0) is introduced and should be greater than 0 to enable chunked compaction. Original commit: 2fd12083996acf2f8a481729d25ac8bcf1f76fb8 / D54283 **Upgrade/Rollback safety:** Just a comment update in .proto file. Test Plan: ./yb_build.sh --cxx-test ann_methods_vector_lsm-test --gtest_filter VectorLSMTest.ChunkedCompactionRespectsMemStoreLimit/kHnswlib ./yb_build.sh --cxx-test ann_methods_vector_lsm-test --gtest_filter VectorLSMTest.ChunkedCompactionRespectsMemStoreLimit/kUsearch ./yb_build.sh --cxx-test ann_methods_vector_lsm-test --gtest_filter VectorLSMTest.OpenAfterChunkedCompaction/kHnswlib ./yb_build.sh --cxx-test ann_methods_vector_lsm-test --gtest_filter VectorLSMTest.OpenAfterChunkedCompaction/kUsearch ./yb_build.sh --cxx-test ann_methods_vector_lsm-test --gtest_filter VectorLSMTest.DefaultCompactionMergesMultipleChunks/kHnswlib ./yb_build.sh --cxx-test ann_methods_vector_lsm-test --gtest_filter VectorLSMTest.DefaultCompactionMergesMultipleChunks/kUsearch Reviewers: sergei, zdrudi Reviewed By: zdrudi Subscribers: ybase Differential Revision: https://phorge.dev.yugabyte.com/D55381

2026-07-08

[#32229,#32352] DocDB: Add support for labeled disks in fs_data_dirs Summary: Adds tiered-storage groundwork: labeled `--fs_data_dirs` with a fixed storage-class set `{ssd, hdd}`, default `ssd` for unlabeled paths, validation of explicit labels at tserver startup, and propagation of `storage_tier` on heartbeat `path_metrics` to the master. - `--fs_data_dirs`: tokens `path[:tier`] into path and tier; invalid explicit tiers fail init with a clear error; unlabeled paths are defaulted to `ssd`. - `FsManager`: `ValidStorageTiers()`, `IsValidStorageTier()`, tier maps and accessors; WAL defaulting still strips `:tier` from data dirs when building `--fs_wal_dirs`. - Master path: optional `path_metrics.storage_tier` in master proto; tserver heartbeat sets tier per data root; master stores/exposes it (including JSON on tablet-servers API). **Upgrade/Rollback safety:** Describe how this change handles upgrade and rollback of YugabyteDB. - New format: `storage_tier` is an optional field on `PathMetrics`. Old masters ignore unknown fields; old tservers do not send it. - New tservers on old masters: heartbeats remain valid; tier may not appear until master is upgraded. - New master on old tservers: path metrics without `storage_tier` are treated as empty on the master side (existing JSON/API checks skip empty tier). - Config / behavior: Clusters that never use `:tier` behave as before; all data dirs are classified as `ssd`. - Rollback to a binary without tier parsing: use plain paths only (no `:tier` suffixes). What Test/Preview/AutoFlag is used to guard the feature? Feature guard: No AutoFlag; behavior is backward compatible via optional proto + default tier for unlabeled disks. Explicit invalid tiers are rejected only on new binaries that run tier validation. Test Plan: Ran unit tests using following command: ``` ./yb_build.sh release --cxx-test fs_manager-test \ --gtest_filter 'StorageTierTest.*' \ --sj --skip-pg-parquet --no-odyssey --no-ybc ``` Integration test command: ``` ./yb_build.sh release --cxx-test tserver_metrics_heartbeat_data_provider-itest \ --gtest_filter 'TServerMetricsHeartbeatDataProviderITest.PathMetricsCarryStorageTier' \ --sj --skip-pg-parquet --no-odyssey --no-ybc ``` Manual testing: Ran the following commands on terminals. Teminal 1 (master): ``` ./build/latest/bin/yb-master \ --master_addresses=127.0.0.1:7100 \ --rpc_bind_addresses=127.0.0.1:7100 \ --fs_data_dirs=/tmp/yb_master_data \ --replication_factor=1 ``` Terminal 2 (tserver): ``` ./build/latest/bin/yb-tserver \ --tserver_master_addrs=127.0.0.1:7100 \ --rpc_bind_addresses=127.0.0.1:9100 \ --fs_data_dirs=/tmp/yb_data_ssd:ssd,/tmp/yb_data_hdd:hdd,/tmp/yb_data:samson_tier \ --fs_wal_dirs=/tmp/yb_data_nvme ``` Terminal 3 (curl master to see output): ``` curl -s http://127.0.0.1:7000/api/v1/tablet-servers | python3 -m json.tool ``` Screenshot of output: - Right terminal shows error message for having a tier label that is NOT in `{ssd, hdd}`. - Left terminal shows output of the curl, which includes the newly added `storage_tier` field. {F513874} Reviewers: mhaddad, anijhawan Reviewed By: mhaddad Subscribers: svc_phabricator, ybase Differential Revision: https://phorge.dev.yugabyte.com/D54935

[BACKPORT 2025.2.4][#32353] DocDB: Proactively mark leader blacklisted tservers as followers in MetaCache Summary: YBA leader blacklists tservers before restarting each tserver in the process of a rolling restart. Though the leaders have moved away from this node, the other tservers operating on a stale meta-cache view would still try hitting the down tserver for reads/writes if they haven't already noticed that the leader has been moved away. It isn't much of a problem in the usual case since the node might RST the packet and the tserver would move on to try the other replicas before going to the master to re-fetch the tablet locations. But in case of an ungraceful termination/shutdown of the node (or in a kubernetes environment), the connection could keep blackholing and be kept alive for `rpc_connection_timeout_ms` (defaults to 15s). This might affect op latencies which is expected to some extent in an ungraceful shutdown, but isn't really ok on a planned one. Since the latencies get affected for even a planned shutdown in a kubernetes env, it is better it the other tservers proactively mark the tserver to go down as a follower and not route read/write traffic to it (It could still route reads if follower reads is on though). Changes 1. This revision addresses the above issue by propagating the leader blacklisted tservers info on the master -> tserver heartbeat response once it sees the leader count has dropped to 0 (for the blacklisted tsevrer). Tservers mark these leader blacklisted tservers (with 0 leaders) as followers and hence wouldn't route read/write requests to them. When the leader blacklisted tserver gets un-blacklisted though, we don't explicitly reset this info and let the meta-cache figure it out eventually when a follower peer sends the latest consensus info or when the meta-cache ends up going to the master. This propagation is done at most once per tserver per leader blacklist addition. 2. Incase the leader load on the leader blacklisted tservers is 0 while responding to `GetLoadMoveCompletionPercent`, delay until master has seen a heartbeat from all live tservers. This is being done as best effort for the other live tservers to mark the leader blacklisted tserver as follower. Both the above changes are protected under new gflags which default to true (optimizations enabled by default). **Upgrade/Downgrade safety** Added new `leader_blacklisted_tservers_with_no_leaders` to the tserver-master heartbeat message. The tserver checks for the size of the field before trying to access the repeated field, so the opeartion is safe ic case of upgrades/downgrades/ and mixed mode operations. This filed is purely used for an optimization as mentioned above. Note: We could also do the same for blacklisted tservers, and execute `MetaCache::MarkTSFailed`. This would be useful on cluster scale in/scale out activities where requests are being sent to followers (which is rare, and hence covering just the leader case here). Original commit: 5292a99c4417137a2b306ccd65447701ec301168 / D54742 Test Plan: Jenkins Manually tested the following scenario on a local rf3 cluster: 1. `create table test(k int) split into 10 tablets;` 2. set vmodule on `tablet_rpc=1` 3. randomly execut eleader stepdown using `yb-admin --master_addresses=127.0.0.1:7100,127.0.0.2:7100.127.0.0.3:7100 leader_stepdown <tablet>` 4. `insert into test select i from generate_series(1, 10) as i;` and notice the following logs on the tserver hosting the connection ``` I0618 23:14:27.320129 1333856 tablet_rpc.cc:312] vlog1: Not the leader for Write(tablet: 6f37f80451b14092b7af7cee4efa39b5, num_ops: 4, num_attempts: 1, txn: 152ea27d-b9a9-4f0c-9371-87b3d7ca5fc8, subtxn: [none]) retrying with a different replica I0618 23:14:27.322331 1334100 tablet_rpc.cc:312] vlog1: Not the leader for Write(tablet: 5cb9fe88fa8a4ff69f7effa2e8aabcaa, num_ops: 2, num_attempts: 1, txn: 152ea27d-b9a9-4f0c-9371-87b3d7ca5fc8, subtxn: [none]) retrying with a different replica ``` 5. now blacklist one of the tservers using `yb-admin --master_addresses=127.0.0.1:7100,127.0.0.2:7100.127.0.0.3:7100 change_leader_blacklist ADD <ip>:9100` 6. `insert into test select i from generate_series(1, 10) as i;` and notice we don't see any complaints of `Not the leader for Write` (assuming the write is issue after 1s, that is, after the tserver processes the master heartbeat resp). Reviewers: amitanand, neera.mital, mlillibridge, #db-approvers, hsunder Reviewed By: amitanand, #db-approvers, hsunder Subscribers: svc_phabricator, ybase Differential Revision: https://phorge.dev.yugabyte.com/D55354

[#32321] DocDB: Postpone tablet split until vector index backfill completes Summary: During vector index creation the reverse mapping records are written into the tablet's regular RocksDB as part of the tserver-side backfill. This growth can push the tablet past the split threshold, and because the vector index backfill runs on the tserver (the master skips its backfill orchestration) the master's existing `table->IsBackfilling()` gate does not cover it. As a result an automatic tablet split could start while the backfill was still running. Report a new `has_active_vector_index_backfill` flag from the tserver to the master, mirroring `may_have_orphaned_post_split_data`. `TabletVectorIndexes::HasActiveBackfill` returns true while any vector index on the tablet has not finished backfilling. `CatalogManager::ShouldSplitValidCandidate` now rejects a tablet whose leader reports an active vector index backfill, so the split is postponed until the backfill completes. Manual splits are unaffected, consistent with the existing checks in that method. **Upgrade/Rollback safety:** Added new field for new functionality. --- _automated · Claude Code (Opus 4.8)_ Test Plan: ./yb_build.sh debug --cxx-test pg_vector_index-test --gtest_filter 'PgDistributedVectorIndexTest.AutoSplitDuringBackfill/None' Reviewers: arybochkin, hsunder Reviewed By: arybochkin Subscribers: ybase, yql Tags: #jenkins-ready Differential Revision: https://phorge.dev.yugabyte.com/D55282

2026-07-03

[BACKPORT 2025.2.4][#31930,#31241] docdb: Fix RBS vs ConfigChange races Summary: There is a possibility of RBS vs ConfigChange races, for example: 1. Raft config for the tablet has nodes A (leader), B, C. 2. D is added to Raft config, CHANGE_CONFIG operation is committed and applied on A, B, C 3. RBS A -> D started 4. D is removed from Raft config, CHANGE_CONFIG operation is committed and applied on leader 5. RBS A -> D downloads WAL and completed, D has the latest committed Raft config **Expected result:** orphaned tablet replica on D should be deleted. **Actual result:** we have an orphaned tablet replica (lagging follower) on D that is not a part of Raft group (which has 3 peers) and therefore is not receiving consensus updates from leader. In order to fix that, behaviour inside `MasterHeartbeatServiceImpl::ProcessTabletReport` is changed to delete a tablet replica which is no longer part of the committed Raft config and Raft config that added that replica is no longer pending (either committed or aborted). The latter condition avoids deleting the newly RBSed tablet replica that is being added to the tablet Raft group. The following logic is implemented to support that: 1. Once tablet leader decides to start RBS replica on another tserver, it will include the current pending Raft config op id (both term and index) into `StartRemoteBootstrapRequestPB` or empty op id when no config change is pending. 2. The bootstrapping replica persists this op id in its consensus metadata as `pending_config_op_id_from_rbs`. It is cleared once the replica's last committed op id either advances to a higher term, or its index reaches/passes the stored pending op id's index - i.e. once the original `CHANGE_CONFIG` operation can no longer be pending (it has either committed or been aborted). 3. `ReportedTabletPB::pending_config_op_id` is added to tserver->master heartbeats. Its value is whichever is set: the replica's currently active pending config op id (a config change in progress on the replica itself) or the pending_config_op_id_from_rbs from (2). 4. When master leader receives a tablet report from a replica that is *not* in the committed Raft config last known by the master, the master triggers `DeleteTabletRequestPB` to that replica if both: a. The reported committed Raft config op id index is <= the committed Raft config op id index last known by master for this tablet. b. Either the report carries no pending op id (empty/missing), or the master can prove it is no longer pending: if the pending op id's term is strictly less than the `current_term` in the master's last-known committed consensus state for this tablet, that `CHANGE_CONFIG` must have already been aborted or committed, and condition (4.a) alone is sufficient. 5. TServer will include tablet in next heartbeat in case DeleteTablet failed due to stale cas_config_opid_index_less_or_equal (this scenario is covered by TabletReplacementITest.TombstoneEvictedReplicaWithRbsAndConfigChangeRaceAndMasterRestart). Also update master-side of CloneTablet operation to seed the target tablet's committed_consensus_state peers on the master from the source tablet's config so that when the cloned replicas heartbeat back, `ProcessTabletReportBatch` sees them as part of the expected Raft config and does not tombstone them. Added several unit-tests for other RBS vs ConfigChange race scenarios. They are also fixed by the implemented change. Also renamed `RaftConfigPB.opid_index` to `committed_op_index` in order to reflect the actual purpose of this field. **Upgrade/Rollback safety:** New logic at master-side is gated by a new `use_tablet_report_pending_config_op_id` auto flag. Until the whole cluster is fully upgraded, master won't rely on newly added `ReportedTabletPB::pending_config_op_id` field. Original commit: 54c3d4ca73c593b364abf3f9462e788d32a20f53 / D52759 Test Plan: Run the following tests for asan/tsan/debug/relelase for 20 iterations each: - TabletSplitITest.SplitWithParentTabletRbsFromFollower - TabletReplacementITest.TombstoneEvictedReplicaWithRbsAndConfigChangeRace - TabletReplacementITest.TombstoneEvictedReplicaWithRbsAndConfigChangeRaceAndMasterRestart - TabletReplacementITest.TombstoneEvictedReplicaAfterAbortedAddServer - covers https://github.com/yugabyte/yugabyte-db/issues/31241 - TabletReplacementITest.DontDeleteNewReplicaInPendingConfig - TabletReplacementITest.DontDeleteNewReplicaInPendingConfigAfterRbsFromFollowerRf5 Reviewers: zdrudi Reviewed By: zdrudi Subscribers: ybase Tags: #jenkins-ready Differential Revision: https://phorge.dev.yugabyte.com/D55227

[BACKPORT 2026.1.0][#31930,#31241] docdb: Fix RBS vs ConfigChange races Summary: There is a possibility of RBS vs ConfigChange races, for example: 1. Raft config for the tablet has nodes A (leader), B, C. 2. D is added to Raft config, CHANGE_CONFIG operation is committed and applied on A, B, C 3. RBS A -> D started 4. D is removed from Raft config, CHANGE_CONFIG operation is committed and applied on leader 5. RBS A -> D downloads WAL and completed, D has the latest committed Raft config **Expected result:** orphaned tablet replica on D should be deleted. **Actual result:** we have an orphaned tablet replica (lagging follower) on D that is not a part of Raft group (which has 3 peers) and therefore is not receiving consensus updates from leader. In order to fix that, behaviour inside `MasterHeartbeatServiceImpl::ProcessTabletReport` is changed to delete a tablet replica which is no longer part of the committed Raft config and Raft config that added that replica is no longer pending (either committed or aborted). The latter condition avoids deleting the newly RBSed tablet replica that is being added to the tablet Raft group. The following logic is implemented to support that: 1. Once tablet leader decides to start RBS replica on another tserver, it will include the current pending Raft config op id (both term and index) into `StartRemoteBootstrapRequestPB` or empty op id when no config change is pending. 2. The bootstrapping replica persists this op id in its consensus metadata as `pending_config_op_id_from_rbs`. It is cleared once the replica's last committed op id either advances to a higher term, or its index reaches/passes the stored pending op id's index - i.e. once the original `CHANGE_CONFIG` operation can no longer be pending (it has either committed or been aborted). 3. `ReportedTabletPB::pending_config_op_id` is added to tserver->master heartbeats. Its value is whichever is set: the replica's currently active pending config op id (a config change in progress on the replica itself) or the pending_config_op_id_from_rbs from (2). 4. When master leader receives a tablet report from a replica that is *not* in the committed Raft config last known by the master, the master triggers `DeleteTabletRequestPB` to that replica if both: a. The reported committed Raft config op id index is <= the committed Raft config op id index last known by master for this tablet. b. Either the report carries no pending op id (empty/missing), or the master can prove it is no longer pending: if the pending op id's term is strictly less than the `current_term` in the master's last-known committed consensus state for this tablet, that `CHANGE_CONFIG` must have already been aborted or committed, and condition (4.a) alone is sufficient. 5. TServer will include tablet in next heartbeat in case DeleteTablet failed due to stale cas_config_opid_index_less_or_equal (this scenario is covered by TabletReplacementITest.TombstoneEvictedReplicaWithRbsAndConfigChangeRaceAndMasterRestart). Also update master-side of CloneTablet operation to seed the target tablet's committed_consensus_state peers on the master from the source tablet's config so that when the cloned replicas heartbeat back, `ProcessTabletReportBatch` sees them as part of the expected Raft config and does not tombstone them. Added several unit-tests for other RBS vs ConfigChange race scenarios. They are also fixed by the implemented change. Also renamed `RaftConfigPB.opid_index` to `committed_op_index` in order to reflect the actual purpose of this field. **Upgrade/Rollback safety:** New logic at master-side is gated by a new `use_tablet_report_pending_config_op_id` auto flag. Until the whole cluster is fully upgraded, master won't rely on newly added `ReportedTabletPB::pending_config_op_id` field. Original commit: 54c3d4ca73c593b364abf3f9462e788d32a20f53 / D52759 Test Plan: Run the following tests for asan/tsan/debug/relelase for 20 iterations each: - TabletSplitITest.SplitWithParentTabletRbsFromFollower - TabletReplacementITest.TombstoneEvictedReplicaWithRbsAndConfigChangeRace - TabletReplacementITest.TombstoneEvictedReplicaWithRbsAndConfigChangeRaceAndMasterRestart - TabletReplacementITest.TombstoneEvictedReplicaAfterAbortedAddServer - covers https://github.com/yugabyte/yugabyte-db/issues/31241 - TabletReplacementITest.DontDeleteNewReplicaInPendingConfig - TabletReplacementITest.DontDeleteNewReplicaInPendingConfigAfterRbsFromFollowerRf5 Reviewers: zdrudi Reviewed By: zdrudi Subscribers: ybase Tags: #jenkins-ready Differential Revision: https://phorge.dev.yugabyte.com/D55224

[BACKPORT 2025.2.4][#27056] docdb: Fixed tablet split vs RBS from the follower race Summary: There is a possibility of tablet split vs RBS from follower race: 1. Parent tablet leader peers A-C accept a SPLIT_OP (op_id: 1.4). Leader (A) and 1st follower (B) apply SPLIT_OP, 2nd follower (C) doesn't apply it yet. 2. Parent tablet leader (node A) accepts CHANGE_CONFIG_OP (op_id: 1.5) to add a fourth peer (D) but doesn't apply it yet. 3. Parent tablet 2nd follower (C) still hasn't yet applied SPLIT_OP (1.4). 4. RBS for parent tablet peer (D) starts from the follower (C) and tablet metadata (tablet_data_state == TABLET_DATA_READY) is downloaded. 5. Parent tablet peers A-C completed applying the SPLIT_OP (1.4), child tablets have Raft config with 3 peers. 6. Parent tablet peers A-C apply CHANGE_CONFIG_OP (1.5) and now have committed Raft config with 4 peers. 7. Parent tablet peer D does local bootstrap and replays SPLIT_OP (1.4) as part of bootstrap. Due to tablet_data_state is TABLET_DATA_READY but not TABLET_DATA_SPLIT_COMPLETED replay does SPLIT_OP apply and creates child tablet peer. After that, 4th child tablet peer (D) is not a part of Raft group (which has 3 peers) and therefore is not receiving consensus updates from leader. This change fixes this race by rejecting RBS from the follower that is in progress of applying SPLIT_OP and RBS attempt will be retried later. Original commit: 78af3a208a6bfc005eedc0ed6e410d22f4d24758 / D48853 **Upgrade/Rollback safety:** New error code will be printed by old nodes as just number in case of RBS failure during upgrade but this is safe. Test Plan: TabletSplitITest.SplitWithParentTabletRbsFromFollower, TabletSplitITest.SplitWithParentTabletMove, RemoteBootstrapsFromNodeWithUncommittedSplitOp - 30 runs per each of asan/tsan/debug/release builds Jenkins: urgent Reviewers: arybochkin, #db-approvers Reviewed By: arybochkin, #db-approvers Subscribers: hbhanawat, svc_phabricator, ybase, zdrudi Tags: #jenkins-ready Differential Revision: https://phorge.dev.yugabyte.com/D55181

[BACKPORT 2026.1.0][#27056] docdb: Fixed tablet split vs RBS from the follower race Summary: There is a possibility of tablet split vs RBS from follower race: 1. Parent tablet leader peers A-C accept a SPLIT_OP (op_id: 1.4). Leader (A) and 1st follower (B) apply SPLIT_OP, 2nd follower (C) doesn't apply it yet. 2. Parent tablet leader (node A) accepts CHANGE_CONFIG_OP (op_id: 1.5) to add a fourth peer (D) but doesn't apply it yet. 3. Parent tablet 2nd follower (C) still hasn't yet applied SPLIT_OP (1.4). 4. RBS for parent tablet peer (D) starts from the follower (C) and tablet metadata (tablet_data_state == TABLET_DATA_READY) is downloaded. 5. Parent tablet peers A-C completed applying the SPLIT_OP (1.4), child tablets have Raft config with 3 peers. 6. Parent tablet peers A-C apply CHANGE_CONFIG_OP (1.5) and now have committed Raft config with 4 peers. 7. Parent tablet peer D does local bootstrap and replays SPLIT_OP (1.4) as part of bootstrap. Due to tablet_data_state is TABLET_DATA_READY but not TABLET_DATA_SPLIT_COMPLETED replay does SPLIT_OP apply and creates child tablet peer. After that, 4th child tablet peer (D) is not a part of Raft group (which has 3 peers) and therefore is not receiving consensus updates from leader. This change fixes this race by rejecting RBS from the follower that is in progress of applying SPLIT_OP and RBS attempt will be retried later. Original commit: 78af3a208a6bfc005eedc0ed6e410d22f4d24758 / D48853 **Upgrade/Rollback safety:** New error code will be printed by old nodes as just number in case of RBS failure during upgrade but this is safe. Test Plan: TabletSplitITest.SplitWithParentTabletRbsFromFollower, TabletSplitITest.SplitWithParentTabletMove, RemoteBootstrapsFromNodeWithUncommittedSplitOp - 30 runs per each of asan/tsan/debug/release builds Jenkins: urgent, all tests Reviewers: arybochkin, #db-approvers Reviewed By: arybochkin, #db-approvers Subscribers: hbhanawat, svc_phabricator, zdrudi, ybase Tags: #jenkins-ready Differential Revision: https://phorge.dev.yugabyte.com/D55184

2026-07-02

[#32335] YSQL: Pass single read time option from pggate to pg client session Summary: #### Problem Prior to this revision, multiple read time options may be set on PerformOptionsPB. It is unclear which read time option should be preferred. Example: - Parallel query may set ENSURE_READ_TIME_IS_SET read_time_manipulation. - GUC yb_read_after_commit_visibility=relaxed sets clamp_uncertainty_window option. Both options are set on the proto and the clamp option should be preferred. This is not obvious. #### Approach 1. Decide the preference in pggate itself. 2. Send only a single read time option in the perform RPC. This simplifies the contract between pggate and pg client session. Additionally: 1. Collect read time options logic into PgTxnManager::SetReadTimeOptions 2. Add appropriate validation checks for incompatible read time options. 3. Document this in pggate/README in section named Read point selection section. 4. Rename catalog session to legacy catalog session since plain sessions are used for catalog reads with newer concurrent DDL feature. 5. On similar lines, rename kDDL sessions to kAutonomousDDL. #### Some notes: 1. Follower reads are only applicable to read only transactions. And serializable never occurs in read only transactions. Therefore, follower reads and serializable do not co-occur. 2. Previously, parallel scans overwrote RESTART flag to ENSURE_READ_TIME_IS_SET. Instead, RESTART should take precedence because RESTART picks a read time on pg client session as well. 3. Previously, clamp and deferred options were ignored if any of the writes within the txn are non transactional. Now, the perform RPC must have a non transactional write to ignore clamp and deferred options. #### Upgrade/Rollback safety Perform RPC is intra node and therefore the client and the server are co-versioned. Test Plan: Jenkins ./yb_build.sh release --cxx-test pg_read_time-test Reviewers: pjain, smishra, sanketh, bkolagani Reviewed By: pjain Subscribers: yql, ybase Differential Revision: https://phorge.dev.yugabyte.com/D54921

[BACKPORT 2025.2][#32353] DocDB: Proactively mark leader blacklisted tservers as followers in MetaCache Summary: YBA leader blacklists tservers before restarting each tserver in the process of a rolling restart. Though the leaders have moved away from this node, the other tservers operating on a stale meta-cache view would still try hitting the down tserver for reads/writes if they haven't already noticed that the leader has been moved away. It isn't much of a problem in the usual case since the node might RST the packet and the tserver would move on to try the other replicas before going to the master to re-fetch the tablet locations. But in case of an ungraceful termination/shutdown of the node (or in a kubernetes environment), the connection could keep blackholing and be kept alive for `rpc_connection_timeout_ms` (defaults to 15s). This might affect op latencies which is expected to some extent in an ungraceful shutdown, but isn't really ok on a planned one. Since the latencies get affected for even a planned shutdown in a kubernetes env, it is better it the other tservers proactively mark the tserver to go down as a follower and not route read/write traffic to it (It could still route reads if follower reads is on though). Changes 1. This revision addresses the above issue by propagating the leader blacklisted tservers info on the master -> tserver heartbeat response once it sees the leader count has dropped to 0 (for the blacklisted tsevrer). Tservers mark these leader blacklisted tservers (with 0 leaders) as followers and hence wouldn't route read/write requests to them. When the leader blacklisted tserver gets un-blacklisted though, we don't explicitly reset this info and let the meta-cache figure it out eventually when a follower peer sends the latest consensus info or when the meta-cache ends up going to the master. This propagation is done at most once per tserver per leader blacklist addition. 2. Incase the leader load on the leader blacklisted tservers is 0 while responding to `GetLoadMoveCompletionPercent`, delay until master has seen a heartbeat from all live tservers. This is being done as best effort for the other live tservers to mark the leader blacklisted tserver as follower. Both the above changes are protected under new gflags which default to true (optimizations enabled by default). **Upgrade/Downgrade safety** Added new `leader_blacklisted_tservers_with_no_leaders` to the tserver-master heartbeat message. The tserver checks for the size of the field before trying to access the repeated field, so the opeartion is safe ic case of upgrades/downgrades/ and mixed mode operations. This filed is purely used for an optimization as mentioned above. Note: We could also do the same for blacklisted tservers, and execute `MetaCache::MarkTSFailed`. This would be useful on cluster scale in/scale out activities where requests are being sent to followers (which is rare, and hence covering just the leader case here). Original commit: 5292a99c4417137a2b306ccd65447701ec301168 / D54742 Test Plan: Jenkins Manually tested the following scenario on a local rf3 cluster: 1. `create table test(k int) split into 10 tablets;` 2. set vmodule on `tablet_rpc=1` 3. randomly execut eleader stepdown using `yb-admin --master_addresses=127.0.0.1:7100,127.0.0.2:7100.127.0.0.3:7100 leader_stepdown <tablet>` 4. `insert into test select i from generate_series(1, 10) as i;` and notice the following logs on the tserver hosting the connection ``` I0618 23:14:27.320129 1333856 tablet_rpc.cc:312] vlog1: Not the leader for Write(tablet: 6f37f80451b14092b7af7cee4efa39b5, num_ops: 4, num_attempts: 1, txn: 152ea27d-b9a9-4f0c-9371-87b3d7ca5fc8, subtxn: [none]) retrying with a different replica I0618 23:14:27.322331 1334100 tablet_rpc.cc:312] vlog1: Not the leader for Write(tablet: 5cb9fe88fa8a4ff69f7effa2e8aabcaa, num_ops: 2, num_attempts: 1, txn: 152ea27d-b9a9-4f0c-9371-87b3d7ca5fc8, subtxn: [none]) retrying with a different replica ``` 5. now blacklist one of the tservers using `yb-admin --master_addresses=127.0.0.1:7100,127.0.0.2:7100.127.0.0.3:7100 change_leader_blacklist ADD <ip>:9100` 6. `insert into test select i from generate_series(1, 10) as i;` and notice we don't see any complaints of `Not the leader for Write` (assuming the write is issue after 1s, that is, after the tserver processes the master heartbeat resp). Reviewers: amitanand, neera.mital, mlillibridge Reviewed By: amitanand Subscribers: ybase Differential Revision: https://phorge.dev.yugabyte.com/D55077

[BACKPORT 2025.1][#31930,#31241] docdb: Fix RBS vs ConfigChange races Summary: There is a possibility of RBS vs ConfigChange races, for example: 1. Raft config for the tablet has nodes A (leader), B, C. 2. D is added to Raft config, CHANGE_CONFIG operation is committed and applied on A, B, C 3. RBS A -> D started 4. D is removed from Raft config, CHANGE_CONFIG operation is committed and applied on leader 5. RBS A -> D downloads WAL and completed, D has the latest committed Raft config **Expected result:** orphaned tablet replica on D should be deleted. **Actual result:** we have an orphaned tablet replica (lagging follower) on D that is not a part of Raft group (which has 3 peers) and therefore is not receiving consensus updates from leader. In order to fix that, behaviour inside `MasterHeartbeatServiceImpl::ProcessTabletReport` is changed to delete a tablet replica which is no longer part of the committed Raft config and Raft config that added that replica is no longer pending (either committed or aborted). The latter condition avoids deleting the newly RBSed tablet replica that is being added to the tablet Raft group. The following logic is implemented to support that: 1. Once tablet leader decides to start RBS replica on another tserver, it will include the current pending Raft config op id (both term and index) into `StartRemoteBootstrapRequestPB` or empty op id when no config change is pending. 2. The bootstrapping replica persists this op id in its consensus metadata as `pending_config_op_id_from_rbs`. It is cleared once the replica's last committed op id either advances to a higher term, or its index reaches/passes the stored pending op id's index - i.e. once the original `CHANGE_CONFIG` operation can no longer be pending (it has either committed or been aborted). 3. `ReportedTabletPB::pending_config_op_id` is added to tserver->master heartbeats. Its value is whichever is set: the replica's currently active pending config op id (a config change in progress on the replica itself) or the pending_config_op_id_from_rbs from (2). 4. When master leader receives a tablet report from a replica that is *not* in the committed Raft config last known by the master, the master triggers `DeleteTabletRequestPB` to that replica if both: a. The reported committed Raft config op id index is <= the committed Raft config op id index last known by master for this tablet. b. Either the report carries no pending op id (empty/missing), or the master can prove it is no longer pending: if the pending op id's term is strictly less than the `current_term` in the master's last-known committed consensus state for this tablet, that `CHANGE_CONFIG` must have already been aborted or committed, and condition (4.a) alone is sufficient. 5. TServer will include tablet in next heartbeat in case DeleteTablet failed due to stale cas_config_opid_index_less_or_equal (this scenario is covered by TabletReplacementITest.TombstoneEvictedReplicaWithRbsAndConfigChangeRaceAndMasterRestart). Also update master-side of CloneTablet operation to seed the target tablet's committed_consensus_state peers on the master from the source tablet's config so that when the cloned replicas heartbeat back, `ProcessTabletReportBatch` sees them as part of the expected Raft config and does not tombstone them. Added several unit-tests for other RBS vs ConfigChange race scenarios. They are also fixed by the implemented change. Also renamed `RaftConfigPB.opid_index` to `committed_op_index` in order to reflect the actual purpose of this field. **Upgrade/Rollback safety:** New logic at master-side is gated by a new `use_tablet_report_pending_config_op_id` auto flag. Until the whole cluster is fully upgraded, master won't rely on newly added `ReportedTabletPB::pending_config_op_id` field. Original commit: 54c3d4ca73c593b364abf3f9462e788d32a20f53 / D52759 Test Plan: Run the following tests for asan/tsan/debug/relelase for 20 iterations each: - TabletSplitITest.SplitWithParentTabletRbsFromFollower - TabletReplacementITest.TombstoneEvictedReplicaWithRbsAndConfigChangeRace - TabletReplacementITest.TombstoneEvictedReplicaWithRbsAndConfigChangeRaceAndMasterRestart - TabletReplacementITest.TombstoneEvictedReplicaAfterAbortedAddServer - covers https://github.com/yugabyte/yugabyte-db/issues/31241 - TabletReplacementITest.DontDeleteNewReplicaInPendingConfig - TabletReplacementITest.DontDeleteNewReplicaInPendingConfigAfterRbsFromFollowerRf5 Reviewers: zdrudi Reviewed By: zdrudi Subscribers: ybase Tags: #jenkins-ready Differential Revision: https://phorge.dev.yugabyte.com/D55088

[BACKPORT 2024.2][#31930,#31241] docdb: Fix RBS vs ConfigChange races Summary: There is a possibility of RBS vs ConfigChange races, for example: 1. Raft config for the tablet has nodes A (leader), B, C. 2. D is added to Raft config, CHANGE_CONFIG operation is committed and applied on A, B, C 3. RBS A -> D started 4. D is removed from Raft config, CHANGE_CONFIG operation is committed and applied on leader 5. RBS A -> D downloads WAL and completed, D has the latest committed Raft config **Expected result:** orphaned tablet replica on D should be deleted. **Actual result:** we have an orphaned tablet replica (lagging follower) on D that is not a part of Raft group (which has 3 peers) and therefore is not receiving consensus updates from leader. In order to fix that, behaviour inside `MasterHeartbeatServiceImpl::ProcessTabletReport` is changed to delete a tablet replica which is no longer part of the committed Raft config and Raft config that added that replica is no longer pending (either committed or aborted). The latter condition avoids deleting the newly RBSed tablet replica that is being added to the tablet Raft group. The following logic is implemented to support that: 1. Once tablet leader decides to start RBS replica on another tserver, it will include the current pending Raft config op id (both term and index) into `StartRemoteBootstrapRequestPB` or empty op id when no config change is pending. 2. The bootstrapping replica persists this op id in its consensus metadata as `pending_config_op_id_from_rbs`. It is cleared once the replica's last committed op id either advances to a higher term, or its index reaches/passes the stored pending op id's index - i.e. once the original `CHANGE_CONFIG` operation can no longer be pending (it has either committed or been aborted). 3. `ReportedTabletPB::pending_config_op_id` is added to tserver->master heartbeats. Its value is whichever is set: the replica's currently active pending config op id (a config change in progress on the replica itself) or the pending_config_op_id_from_rbs from (2). 4. When master leader receives a tablet report from a replica that is *not* in the committed Raft config last known by the master, the master triggers `DeleteTabletRequestPB` to that replica if both: a. The reported committed Raft config op id index is <= the committed Raft config op id index last known by master for this tablet. b. Either the report carries no pending op id (empty/missing), or the master can prove it is no longer pending: if the pending op id's term is strictly less than the `current_term` in the master's last-known committed consensus state for this tablet, that `CHANGE_CONFIG` must have already been aborted or committed, and condition (4.a) alone is sufficient. 5. TServer will include tablet in next heartbeat in case DeleteTablet failed due to stale cas_config_opid_index_less_or_equal (this scenario is covered by TabletReplacementITest.TombstoneEvictedReplicaWithRbsAndConfigChangeRaceAndMasterRestart). Also update master-side of CloneTablet operation to seed the target tablet's committed_consensus_state peers on the master from the source tablet's config so that when the cloned replicas heartbeat back, `ProcessTabletReportBatch` sees them as part of the expected Raft config and does not tombstone them. Added several unit-tests for other RBS vs ConfigChange race scenarios. They are also fixed by the implemented change. Also renamed `RaftConfigPB.opid_index` to `committed_op_index` in order to reflect the actual purpose of this field. **Upgrade/Rollback safety:** New logic at master-side is gated by a new `use_tablet_report_pending_config_op_id` auto flag. Until the whole cluster is fully upgraded, master won't rely on newly added `ReportedTabletPB::pending_config_op_id` field. Original commit: 54c3d4ca73c593b364abf3f9462e788d32a20f53 / D52759 Test Plan: Run the following tests for asan/tsan/debug/relelase for 20 iterations each: - TabletSplitITest.SplitWithParentTabletRbsFromFollower - TabletReplacementITest.TombstoneEvictedReplicaWithRbsAndConfigChangeRace - TabletReplacementITest.TombstoneEvictedReplicaWithRbsAndConfigChangeRaceAndMasterRestart - TabletReplacementITest.TombstoneEvictedReplicaAfterAbortedAddServer - covers https://github.com/yugabyte/yugabyte-db/issues/31241 - TabletReplacementITest.DontDeleteNewReplicaInPendingConfig - TabletReplacementITest.DontDeleteNewReplicaInPendingConfigAfterRbsFromFollowerRf5 Reviewers: zdrudi Reviewed By: zdrudi Subscribers: ybase Tags: #jenkins-ready Differential Revision: https://phorge.dev.yugabyte.com/D55089

[BACKPORT 2024.2.10][#31930,#31241] docdb: Fix RBS vs ConfigChange races Summary: There is a possibility of RBS vs ConfigChange races, for example: 1. Raft config for the tablet has nodes A (leader), B, C. 2. D is added to Raft config, CHANGE_CONFIG operation is committed and applied on A, B, C 3. RBS A -> D started 4. D is removed from Raft config, CHANGE_CONFIG operation is committed and applied on leader 5. RBS A -> D downloads WAL and completed, D has the latest committed Raft config **Expected result:** orphaned tablet replica on D should be deleted. **Actual result:** we have an orphaned tablet replica (lagging follower) on D that is not a part of Raft group (which has 3 peers) and therefore is not receiving consensus updates from leader. In order to fix that, behaviour inside `MasterHeartbeatServiceImpl::ProcessTabletReport` is changed to delete a tablet replica which is no longer part of the committed Raft config and Raft config that added that replica is no longer pending (either committed or aborted). The latter condition avoids deleting the newly RBSed tablet replica that is being added to the tablet Raft group. The following logic is implemented to support that: 1. Once tablet leader decides to start RBS replica on another tserver, it will include the current pending Raft config op id (both term and index) into `StartRemoteBootstrapRequestPB` or empty op id when no config change is pending. 2. The bootstrapping replica persists this op id in its consensus metadata as `pending_config_op_id_from_rbs`. It is cleared once the replica's last committed op id either advances to a higher term, or its index reaches/passes the stored pending op id's index - i.e. once the original `CHANGE_CONFIG` operation can no longer be pending (it has either committed or been aborted). 3. `ReportedTabletPB::pending_config_op_id` is added to tserver->master heartbeats. Its value is whichever is set: the replica's currently active pending config op id (a config change in progress on the replica itself) or the pending_config_op_id_from_rbs from (2). 4. When master leader receives a tablet report from a replica that is *not* in the committed Raft config last known by the master, the master triggers `DeleteTabletRequestPB` to that replica if both: a. The reported committed Raft config op id index is <= the committed Raft config op id index last known by master for this tablet. b. Either the report carries no pending op id (empty/missing), or the master can prove it is no longer pending: if the pending op id's term is strictly less than the `current_term` in the master's last-known committed consensus state for this tablet, that `CHANGE_CONFIG` must have already been aborted or committed, and condition (4.a) alone is sufficient. 5. TServer will include tablet in next heartbeat in case DeleteTablet failed due to stale cas_config_opid_index_less_or_equal (this scenario is covered by TabletReplacementITest.TombstoneEvictedReplicaWithRbsAndConfigChangeRaceAndMasterRestart). Also update master-side of CloneTablet operation to seed the target tablet's committed_consensus_state peers on the master from the source tablet's config so that when the cloned replicas heartbeat back, `ProcessTabletReportBatch` sees them as part of the expected Raft config and does not tombstone them. Added several unit-tests for other RBS vs ConfigChange race scenarios. They are also fixed by the implemented change. Also renamed `RaftConfigPB.opid_index` to `committed_op_index` in order to reflect the actual purpose of this field. **Upgrade/Rollback safety:** New logic at master-side is gated by a new `use_tablet_report_pending_config_op_id` auto flag. Until the whole cluster is fully upgraded, master won't rely on newly added `ReportedTabletPB::pending_config_op_id` field. Original commit: 54c3d4ca73c593b364abf3f9462e788d32a20f53 / D52759 Test Plan: Run the following tests for asan/tsan/debug/relelase for 20 iterations each: - TabletSplitITest.SplitWithParentTabletRbsFromFollower - TabletReplacementITest.TombstoneEvictedReplicaWithRbsAndConfigChangeRace - TabletReplacementITest.TombstoneEvictedReplicaWithRbsAndConfigChangeRaceAndMasterRestart - TabletReplacementITest.TombstoneEvictedReplicaAfterAbortedAddServer - covers https://github.com/yugabyte/yugabyte-db/issues/31241 - TabletReplacementITest.DontDeleteNewReplicaInPendingConfig - TabletReplacementITest.DontDeleteNewReplicaInPendingConfigAfterRbsFromFollowerRf5 Reviewers: zdrudi Reviewed By: zdrudi Subscribers: ybase Tags: #jenkins-ready Differential Revision: https://phorge.dev.yugabyte.com/D55166

[BACKPORT 2024.2.10][#27056] docdb: Fixed tablet split vs RBS from the follower race Summary: There is a possibility of tablet split vs RBS from follower race: 1. Parent tablet leader peers A-C accept a SPLIT_OP (op_id: 1.4). Leader (A) and 1st follower (B) apply SPLIT_OP, 2nd follower (C) doesn't apply it yet. 2. Parent tablet leader (node A) accepts CHANGE_CONFIG_OP (op_id: 1.5) to add a fourth peer (D) but doesn't apply it yet. 3. Parent tablet 2nd follower (C) still hasn't yet applied SPLIT_OP (1.4). 4. RBS for parent tablet peer (D) starts from the follower (C) and tablet metadata (tablet_data_state == TABLET_DATA_READY) is downloaded. 5. Parent tablet peers A-C completed applying the SPLIT_OP (1.4), child tablets have Raft config with 3 peers. 6. Parent tablet peers A-C apply CHANGE_CONFIG_OP (1.5) and now have committed Raft config with 4 peers. 7. Parent tablet peer D does local bootstrap and replays SPLIT_OP (1.4) as part of bootstrap. Due to tablet_data_state is TABLET_DATA_READY but not TABLET_DATA_SPLIT_COMPLETED replay does SPLIT_OP apply and creates child tablet peer. After that, 4th child tablet peer (D) is not a part of Raft group (which has 3 peers) and therefore is not receiving consensus updates from leader. This change fixes this race by rejecting RBS from the follower that is in progress of applying SPLIT_OP and RBS attempt will be retried later. Original commit: 78af3a208a6bfc005eedc0ed6e410d22f4d24758 / D48853 **Upgrade/Rollback safety:** New error code will be printed by old nodes as just number in case of RBS failure during upgrade but this is safe. Test Plan: TabletSplitITest.SplitWithParentTabletRbsFromFollower, TabletSplitITest.SplitWithParentTabletMove, RemoteBootstrapsFromNodeWithUncommittedSplitOp - 30 runs per each of asan/tsan/debug/release builds Reviewers: arybochkin Reviewed By: arybochkin Subscribers: ybase Tags: #jenkins-ready Differential Revision: https://phorge.dev.yugabyte.com/D55142

2026-07-01

[BACKPORT 2026.1][#32353] DocDB: Proactively mark leader blacklisted tservers as followers in MetaCache Summary: YBA leader blacklists tservers before restarting each tserver in the process of a rolling restart. Though the leaders have moved away from this node, the other tservers operating on a stale meta-cache view would still try hitting the down tserver for reads/writes if they haven't already noticed that the leader has been moved away. It isn't much of a problem in the usual case since the node might RST the packet and the tserver would move on to try the other replicas before going to the master to re-fetch the tablet locations. But in case of an ungraceful termination/shutdown of the node (or in a kubernetes environment), the connection could keep blackholing and be kept alive for `rpc_connection_timeout_ms` (defaults to 15s). This might affect op latencies which is expected to some extent in an ungraceful shutdown, but isn't really ok on a planned one. Since the latencies get affected for even a planned shutdown in a kubernetes env, it is better it the other tservers proactively mark the tserver to go down as a follower and not route read/write traffic to it (It could still route reads if follower reads is on though). Changes 1. This revision addresses the above issue by propagating the leader blacklisted tservers info on the master -> tserver heartbeat response once it sees the leader count has dropped to 0 (for the blacklisted tsevrer). Tservers mark these leader blacklisted tservers (with 0 leaders) as followers and hence wouldn't route read/write requests to them. When the leader blacklisted tserver gets un-blacklisted though, we don't explicitly reset this info and let the meta-cache figure it out eventually when a follower peer sends the latest consensus info or when the meta-cache ends up going to the master. This propagation is done at most once per tserver per leader blacklist addition. 2. Incase the leader load on the leader blacklisted tservers is 0 while responding to `GetLoadMoveCompletionPercent`, delay until master has seen a heartbeat from all live tservers. This is being done as best effort for the other live tservers to mark the leader blacklisted tserver as follower. Both the above changes are protected under new gflags which default to true (optimizations enabled by default). **Upgrade/Downgrade safety** Added new `leader_blacklisted_tservers_with_no_leaders` to the tserver-master heartbeat message. The tserver checks for the size of the field before trying to access the repeated field, so the opeartion is safe ic case of upgrades/downgrades/ and mixed mode operations. This filed is purely used for an optimization as mentioned above. Note: We could also do the same for blacklisted tservers, and execute `MetaCache::MarkTSFailed`. This would be useful on cluster scale in/scale out activities where requests are being sent to followers (which is rare, and hence covering just the leader case here). Original commit: 5292a99c4417137a2b306ccd65447701ec301168 / D54742 Test Plan: Jenkins Manually tested the following scenario on a local rf3 cluster: 1. `create table test(k int) split into 10 tablets;` 2. set vmodule on `tablet_rpc=1` 3. randomly execut eleader stepdown using `yb-admin --master_addresses=127.0.0.1:7100,127.0.0.2:7100.127.0.0.3:7100 leader_stepdown <tablet>` 4. `insert into test select i from generate_series(1, 10) as i;` and notice the following logs on the tserver hosting the connection ``` I0618 23:14:27.320129 1333856 tablet_rpc.cc:312] vlog1: Not the leader for Write(tablet: 6f37f80451b14092b7af7cee4efa39b5, num_ops: 4, num_attempts: 1, txn: 152ea27d-b9a9-4f0c-9371-87b3d7ca5fc8, subtxn: [none]) retrying with a different replica I0618 23:14:27.322331 1334100 tablet_rpc.cc:312] vlog1: Not the leader for Write(tablet: 5cb9fe88fa8a4ff69f7effa2e8aabcaa, num_ops: 2, num_attempts: 1, txn: 152ea27d-b9a9-4f0c-9371-87b3d7ca5fc8, subtxn: [none]) retrying with a different replica ``` 5. now blacklist one of the tservers using `yb-admin --master_addresses=127.0.0.1:7100,127.0.0.2:7100.127.0.0.3:7100 change_leader_blacklist ADD <ip>:9100` 6. `insert into test select i from generate_series(1, 10) as i;` and notice we don't see any complaints of `Not the leader for Write` (assuming the write is issue after 1s, that is, after the tserver processes the master heartbeat resp). Reviewers: amitanand, neera.mital, mlillibridge Reviewed By: amitanand Subscribers: ybase Differential Revision: https://phorge.dev.yugabyte.com/D55076

[BACKPORT 2025.2][#30578] YSQL: Reset auto-analyze mutation counts after a manual ANALYZE Summary: Reset auto-analyze mutation counters after user-initiated ANALYZE Original commit: b6bbc7a663df2252865be498931bd4484ac7c224 / #31849 Cherry-picked from: 62358fae62da8e85c60a8dfb9c27e6e0275668b8 (2026.1 backport) Backport conflicts resolved (2026.1 -> 2025.2): - src/yb/ash/wait_state.h: kept only the new kResetAutoAnalyzeMutationCounters enumerator; the adjacent kGetTabletForKey / kRemotePgExec entries are 2026.1-only features not present in 2025.2. - src/yb/yql/pggate/ybc_pggate.h: kept only the YBCResetAutoAnalyzeMutationCounters declaration; the adjacent PgGlobalViewRead API block is a 2026.1-only feature. - src/yb/tserver/pg_client_service.cc: added the ResetAutoAnalyzeMutationCounters handler; dropped the bundled RemoteExec handler, which is a 2026.1-only feature absent from 2025.2. - src/yb/tserver/stateful_services/pg_auto_analyze_service.cc: applied the UpdateTableMutationsAfterAnalyze refactor to call SubtractPgAutoAnalyzeMutationCounts; the conflict was caused by 2025.2 using explicit PB types and a NewWriteOp overload without session->arena(). The table_tuple_count_ erase is preserved in the new snapshot-building loop. User-initiated ANALYZE does not reset the auto-analyze service's accumulated mutation count. The next periodic tick can therefore trigger an auto-analyze on a table the user just analyzed, wasting work. Add ResetAutoAnalyzeMutationCounters RPC (pg_client.proto / pg_client_service) called from do_analyze_rel after a full ANALYZE. The RPC resets the YCQL service table mutations to 0 for the analyzed table. Reset is gated to match PostgreSQL's semantics for updating its changes_since_analyze counters (fires only when no column list given) and is suppressed for auto-analyze service's internal connections and other internal backends. Auto analyze continues to reset mutations using the existing logic to subtract mutations on its side. Mutations are stored in a separate YCQL table so the mutations update is not transactional. A failed manual ANALYZE can still reset counters to 0. Fixing this is tracked in #32081. Extract mutation-update logic into helper functions in pg_auto_analyze_table that use conditional YCQL writes for better reuse. - ResetPgAutoAnalyzeMutationCounts: sets mutations to 0 (IF EXISTS) - SubtractPgAutoAnalyzeMutationCounts: subtracts snapshot mutations with clamping. Emits two conditional writes per table — one sets to 0 if current < snapshot, another subtracts if current >= snapshot. This prevents the count from going negative when a manual-ANALYZE reset races with auto-analyze post-ANALYZE mutation subtraction. UpdateTableMutationsAfterAnalyze refactored to use SubtractPgAutoAnalyzeMutationCounts instead of building operations directly. New RPC: PgClientService.ResetAutoAnalyzeMutationCounters (pg_client.proto). Additive only — no existing field numbers or messages altered. Upgrade/rollback safety: New RPC is between PG and local tserver so no upgrade/rollback issues. Test Plan: - PgAutoAnalyzeTest.ManualAnalyzeResetsMutationCount: covers ANALYZE, ANALYZE(col), VACUUM ANALYZE, VACUUM ANALYZE(col), asserting reset only when no column list provided. - PgAutoAnalyzeTest.InternalAnalyzeDoesNotResetMutationCount: confirms internal connections do not trigger reset. - PgAutoAnalyzeTest.ManualAnalyzePartitionedTableResetsPartitionMutationCounts: exercises the partitioned-table path. - Existing PgAutoAnalyzeTest cases continue to pass. Reviewers: kfranz, pjain Reviewed By: kfranz Differential Revision: https://phorge.dev.yugabyte.com/D55092

2026-06-30

[BACKPORT 2025.2][#31930,#31241] docdb: Fix RBS vs ConfigChange races Summary: There is a possibility of RBS vs ConfigChange races, for example: 1. Raft config for the tablet has nodes A (leader), B, C. 2. D is added to Raft config, CHANGE_CONFIG operation is committed and applied on A, B, C 3. RBS A -> D started 4. D is removed from Raft config, CHANGE_CONFIG operation is committed and applied on leader 5. RBS A -> D downloads WAL and completed, D has the latest committed Raft config **Expected result:** orphaned tablet replica on D should be deleted. **Actual result:** we have an orphaned tablet replica (lagging follower) on D that is not a part of Raft group (which has 3 peers) and therefore is not receiving consensus updates from leader. In order to fix that, behaviour inside `MasterHeartbeatServiceImpl::ProcessTabletReport` is changed to delete a tablet replica which is no longer part of the committed Raft config and Raft config that added that replica is no longer pending (either committed or aborted). The latter condition avoids deleting the newly RBSed tablet replica that is being added to the tablet Raft group. The following logic is implemented to support that: 1. Once tablet leader decides to start RBS replica on another tserver, it will include the current pending Raft config op id (both term and index) into `StartRemoteBootstrapRequestPB` or empty op id when no config change is pending. 2. The bootstrapping replica persists this op id in its consensus metadata as `pending_config_op_id_from_rbs`. It is cleared once the replica's last committed op id either advances to a higher term, or its index reaches/passes the stored pending op id's index - i.e. once the original `CHANGE_CONFIG` operation can no longer be pending (it has either committed or been aborted). 3. `ReportedTabletPB::pending_config_op_id` is added to tserver->master heartbeats. Its value is whichever is set: the replica's currently active pending config op id (a config change in progress on the replica itself) or the pending_config_op_id_from_rbs from (2). 4. When master leader receives a tablet report from a replica that is *not* in the committed Raft config last known by the master, the master triggers `DeleteTabletRequestPB` to that replica if both: a. The reported committed Raft config op id index is <= the committed Raft config op id index last known by master for this tablet. b. Either the report carries no pending op id (empty/missing), or the master can prove it is no longer pending: if the pending op id's term is strictly less than the `current_term` in the master's last-known committed consensus state for this tablet, that `CHANGE_CONFIG` must have already been aborted or committed, and condition (4.a) alone is sufficient. 5. TServer will include tablet in next heartbeat in case DeleteTablet failed due to stale cas_config_opid_index_less_or_equal (this scenario is covered by TabletReplacementITest.TombstoneEvictedReplicaWithRbsAndConfigChangeRaceAndMasterRestart). Also update master-side of CloneTablet operation to seed the target tablet's committed_consensus_state peers on the master from the source tablet's config so that when the cloned replicas heartbeat back, `ProcessTabletReportBatch` sees them as part of the expected Raft config and does not tombstone them. Added several unit-tests for other RBS vs ConfigChange race scenarios. They are also fixed by the implemented change. Also renamed `RaftConfigPB.opid_index` to `committed_op_index` in order to reflect the actual purpose of this field. **Upgrade/Rollback safety:** New logic at master-side is gated by a new `use_tablet_report_pending_config_op_id` auto flag. Until the whole cluster is fully upgraded, master won't rely on newly added `ReportedTabletPB::pending_config_op_id` field. Original commit: 54c3d4ca73c593b364abf3f9462e788d32a20f53 / D52759 Test Plan: Run the following tests for asan/tsan/debug/relelase for 20 iterations each: - TabletSplitITest.SplitWithParentTabletRbsFromFollower - TabletReplacementITest.TombstoneEvictedReplicaWithRbsAndConfigChangeRace - TabletReplacementITest.TombstoneEvictedReplicaWithRbsAndConfigChangeRaceAndMasterRestart - TabletReplacementITest.TombstoneEvictedReplicaAfterAbortedAddServer - covers https://github.com/yugabyte/yugabyte-db/issues/31241 - TabletReplacementITest.DontDeleteNewReplicaInPendingConfig - TabletReplacementITest.DontDeleteNewReplicaInPendingConfigAfterRbsFromFollowerRf5 Reviewers: zdrudi Reviewed By: zdrudi Subscribers: ybase Tags: #jenkins-ready Differential Revision: https://phorge.dev.yugabyte.com/D55040

[BACKPORT 2026.1][#31930,#31241] docdb: Fix RBS vs ConfigChange races Summary: There is a possibility of RBS vs ConfigChange races, for example: 1. Raft config for the tablet has nodes A (leader), B, C. 2. D is added to Raft config, CHANGE_CONFIG operation is committed and applied on A, B, C 3. RBS A -> D started 4. D is removed from Raft config, CHANGE_CONFIG operation is committed and applied on leader 5. RBS A -> D downloads WAL and completed, D has the latest committed Raft config **Expected result:** orphaned tablet replica on D should be deleted. **Actual result:** we have an orphaned tablet replica (lagging follower) on D that is not a part of Raft group (which has 3 peers) and therefore is not receiving consensus updates from leader. In order to fix that, behaviour inside `MasterHeartbeatServiceImpl::ProcessTabletReport` is changed to delete a tablet replica which is no longer part of the committed Raft config and Raft config that added that replica is no longer pending (either committed or aborted). The latter condition avoids deleting the newly RBSed tablet replica that is being added to the tablet Raft group. The following logic is implemented to support that: 1. Once tablet leader decides to start RBS replica on another tserver, it will include the current pending Raft config op id (both term and index) into `StartRemoteBootstrapRequestPB` or empty op id when no config change is pending. 2. The bootstrapping replica persists this op id in its consensus metadata as `pending_config_op_id_from_rbs`. It is cleared once the replica's last committed op id either advances to a higher term, or its index reaches/passes the stored pending op id's index - i.e. once the original `CHANGE_CONFIG` operation can no longer be pending (it has either committed or been aborted). 3. `ReportedTabletPB::pending_config_op_id` is added to tserver->master heartbeats. Its value is whichever is set: the replica's currently active pending config op id (a config change in progress on the replica itself) or the pending_config_op_id_from_rbs from (2). 4. When master leader receives a tablet report from a replica that is *not* in the committed Raft config last known by the master, the master triggers `DeleteTabletRequestPB` to that replica if both: a. The reported committed Raft config op id index is <= the committed Raft config op id index last known by master for this tablet. b. Either the report carries no pending op id (empty/missing), or the master can prove it is no longer pending: if the pending op id's term is strictly less than the `current_term` in the master's last-known committed consensus state for this tablet, that `CHANGE_CONFIG` must have already been aborted or committed, and condition (4.a) alone is sufficient. 5. TServer will include tablet in next heartbeat in case DeleteTablet failed due to stale cas_config_opid_index_less_or_equal (this scenario is covered by TabletReplacementITest.TombstoneEvictedReplicaWithRbsAndConfigChangeRaceAndMasterRestart). Also update master-side of CloneTablet operation to seed the target tablet's committed_consensus_state peers on the master from the source tablet's config so that when the cloned replicas heartbeat back, `ProcessTabletReportBatch` sees them as part of the expected Raft config and does not tombstone them. Added several unit-tests for other RBS vs ConfigChange race scenarios. They are also fixed by the implemented change. Also renamed `RaftConfigPB.opid_index` to `committed_op_index` in order to reflect the actual purpose of this field. **Upgrade/Rollback safety:** New logic at master-side is gated by a new `use_tablet_report_pending_config_op_id` auto flag. Until the whole cluster is fully upgraded, master won't rely on newly added `ReportedTabletPB::pending_config_op_id` field. Original commit: 54c3d4ca73c593b364abf3f9462e788d32a20f53 / D52759 Test Plan: Run the following tests for asan/tsan/debug/relelase for 20 iterations each: - TabletSplitITest.SplitWithParentTabletRbsFromFollower - TabletReplacementITest.TombstoneEvictedReplicaWithRbsAndConfigChangeRace - TabletReplacementITest.TombstoneEvictedReplicaWithRbsAndConfigChangeRaceAndMasterRestart - TabletReplacementITest.TombstoneEvictedReplicaAfterAbortedAddServer - covers https://github.com/yugabyte/yugabyte-db/issues/31241 - TabletReplacementITest.DontDeleteNewReplicaInPendingConfig - TabletReplacementITest.DontDeleteNewReplicaInPendingConfigAfterRbsFromFollowerRf5 Reviewers: zdrudi Reviewed By: zdrudi Subscribers: ybase Tags: #jenkins-ready Differential Revision: https://phorge.dev.yugabyte.com/D54974

[BACKPORT 2025.2][#32116] CDC: Add support to create table bound gRPC streams Summary: ### Backport Description Minor merge conflicts in `src/yb/master/xrepl_catalog_manager.cc`, due to a parameter not being present in the function `IsTableEligibleForCDCSDKStream()`. ### Original Description ##### Code changes summary Currently when a replication slot is created, retention barriers are setup and cdc_state table entries are written for all the tablets in the DB. In an environment where large number of tables are present and only a small subset is being used for CDC, this leads to unnecessary retention barrier setup. Also the cdc_state table is bloated with unnecessary entries. To prevent this, this diff introduces a mechanism to create gRPC streams that are bound to only specific tables at the time of their creation. To create such a stream, a comma separated list of table ids should be provided to the create_change_data_stream yb-admin command. The syntax is as follows: ``` ./yb-admin create_change_data_stream ysql.<DB-name> EXPLICIT CHANGE NOEXPORT_SNAPSHOT DYNAMIC_TABLES_DISABLED <comma separated table_ids> ``` For example: ``` ./yb-admin create_change_data_stream ysql.yugabyte EXPLICIT CHANGE NOEXPORT_SNAPSHOT DYNAMIC_TABLES_DISABLED 000034e1000030008000000000004000,000034e1000030008000000000004005 CDC Stream ID: 9fbec9b0395a2caacd48676d714ced0d ``` The table_ids are passed to the `CreateCDCStream` rpc by populating the `bound_table_ids` field in the `CDCSDKStreamCreateOptionsPB`. Only these table_ids are written to the stream metadata. The retention barriers are set on the tablets of only these tables, and their entries are written to the cdc_state table. Dynamic table addition is disabled for the table bound streams, meaning that the tables which can be polled using these streams is fixed at the stream creation. Any attmept to create such streams with dynamic tables enabled will fail. Also such streams can only be created for gRPC model. ##### Upgrade / Rollback safety Only proto change made in this diff is in `CDCSDKStreamCreateOptionsPB` which is a part of `CreateCDCStreamRequestPB`. The CreateCDCStream rpc flows from the tserver to the master. Since all the masters are upgraded before the tservers, this change is upgrade safe. Additionally the bound_table_ids field added in `CDCSDKStreamCreateOptionsPB` is an optional field. To make the repeated field optional it has been wrapped in a separate proto called `CDCSDKBoundTableIds`. A table bound stream created before rollback will continue to operate as intended after rollback, i.e post rollback user can use the table bound stream to get the change events from the tables present in the stream metadata. Hence this change is rollback safe. ##### Considerations for colocated tables If a stream is created such that it is bound to subset of colocated tables residing on the tablet, cdcsdk_producer will filter out the change records corresponding to other tables. ##### Considerations for connector NA Original commit: 6d1ef2a86b3e1b0e3055ba1614944245c88c479e / D54632 Test Plan: New tests added: - ./yb_build.sh --cxx-test integration-tests_cdcsdk_ysql-test --gtest_filter 'CDCSDKYsqlTest.TestgRPCStreamBoundToSpecificTables' - ./yb_build.sh --cxx-test integration-tests_cdcsdk_ysql-test --gtest_filter 'CDCSDKYsqlTest.TestgRPCStreamBoundToSpecificColocatedTables' - ./yb_build.sh --cxx-test integration-tests_cdcsdk_ysql-test --gtest_filter 'CDCSDKYsqlTest.TestTableBoundStreamRejectsWithReplicationSlot' - ./yb_build.sh --cxx-test integration-tests_cdcsdk_ysql-test --gtest_filter 'CDCSDKYsqlTest.TestTableBoundStreamDisablesDynamicAddition' - ./yb_build.sh --cxx-test integration-tests_cdcsdk_ysql-test --gtest_filter 'CDCSDKYsqlTest.TestCreationOfgRPCStreamBoundToSpecificTablesViaYBAdmin' - ./yb_build.sh --cxx-test integration-tests_cdcsdk_ysql-test --gtest_filter 'CDCSDKYsqlTest.TestTableBoundStreamYbAdminRejectsTableFromDifferentNamespace' - ./yb_build.sh --cxx-test integration-tests_cdcsdk_ysql-test --gtest_filter 'CDCSDKYsqlTest.TestTableBoundStreamYbAdminRejectsTableFromDifferentNamespaceTestTableBoundStreamYbAdminRejectsDynamicTablesEnabled' Reviewers: #db-approvers, xCluster, hsunder, skumar, asrinivasan, devansh.singhal Reviewed By: devansh.singhal Subscribers: svc_phabricator, ybase Tags: #jenkins-ready Differential Revision: https://phorge.dev.yugabyte.com/D55015

[#32353] DocDB: Proactively mark leader blacklisted tservers as followers in MetaCache Summary: YBA leader blacklists tservers before restarting each tserver in the process of a rolling restart. Though the leaders have moved away from this node, the other tservers operating on a stale meta-cache view would still try hitting the down tserver for reads/writes if they haven't already noticed that the leader has been moved away. It isn't much of a problem in the usual case since the node might RST the packet and the tserver would move on to try the other replicas before going to the master to re-fetch the tablet locations. But in case of an ungraceful termination/shutdown of the node (or in a kubernetes environment), the connection could keep blackholing and be kept alive for `rpc_connection_timeout_ms` (defaults to 15s). This might affect op latencies which is expected to some extent in an ungraceful shutdown, but isn't really ok on a planned one. Since the latencies get affected for even a planned shutdown in a kubernetes env, it is better it the other tservers proactively mark the tserver to go down as a follower and not route read/write traffic to it (It could still route reads if follower reads is on though). Changes 1. This revision addresses the above issue by propagating the leader blacklisted tservers info on the master -> tserver heartbeat response once it sees the leader count has dropped to 0 (for the blacklisted tsevrer). Tservers mark these leader blacklisted tservers (with 0 leaders) as followers and hence wouldn't route read/write requests to them. When the leader blacklisted tserver gets un-blacklisted though, we don't explicitly reset this info and let the meta-cache figure it out eventually when a follower peer sends the latest consensus info or when the meta-cache ends up going to the master. This propagation is done at most once per tserver per leader blacklist addition. 2. Incase the leader load on the leader blacklisted tservers is 0 while responding to `GetLoadMoveCompletionPercent`, delay until master has seen a heartbeat from all live tservers. This is being done as best effort for the other live tservers to mark the leader blacklisted tserver as follower. Both the above changes are protected under new gflags which default to true (optimizations enabled by default). **Upgrade/Downgrade safety** Added new `leader_blacklisted_tservers_with_no_leaders` to the tserver-master heartbeat message. The tserver checks for the size of the field before trying to access the repeated field, so the opeartion is safe ic case of upgrades/downgrades/ and mixed mode operations. This filed is purely used for an optimization as mentioned above. Note: We could also do the same for blacklisted tservers, and execute `MetaCache::MarkTSFailed`. This would be useful on cluster scale in/scale out activities where requests are being sent to followers (which is rare, and hence covering just the leader case here). Test Plan: Jenkins Manually tested the following scenario on a local rf3 cluster: 1. `create table test(k int) split into 10 tablets;` 2. set vmodule on `tablet_rpc=1` 3. randomly execut eleader stepdown using `yb-admin --master_addresses=127.0.0.1:7100,127.0.0.2:7100.127.0.0.3:7100 leader_stepdown <tablet>` 4. `insert into test select i from generate_series(1, 10) as i;` and notice the following logs on the tserver hosting the connection ``` I0618 23:14:27.320129 1333856 tablet_rpc.cc:312] vlog1: Not the leader for Write(tablet: 6f37f80451b14092b7af7cee4efa39b5, num_ops: 4, num_attempts: 1, txn: 152ea27d-b9a9-4f0c-9371-87b3d7ca5fc8, subtxn: [none]) retrying with a different replica I0618 23:14:27.322331 1334100 tablet_rpc.cc:312] vlog1: Not the leader for Write(tablet: 5cb9fe88fa8a4ff69f7effa2e8aabcaa, num_ops: 2, num_attempts: 1, txn: 152ea27d-b9a9-4f0c-9371-87b3d7ca5fc8, subtxn: [none]) retrying with a different replica ``` 5. now blacklist one of the tservers using `yb-admin --master_addresses=127.0.0.1:7100,127.0.0.2:7100.127.0.0.3:7100 change_leader_blacklist ADD <ip>:9100` 6. `insert into test select i from generate_series(1, 10) as i;` and notice we don't see any complaints of `Not the leader for Write` (assuming the write is issue after 1s, that is, after the tserver processes the master heartbeat resp). Reviewers: amitanand, neera.mital, mlillibridge Reviewed By: amitanand Subscribers: ybase Differential Revision: https://phorge.dev.yugabyte.com/D54742

[BACKPORT 2025.2][#30883] DocDB: Show vector index space usage in yb-master and yb-tserver UI pages Summary: The on-disk size breakdown shown on the yb-tserver and yb-master web UI pages did not account for vector indexes, so a table with a vector index under-reported its actual disk footprint. Add VectorLSM::OnDiskSize, which sums the sizes of the immutable chunk files currently on disk, and expose it through DocVectorIndex::OnDiskSize and VectorIndexList::OnDiskSize. TabletPeer aggregates the per-tablet vector index size into a new TabletOnDiskSizeInfo::vector_index_disk_size field, which is folded into active_on_disk_size and serialized in TabletStatusPB. Propagate the size to the master via a new vector_index_size field in TabletDriveStorageMetadataPB and TabletReplicaDriveInfo. The tserver tables/tablets pages and the master tables page now render a "Vector Indexes" line in the size breakdown, and the corresponding JSON endpoints expose vector_index_size. The HTML line is rendered only when the size is non-zero, so tables without a vector index are not cluttered with "Vector Indexes: 0B". Also make TabletVectorIndexes::List, TabletVectorIndexes::Collect, and TabletComponent::VectorIndexesList return the VectorIndexList wrapper instead of the raw docdb::DocVectorIndexesPtr. This removes the repeated VectorIndexList(...) wrapping at call sites; the few places that need the underlying pointer for the docdb write/apply path use the new VectorIndexList::impl accessor. --- **Upgrade / Rollback safety:** Adds field used by new functionality. The old code would just ignore it. New code would not show usage if information is received from node with an old code. --- _automated · Claude Code (Opus 4.8)_ Original commit: 3b4a0bafbf0850cbc7ae982c0cebf89fcca6021a / D54922 Test Plan: ./yb_build.sh debug --cxx-test pg_vector_index-test --gtest_filter 'PgVectorIndexTest.OnDiskSize*' Manually verified on a local cluster: a table with an ybhnsw vector index reports a non-zero "Vector Indexes" size on both the tserver (:9000) and master (:7000) UI pages and in their JSON endpoints, while a table without a vector index reports none. Reviewers: arybochkin Reviewed By: arybochkin Subscribers: ybase, yql Tags: #jenkins-ready Differential Revision: https://phorge.dev.yugabyte.com/D55051

[#30553] CDC: support publish action subsets in publications Summary: ###### Code changes summary: We did not support filtering publication DMLs by operation type using PostgreSQL `publish` syntax, for example: `CREATE PUBLICATION pub FOR TABLE t WITH (publish = 'insert')` and `ALTER PUBLICATION pub SET (publish = 'update, delete')`. This change enables that syntax by removing the YB-only restriction that required all publication actions to be enabled together, allowing subset publish options. It also adds `pg_publication` as a new catalog table in the stream metadata (alongside `pg_class` and `pg_publication_rel`), so the virtual WAL can poll it and notice when the publish actions of a publication change via `ALTER PUBLICATION SET (publish = ...)`. On an `UPDATE` to `pg_publication` for a tracked publication, we trigger a publication refresh. To tell a real `ALTER PUBLICATION` apart from the implicit refresh we do when a `CREATE TABLE` gets auto-added to a `FOR ALL TABLES` publication, we added a boolean field `explicit_alter_publication_detected` to `GetConsistentChangesResponsePB`. It's true for `ALTER PUBLICATION` and false for the `CREATE TABLE` auto-add case. The catalog cache reset and `syscache` invalidation callbacks are now fired only when this field is true, so we avoid re-sending `RELATION` messages for tables the consumer already knows about in the `CREATE TABLE` case. ###### Upgrade/rollback safety considerations: The new proto field `explicit_alter_publication_detected ` (field 5 in GetConsistentChangesResponsePB) is an optional bool that defaults to false. This is a local tserver-to-pg RPC message change (CDC consistent changes response), so no AutoFlag is needed per the upgrade safety handbook. During rollback, the field is absent from responses, defaulting to false, which matches the pre-existing behavior before this diff. Adding `pg_publication` to the stream metadata is safe because it is added alongside `pg_class` and `pg_publication_rel` during stream initialization in `xrepl_catalog_manager.cc`. New streams created after the upgrade will include `pg_publication` in their polling list. During rollback, old tserver code will not poll `pg_publication`, so ALTER PUBLICATION SET changes won't be detected this matches the pre-upgrade behavior. Existing streams are unaffected since the table is added at stream creation time. ###### Considerations for colocated tables: This fixes `ALTER PUBLICATION` for colocated tables. Earlier the change was ignored by the stream; now it is applied correctly. ###### Compatibility with logical and gRPC streams: Compatible with logical stream, gRPC streams are not affected. ###### Considerations for connector: N/A Test Plan: ./yb_build.sh debug --java-test org.yb.pgsql.TestPgReplicationSlot#testPublishFilterInsertOnly ./yb_build.sh debug --java-test org.yb.pgsql.TestPgReplicationSlot#testPublishFilterUpdateOnly ./yb_build.sh debug --java-test org.yb.pgsql.TestPgReplicationSlot#testPublishFilterDeleteOnly ./yb_build.sh debug --java-test org.yb.pgsql.TestPgReplicationSlot#testPublishFilterInsertAndUpdate ./yb_build.sh debug --java-test org.yb.pgsql.TestPgReplicationSlot#testPublishFilterUpdateAndDelete ./yb_build.sh debug --java-test org.yb.pgsql.TestPgReplicationSlot#testPublishFilterInsertAndDelete ./yb_build.sh debug --java-test org.yb.pgsql.TestPgReplicationSlot#testAlterPublicationPublishOption Reviewers: sumukh.phalgaonkar, skumar, jason, xCluster, hsunder, stiwary, #db-approvers Reviewed By: sumukh.phalgaonkar, #db-approvers Subscribers: ybase, jason, yql, ycdcxcluster Differential Revision: https://phorge.dev.yugabyte.com/D50894

[#31542] docdb: Vector Index: Chunked compaction implementation Summary: The change introduces a chunked compaction. It allows to not build one in-memory merged index for all input vectors but to output into multiple chunk files, each bounded by a configurable memory budget, to reduce OOM risk. If the limit is very small, compaction still produces at least one vector per output chunk. New gflag `vector_index_compaction_chunk_max_mem_store_size_mb` (runtime, default 0) is introduced and should be greater than 0 to enable chunked compaction. **Upgrade/Rollback safety:** Just a comment update in .proto file. Test Plan: ./yb_build.sh --cxx-test ann_methods_vector_lsm-test --gtest_filter VectorLSMTest.ChunkedCompactionRespectsMemStoreLimit/kHnswlib ./yb_build.sh --cxx-test ann_methods_vector_lsm-test --gtest_filter VectorLSMTest.ChunkedCompactionRespectsMemStoreLimit/kUsearch ./yb_build.sh --cxx-test ann_methods_vector_lsm-test --gtest_filter VectorLSMTest.OpenAfterChunkedCompaction/kHnswlib ./yb_build.sh --cxx-test ann_methods_vector_lsm-test --gtest_filter VectorLSMTest.OpenAfterChunkedCompaction/kUsearch ./yb_build.sh --cxx-test ann_methods_vector_lsm-test --gtest_filter VectorLSMTest.DefaultCompactionMergesMultipleChunks/kHnswlib ./yb_build.sh --cxx-test ann_methods_vector_lsm-test --gtest_filter VectorLSMTest.DefaultCompactionMergesMultipleChunks/kUsearch Reviewers: sergei, zdrudi Reviewed By: sergei, zdrudi Subscribers: ybase Differential Revision: https://phorge.dev.yugabyte.com/D54283

[BACKPORT 2026.1][#30883] DocDB: Show vector index space usage in yb-master and yb-tserver UI pages Summary: The on-disk size breakdown shown on the yb-tserver and yb-master web UI pages did not account for vector indexes, so a table with a vector index under-reported its actual disk footprint. Add VectorLSM::OnDiskSize, which sums the sizes of the immutable chunk files currently on disk, and expose it through DocVectorIndex::OnDiskSize and VectorIndexList::OnDiskSize. TabletPeer aggregates the per-tablet vector index size into a new TabletOnDiskSizeInfo::vector_index_disk_size field, which is folded into active_on_disk_size and serialized in TabletStatusPB. Propagate the size to the master via a new vector_index_size field in TabletDriveStorageMetadataPB and TabletReplicaDriveInfo. The tserver tables/tablets pages and the master tables page now render a "Vector Indexes" line in the size breakdown, and the corresponding JSON endpoints expose vector_index_size. The HTML line is rendered only when the size is non-zero, so tables without a vector index are not cluttered with "Vector Indexes: 0B". Also make TabletVectorIndexes::List, TabletVectorIndexes::Collect, and TabletComponent::VectorIndexesList return the VectorIndexList wrapper instead of the raw docdb::DocVectorIndexesPtr. This removes the repeated VectorIndexList(...) wrapping at call sites; the few places that need the underlying pointer for the docdb write/apply path use the new VectorIndexList::impl accessor. --- **Upgrade / Rollback safety:** Adds field used by new functionality. The old code would just ignore it. New code would not show usage if information is received from node with an old code. --- _automated · Claude Code (Opus 4.8)_ Original commit: 3b4a0bafbf0850cbc7ae982c0cebf89fcca6021a / D54922 Test Plan: ./yb_build.sh debug --cxx-test pg_vector_index-test --gtest_filter 'PgVectorIndexTest.OnDiskSize*' Manually verified on a local cluster: a table with an ybhnsw vector index reports a non-zero "Vector Indexes" size on both the tserver (:9000) and master (:7000) UI pages and in their JSON endpoints, while a table without a vector index reports none. Reviewers: arybochkin Reviewed By: arybochkin Subscribers: yql, ybase Tags: #jenkins-ready Differential Revision: https://phorge.dev.yugabyte.com/D55013

2026-06-29

[BACKPORT 2026.1][#32116] CDC: Add support to create table bound gRPC streams Summary: ### Backport Description No merge conflicts ### Original Description ##### Code changes summary Currently when a replication slot is created, retention barriers are setup and cdc_state table entries are written for all the tablets in the DB. In an environment where large number of tables are present and only a small subset is being used for CDC, this leads to unnecessary retention barrier setup. Also the cdc_state table is bloated with unnecessary entries. To prevent this, this diff introduces a mechanism to create gRPC streams that are bound to only specific tables at the time of their creation. To create such a stream, a comma separated list of table ids should be provided to the create_change_data_stream yb-admin command. The syntax is as follows: ``` ./yb-admin create_change_data_stream ysql.<DB-name> EXPLICIT CHANGE NOEXPORT_SNAPSHOT DYNAMIC_TABLES_DISABLED <comma separated table_ids> ``` For example: ``` ./yb-admin create_change_data_stream ysql.yugabyte EXPLICIT CHANGE NOEXPORT_SNAPSHOT DYNAMIC_TABLES_DISABLED 000034e1000030008000000000004000,000034e1000030008000000000004005 CDC Stream ID: 9fbec9b0395a2caacd48676d714ced0d ``` The table_ids are passed to the `CreateCDCStream` rpc by populating the `bound_table_ids` field in the `CDCSDKStreamCreateOptionsPB`. Only these table_ids are written to the stream metadata. The retention barriers are set on the tablets of only these tables, and their entries are written to the cdc_state table. Dynamic table addition is disabled for the table bound streams, meaning that the tables which can be polled using these streams is fixed at the stream creation. Any attmept to create such streams with dynamic tables enabled will fail. Also such streams can only be created for gRPC model. ##### Upgrade / Rollback safety Only proto change made in this diff is in `CDCSDKStreamCreateOptionsPB` which is a part of `CreateCDCStreamRequestPB`. The CreateCDCStream rpc flows from the tserver to the master. Since all the masters are upgraded before the tservers, this change is upgrade safe. Additionally the bound_table_ids field added in `CDCSDKStreamCreateOptionsPB` is an optional field. To make the repeated field optional it has been wrapped in a separate proto called `CDCSDKBoundTableIds`. A table bound stream created before rollback will continue to operate as intended after rollback, i.e post rollback user can use the table bound stream to get the change events from the tables present in the stream metadata. Hence this change is rollback safe. ##### Considerations for colocated tables If a stream is created such that it is bound to subset of colocated tables residing on the tablet, cdcsdk_producer will filter out the change records corresponding to other tables. ##### Considerations for connector NA Original commit: 6d1ef2a86b3e1b0e3055ba1614944245c88c479e / D54632 Test Plan: New tests added: - ./yb_build.sh --cxx-test integration-tests_cdcsdk_ysql-test --gtest_filter 'CDCSDKYsqlTest.TestgRPCStreamBoundToSpecificTables' - ./yb_build.sh --cxx-test integration-tests_cdcsdk_ysql-test --gtest_filter 'CDCSDKYsqlTest.TestgRPCStreamBoundToSpecificColocatedTables' - ./yb_build.sh --cxx-test integration-tests_cdcsdk_ysql-test --gtest_filter 'CDCSDKYsqlTest.TestTableBoundStreamRejectsWithReplicationSlot' - ./yb_build.sh --cxx-test integration-tests_cdcsdk_ysql-test --gtest_filter 'CDCSDKYsqlTest.TestTableBoundStreamDisablesDynamicAddition' - ./yb_build.sh --cxx-test integration-tests_cdcsdk_ysql-test --gtest_filter 'CDCSDKYsqlTest.TestCreationOfgRPCStreamBoundToSpecificTablesViaYBAdmin' - ./yb_build.sh --cxx-test integration-tests_cdcsdk_ysql-test --gtest_filter 'CDCSDKYsqlTest.TestTableBoundStreamYbAdminRejectsTableFromDifferentNamespace' - ./yb_build.sh --cxx-test integration-tests_cdcsdk_ysql-test --gtest_filter 'CDCSDKYsqlTest.TestTableBoundStreamYbAdminRejectsTableFromDifferentNamespaceTestTableBoundStreamYbAdminRejectsDynamicTablesEnabled' Reviewers: #db-approvers, xCluster, hsunder, skumar, asrinivasan, devansh.singhal Reviewed By: devansh.singhal Subscribers: ybase, svc_phabricator Tags: #jenkins-ready Differential Revision: https://phorge.dev.yugabyte.com/D55014

[#32360] DocDB: Vector index: add a new gflag and table parameter for vector reverse mapping ownership Summary: The change adds master-side gflag `enable_table_owned_vector_reverse_mapping` (default false), `owns_vector_reverse_mapping` on Schema::TableProperties, backup/restore fixup for that property, and plumbing of the flag/property into CompactionSchemaInfo and vector index backfill. The new parameter is set only during table creation and cannot be changed later. Additionally, `skip_reverse_mapping_backfill` is removed; backfill now follows the indexed table's `owns_vector_reverse_mapping` instead. Insert/update changes for vector reverse mapping ownership and compaction behavior are deferred to a follow-up revisions. **Upgrade/Rollback safety:** Guarded by master runtime gflag `enable_table_owned_vector_reverse_mapping`. Test Plan: ./yb_build.sh --cxx-test pg_vector_index-test --gtest_filter=PgVectorIndexUtilTest.BackfillSkipsReverseMapping ./yb_build.sh --cxx-test pg_vector_index-test --gtest_filter=PgVectorIndexUtilTest.BackfillWritesReverseMapping ./yb_build.sh --cxx-test pg_vector_index-test --gtest_filter=PgVectorIndexUtilTest.NumTopVectorsToRemoveExceedsResultEntries ./yb_build.sh --cxx-test yb-backup-cross-feature-test --gtest_filter=YBBackupTest.TestYSQLTableOwnedVectorReverseMapping Reviewers: sergei, hsunder Reviewed By: sergei, hsunder Subscribers: yql, ybase Differential Revision: https://phorge.dev.yugabyte.com/D54966

[BACKPORT 2024.2][#27056] docdb: Fixed tablet split vs RBS from the follower race Summary: There is a possibility of tablet split vs RBS from follower race: 1. Parent tablet leader peers A-C accept a SPLIT_OP (op_id: 1.4). Leader (A) and 1st follower (B) apply SPLIT_OP, 2nd follower (C) doesn't apply it yet. 2. Parent tablet leader (node A) accepts CHANGE_CONFIG_OP (op_id: 1.5) to add a fourth peer (D) but doesn't apply it yet. 3. Parent tablet 2nd follower (C) still hasn't yet applied SPLIT_OP (1.4). 4. RBS for parent tablet peer (D) starts from the follower (C) and tablet metadata (tablet_data_state == TABLET_DATA_READY) is downloaded. 5. Parent tablet peers A-C completed applying the SPLIT_OP (1.4), child tablets have Raft config with 3 peers. 6. Parent tablet peers A-C apply CHANGE_CONFIG_OP (1.5) and now have committed Raft config with 4 peers. 7. Parent tablet peer D does local bootstrap and replays SPLIT_OP (1.4) as part of bootstrap. Due to tablet_data_state is TABLET_DATA_READY but not TABLET_DATA_SPLIT_COMPLETED replay does SPLIT_OP apply and creates child tablet peer. After that, 4th child tablet peer (D) is not a part of Raft group (which has 3 peers) and therefore is not receiving consensus updates from leader. This change fixes this race by rejecting RBS from the follower that is in progress of applying SPLIT_OP and RBS attempt will be retried later. Original commit: 78af3a208a6bfc005eedc0ed6e410d22f4d24758 / D48853 **Upgrade/Rollback safety:** New error code will be printed by old nodes as just number in case of RBS failure during upgrade but this is safe. Test Plan: TabletSplitITest.SplitWithParentTabletRbsFromFollower, TabletSplitITest.SplitWithParentTabletMove, RemoteBootstrapsFromNodeWithUncommittedSplitOp - 30 runs per each of asan/tsan/debug/release builds Reviewers: arybochkin Reviewed By: arybochkin Subscribers: ybase Tags: #jenkins-ready Differential Revision: https://phorge.dev.yugabyte.com/D54972

[BACKPORT 2025.1][#27056] docdb: Fixed tablet split vs RBS from the follower race Summary: There is a possibility of tablet split vs RBS from follower race: 1. Parent tablet leader peers A-C accept a SPLIT_OP (op_id: 1.4). Leader (A) and 1st follower (B) apply SPLIT_OP, 2nd follower (C) doesn't apply it yet. 2. Parent tablet leader (node A) accepts CHANGE_CONFIG_OP (op_id: 1.5) to add a fourth peer (D) but doesn't apply it yet. 3. Parent tablet 2nd follower (C) still hasn't yet applied SPLIT_OP (1.4). 4. RBS for parent tablet peer (D) starts from the follower (C) and tablet metadata (tablet_data_state == TABLET_DATA_READY) is downloaded. 5. Parent tablet peers A-C completed applying the SPLIT_OP (1.4), child tablets have Raft config with 3 peers. 6. Parent tablet peers A-C apply CHANGE_CONFIG_OP (1.5) and now have committed Raft config with 4 peers. 7. Parent tablet peer D does local bootstrap and replays SPLIT_OP (1.4) as part of bootstrap. Due to tablet_data_state is TABLET_DATA_READY but not TABLET_DATA_SPLIT_COMPLETED replay does SPLIT_OP apply and creates child tablet peer. After that, 4th child tablet peer (D) is not a part of Raft group (which has 3 peers) and therefore is not receiving consensus updates from leader. This change fixes this race by rejecting RBS from the follower that is in progress of applying SPLIT_OP and RBS attempt will be retried later. Original commit: 78af3a208a6bfc005eedc0ed6e410d22f4d24758 / D48853 **Upgrade/Rollback safety:** New error code will be printed by old nodes as just number in case of RBS failure during upgrade but this is safe. Test Plan: TabletSplitITest.SplitWithParentTabletRbsFromFollower, TabletSplitITest.SplitWithParentTabletMove, RemoteBootstrapsFromNodeWithUncommittedSplitOp - 30 runs per each of asan/tsan/debug/release builds Reviewers: arybochkin Reviewed By: arybochkin Subscribers: ybase, zdrudi Tags: #jenkins-ready Differential Revision: https://phorge.dev.yugabyte.com/D54946

[BACKPORT 2025.2][#27056] docdb: Fixed tablet split vs RBS from the follower race Summary: There is a possibility of tablet split vs RBS from follower race: 1. Parent tablet leader peers A-C accept a SPLIT_OP (op_id: 1.4). Leader (A) and 1st follower (B) apply SPLIT_OP, 2nd follower (C) doesn't apply it yet. 2. Parent tablet leader (node A) accepts CHANGE_CONFIG_OP (op_id: 1.5) to add a fourth peer (D) but doesn't apply it yet. 3. Parent tablet 2nd follower (C) still hasn't yet applied SPLIT_OP (1.4). 4. RBS for parent tablet peer (D) starts from the follower (C) and tablet metadata (tablet_data_state == TABLET_DATA_READY) is downloaded. 5. Parent tablet peers A-C completed applying the SPLIT_OP (1.4), child tablets have Raft config with 3 peers. 6. Parent tablet peers A-C apply CHANGE_CONFIG_OP (1.5) and now have committed Raft config with 4 peers. 7. Parent tablet peer D does local bootstrap and replays SPLIT_OP (1.4) as part of bootstrap. Due to tablet_data_state is TABLET_DATA_READY but not TABLET_DATA_SPLIT_COMPLETED replay does SPLIT_OP apply and creates child tablet peer. After that, 4th child tablet peer (D) is not a part of Raft group (which has 3 peers) and therefore is not receiving consensus updates from leader. This change fixes this race by rejecting RBS from the follower that is in progress of applying SPLIT_OP and RBS attempt will be retried later. Original commit: 78af3a208a6bfc005eedc0ed6e410d22f4d24758 / D48853 **Upgrade/Rollback safety:** New error code will be printed by old nodes as just number in case of RBS failure during upgrade but this is safe. Test Plan: TabletSplitITest.SplitWithParentTabletRbsFromFollower, TabletSplitITest.SplitWithParentTabletMove, RemoteBootstrapsFromNodeWithUncommittedSplitOp - 30 runs per each of asan/tsan/debug/release builds Reviewers: arybochkin Reviewed By: arybochkin Subscribers: zdrudi, ybase Tags: #jenkins-ready Differential Revision: https://phorge.dev.yugabyte.com/D54937

[BACKPORT 2026.1][#30642] YSQL: Adding column tablet_state, oid and making yb_tablet_metadata view global. Summary: `yb_tablet_metadata` view only shows database specific tablet info along with showcasing tablet info for tables with same `relname` in other databases. Intended behavior for `yb_tablet_metadata`: The view should showcase cluster-wide tablet metadata. All YCQL tablets and YSQL tablets from schema `pg_catalog` and `information_schema` to be excluded. Include `system.transactions` in the view. **FIX:** On the backend, a new `CatalogManager::GetTablets()` method iterates the master's `tablet_map_` directly instead of going through `GetTables()` → `table->GetTablets()`. This ensures colocated tables sharing a physical tablet produce a single row (the colocation parent) rather than duplicate rows per user table. Although, this fix will let any user see `relname` across the cluster without any restrictions. Masking will be enabled in the next diff. A new column `tablet_state` has been added and populated. This field can have possible values as `PREPARING`, `CREATING`, `REPLACED`, `RUNNING` and `DELETED`. NOTE: Tombstoned tablets (REPLACED or DELETED state) are still shown in the function/view up until they are removed by compaction. To avoid creating a new migration in the next diff where the function will have 3 new columns (`start_range`, `end_range`, `tablet_attrs`), made necessary changes in the migration file and marked them as nulls in `pg_yb_utils.c`. The subsequent revision will populate these columns. Also added a new field `oid` to the function. The `oid` in the view previously showed `relfilenode` with was derived from `object_uuid`. **Upgrade/Rollback safety:** This diff adds a new optional fields `tablet_state` and `pg_table_oid` in the protobuf message. This change is safe to upgrade/rollback. Original commit: ac7942d0357d9b4764eb05c21bf6cb7a7b9b78b9 / D52928 Test Plan: ./yb_build.sh release --java-test TestPgRegressMisc#testPgRegressMiscIndependent ./yb_build.sh release --java-test 'org.yb.pgsql.TestPgRegressRules#testPgRegressRules' ./yb_build.sh release --java-test TestYsqlUpgrade#migratingIsEquivalentToReinitdb ./yb_build.sh release --cxx-test yql-test --gtest_filter YqlTest.TabletMetadataViewsWithYcqlAndYsql ./yb_build.sh release --cxx-test pg_conn-test --gtest_filter PgConnTest.TabletMetadataConnectWithLeader ./yb_build.sh release --cxx-test pg_mini-test --gtest_filter PgMiniTest.TabletMetadataOidMatchesPgClass ./yb_build.sh release --cxx-test pg_mini-test --gtest_filter PgMiniTest.TabletMetadataCorrectnessWithHashPartitioning ./yb_build.sh release --cxx-test pg_mini-test --gtest_filter PgMiniTest.TabletMetadataStateColumn Reviewers: ishan.chhangani, asaha, cagrawal, aman.mangal, swapnil.kasaliwal Reviewed By: ishan.chhangani Subscribers: ybase, yql Differential Revision: https://phorge.dev.yugabyte.com/D55007

[BACKPORT 2025.2][#30642] YSQL: Adding column tablet_state, oid and making yb_tablet_metadata view global. Summary: `yb_tablet_metadata` view only shows database specific tablet info along with showcasing tablet info for tables with same `relname` in other databases. Intended behavior for `yb_tablet_metadata`: The view should showcase cluster-wide tablet metadata. All YCQL tablets and YSQL tablets from schema `pg_catalog` and `information_schema` to be excluded. Include `system.transactions` in the view. **FIX:** On the backend, a new `CatalogManager::GetTablets()` method iterates the master's `tablet_map_` directly instead of going through `GetTables()` → `table->GetTablets()`. This ensures colocated tables sharing a physical tablet produce a single row (the colocation parent) rather than duplicate rows per user table. Although, this fix will let any user see `relname` across the cluster without any restrictions. Masking will be enabled in the next diff. A new column `tablet_state` has been added and populated. This field can have possible values as `PREPARING`, `CREATING`, `REPLACED`, `RUNNING` and `DELETED`. NOTE: Tombstoned tablets (REPLACED or DELETED state) are still shown in the function/view up until they are removed by compaction. To avoid creating a new migration in the next diff where the function will have 3 new columns (`start_range`, `end_range`, `tablet_attrs`), made necessary changes in the migration file and marked them as nulls in `pg_yb_utils.c`. The subsequent revision will populate these columns. Also added a new field `oid` to the function. The `oid` in the view previously showed `relfilenode` which was derived from `object_uuid`. **Upgrade/Rollback safety:** This diff adds a new optional fields `tablet_state` and `pg_table_oid` in the protobuf message. This change is safe to upgrade/rollback. Original commit: ac7942d0357d9b4764eb05c21bf6cb7a7b9b78b9 / D52928 Test Plan: ./yb_build.sh release --java-test TestPgRegressMisc#testPgRegressMiscIndependent ./yb_build.sh release --java-test 'org.yb.pgsql.TestPgRegressRules#testPgRegressRules' ./yb_build.sh release --java-test TestYsqlUpgrade#migratingIsEquivalentToReinitdb ./yb_build.sh release --cxx-test yql-test --gtest_filter YqlTest.TabletMetadataViewsWithYcqlAndYsql ./yb_build.sh release --cxx-test pg_conn-test --gtest_filter PgConnTest.TabletMetadataConnectWithLeader ./yb_build.sh release --cxx-test pg_mini-test --gtest_filter PgMiniTest.TabletMetadataOidMatchesPgClass ./yb_build.sh release --cxx-test pg_mini-test --gtest_filter PgMiniTest.TabletMetadataCorrectnessWithHashPartitioning ./yb_build.sh release --cxx-test pg_mini-test --gtest_filter PgMiniTest.TabletMetadataStateColumn Reviewers: ishan.chhangani, asaha, cagrawal, swapnil.kasaliwal, aman.mangal Reviewed By: ishan.chhangani Subscribers: yql, ybase Differential Revision: https://phorge.dev.yugabyte.com/D55003

2026-06-28

[#30883] DocDB: Show vector index space usage in yb-master and yb-tserver UI pages Summary: The on-disk size breakdown shown on the yb-tserver and yb-master web UI pages did not account for vector indexes, so a table with a vector index under-reported its actual disk footprint. Add VectorLSM::OnDiskSize, which sums the sizes of the immutable chunk files currently on disk, and expose it through DocVectorIndex::OnDiskSize and VectorIndexList::OnDiskSize. TabletPeer aggregates the per-tablet vector index size into a new TabletOnDiskSizeInfo::vector_index_disk_size field, which is folded into active_on_disk_size and serialized in TabletStatusPB. Propagate the size to the master via a new vector_index_size field in TabletDriveStorageMetadataPB and TabletReplicaDriveInfo. The tserver tables/tablets pages and the master tables page now render a "Vector Indexes" line in the size breakdown, and the corresponding JSON endpoints expose vector_index_size. The HTML line is rendered only when the size is non-zero, so tables without a vector index are not cluttered with "Vector Indexes: 0B". Also make TabletVectorIndexes::List, TabletVectorIndexes::Collect, and TabletComponent::VectorIndexesList return the VectorIndexList wrapper instead of the raw docdb::DocVectorIndexesPtr. This removes the repeated VectorIndexList(...) wrapping at call sites; the few places that need the underlying pointer for the docdb write/apply path use the new VectorIndexList::impl accessor. --- **Upgrade / Rollback safety:** Adds field used by new functionality. The old code would just ignore it. New code would not show usage if information is received from node with an old code. --- _automated · Claude Code (Opus 4.8)_ Test Plan: ./yb_build.sh debug --cxx-test pg_vector_index-test --gtest_filter 'PgVectorIndexTest.OnDiskSize*' Manually verified on a local cluster: a table with an ybhnsw vector index reports a non-zero "Vector Indexes" size on both the tserver (:9000) and master (:7000) UI pages and in their JSON endpoints, while a table without a vector index reports none. Reviewers: arybochkin Reviewed By: arybochkin Subscribers: ybase, yql Tags: #jenkins-ready Differential Revision: https://phorge.dev.yugabyte.com/D54922

2026-06-27

[#30642] YSQL: Adding column tablet_state, oid and making yb_tablet_metadata view global. Summary: ## SUMMARY `yb_tablet_metadata` view only shows database specific tablet info along with showcasing tablet info for tables with same `relname` in other databases. Intended behavior for `yb_tablet_metadata`: The view should showcase cluster-wide tablet metadata. All YCQL tablets and YSQL tablets from schema `pg_catalog` and `information_schema` to be excluded. Include `system.transactions` in the view. **FIX:** On the backend, a new `CatalogManager::GetTablets()` method iterates the master's `tablet_map_` directly instead of going through `GetTables()` → `table->GetTablets()`. This ensures colocated tables sharing a physical tablet produce a single row (the colocation parent) rather than duplicate rows per user table. Although, this fix will let any user see `relname` across the cluster without any restrictions. Masking will be enabled in the next diff. A new column `tablet_state` has been added and populated. This field can have possible values as `PREPARING`, `CREATING`, `REPLACED`, `RUNNING` and `DELETED`. NOTE: Tombstoned tablets (REPLACED or DELETED state) are still shown in the function/view up until they are removed by compaction. ### Additional changes To avoid creating a new migration in the next diff where the function will have 3 new columns (`start_range`, `end_range`, `tablet_attrs`), made necessary changes in the migration file and marked them as nulls in `pg_yb_utils.c`. The subsequent revision will populate these columns. Also added a new field `oid` to the function. The `oid` in the view previously showed `relfilenode` with was derived from `object_uuid`. **Upgrade/Rollback safety:** This diff adds a new optional fields `tablet_state` and `pg_table_oid` in the protobuf message. This change is safe to upgrade/rollback. Test Plan: ./yb_build.sh release --java-test TestPgRegressMisc#testPgRegressMiscIndependent ./yb_build.sh release --java-test 'org.yb.pgsql.TestPgRegressRules#testPgRegressRules' ./yb_build.sh release --java-test TestYsqlUpgrade#migratingIsEquivalentToReinitdb ./yb_build.sh release --cxx-test yql-test --gtest_filter YqlTest.TabletMetadataViewsWithYcqlAndYsql ./yb_build.sh release --cxx-test pg_conn-test --gtest_filter PgConnTest.TabletMetadataConnectWithLeader ./yb_build.sh release --cxx-test pg_mini-test --gtest_filter PgMiniTest.TabletMetadataOidMatchesPgClass ./yb_build.sh release --cxx-test pg_mini-test --gtest_filter PgMiniTest.TabletMetadataCorrectnessWithHashPartitioning ./yb_build.sh release --cxx-test pg_mini-test --gtest_filter PgMiniTest.TabletMetadataStateColumn Reviewers: ishan.chhangani, kfranz Reviewed By: kfranz Subscribers: ybase, yql Differential Revision: https://phorge.dev.yugabyte.com/D52928

2026-06-26

[BACKPORT 2026.1][#27056] docdb: Fixed tablet split vs RBS from the follower race Summary: There is a possibility of tablet split vs RBS from follower race: 1. Parent tablet leader peers A-C accept a SPLIT_OP (op_id: 1.4). Leader (A) and 1st follower (B) apply SPLIT_OP, 2nd follower (C) doesn't apply it yet. 2. Parent tablet leader (node A) accepts CHANGE_CONFIG_OP (op_id: 1.5) to add a fourth peer (D) but doesn't apply it yet. 3. Parent tablet 2nd follower (C) still hasn't yet applied SPLIT_OP (1.4). 4. RBS for parent tablet peer (D) starts from the follower (C) and tablet metadata (tablet_data_state == TABLET_DATA_READY) is downloaded. 5. Parent tablet peers A-C completed applying the SPLIT_OP (1.4), child tablets have Raft config with 3 peers. 6. Parent tablet peers A-C apply CHANGE_CONFIG_OP (1.5) and now have committed Raft config with 4 peers. 7. Parent tablet peer D does local bootstrap and replays SPLIT_OP (1.4) as part of bootstrap. Due to tablet_data_state is TABLET_DATA_READY but not TABLET_DATA_SPLIT_COMPLETED replay does SPLIT_OP apply and creates child tablet peer. After that, 4th child tablet peer (D) is not a part of Raft group (which has 3 peers) and therefore is not receiving consensus updates from leader. This change fixes this race by rejecting RBS from the follower that is in progress of applying SPLIT_OP and RBS attempt will be retried later. Original commit: 78af3a208a6bfc005eedc0ed6e410d22f4d24758 / D48853 **Upgrade/Rollback safety:** New error code will be printed by old nodes as just number in case of RBS failure during upgrade but this is safe. Test Plan: TabletSplitITest.SplitWithParentTabletRbsFromFollower, TabletSplitITest.SplitWithParentTabletMove, RemoteBootstrapsFromNodeWithUncommittedSplitOp - 30 runs per each of asan/tsan/debug/release builds Reviewers: arybochkin Reviewed By: arybochkin Subscribers: ybase, zdrudi Tags: #jenkins-ready Differential Revision: https://phorge.dev.yugabyte.com/D54936

[pg19] cdc: stop reading removed pg_attribute.attcacheoff Summary: Upstream PG commit 02a8d0c45253eb54e57b1974c8627e5be3e1d852 ("Remove pg_attribute.attcacheoff column") removed the attcacheoff column. SysCatalogTable::ReadPgAttributeInfo still looked it up via ColumnIdByName("attcacheoff") -> "Couldn't find column attcacheoff in the schema" (common/schema.cc), which fails the GetUDTypeMetadata RPC. PgAttributePB.attcacheoff is a required proto field consumed by CDC. Dropping or relaxing the field would break cross-version CDC/xCluster: a pre-PG19 peer still compiled with attcacheoff as required fails to parse any message that omits it. Keep the field as required, stop reading the now-missing column in ReadPgAttributeInfo and set attcacheoff = -1. Upgrade/Rollback safety: The .proto change is comment-only -- no field is added, removed, renumbered, or retyped -- so the wire format is unchanged: PgAttributePB.attcacheoff stays a required int32 (tag 8). The only functional change (sys_catalog.cc) emits a constant 0 for that field instead of reading the now-removed catalog column. Test Plan: ./yb_build.sh release --cxx-test cdcsdk_consumption_consistent_changes-test --gtest_filter 'CDCSDKConsumptionConsistentChangesTest.TestCompactionWithReplicaIdentityDefault Reviewers: aagrawal Reviewed By: aagrawal Subscribers: sumukh.phalgaonkar, ybase Differential Revision: https://phorge.dev.yugabyte.com/D54855

[BACKPORT 2026.1][#31166] YSQL: Fix follower read time for parallel queries Summary: Original commit: 90f9f34b21ccfa980c8ffc4c814c0f9a0e7d3b3b / D52581 #### Problem Below is the log summary of a parallel query with vmodule=pg_session=2,pg_client_session=2 ``` # Leader (PID 589293) launches 2 parallel workers 04:36:25.391 [589293] DEBUG: YB: launching parallel workers 04:36:25.392 [589293] DEBUG: YB: launched 2 parallel workers # pg_session.cc: Leader sets read_time from follower read staleness 04:36:25.391 589293 pg_session.cc:900 Perform options: read_ht: 7281091302930841600 (physical: 1777610181379600) read_from_followers: true, read_time_serial_no: 50 # pg_session.cc: Worker 1 independently computes its own read_time (+46ms) 04:36:25.459 589717 pg_session.cc:900 Perform options: read_ht: 7281091303120982016 (physical: 1777610181426021) read_from_followers: true, read_time_serial_no: 50 # pg_session.cc: Worker 2 independently computes its own read_time (+56ms) 04:36:25.460 589718 pg_session.cc:900 Perform options: read_ht: 7281091303160123392 (physical: 1777610181435577) read_from_followers: true, read_time_serial_no: 50 # pg_client_session.cc: tserver receives all three distinct read_times # through the SAME session (pid 589293), confirming it uses whatever # read_time each worker specified: 04:36:25.393 Session id 6 (pid 589293): read_time={ physical: 1777610181379600 } # leader 04:36:25.460 Session id 6 (pid 589293): read_time={ physical: 1777610181426021 } # worker 1 04:36:25.460 Session id 6 (pid 589293): read_time={ physical: 1777610181435577 } # worker 2 # All workers finish 04:36:25.470 [589293] DEBUG: YB: all 2 parallel workers finished ``` From the log, we can see that all the workers use different read time. This can potentially cause an inconsistent read. #### Root Cause Follower reads computed their read time (now - staleness) in the YSQL backend using the backend's own clock. Each parallel worker is a separate backend, so workers in the same query computed different read times and could return inconsistent results. #### Fix Pick the read time in PgClientSession. When the PgClientSession picks the read point it lowers it by that staleness (SetFollowerReadTime). Every perform and parallel worker that shares the serial number - reads at the same time. Moreover, clock usage is removed from PgTxnManager. This is a step forward in removal of HybridClock from postgres backend since postgres backend is not involved in hybrid time propagation. PgApiImpl keeps a HybridClock only for ANALYZE sampling; see #16034. **Upgrade/Rollback safety:** Only changes proto used for communication between postgres and local tserver proxy. Test Plan: Jenkins ./yb_build.sh release --java-test 'org.yb.pgsql.TestPgFollowerReads#testBankInvariantWithParallelFollowerReads' Fails with the error below without the fix ``` INCONSISTENCY: expected total=100000 but got 100004 (diff=+4) after 611 transfers INCONSISTENCY: expected total=100000 but got 99993 (diff=-7) after 742 transfers INCONSISTENCY: expected total=100000 but got 99988 (diff=-12) after 1065 transfers INCONSISTENCY: expected total=100000 but got 99999 (diff=-1) after 1213 transfers INCONSISTENCY: expected total=100000 but got 100007 (diff=+7) after 1372 transfers INCONSISTENCY: expected total=100000 but got 100007 (diff=+7) after 1645 transfers ``` Reviewers: pjain, amartsinchyk, sanketh, smishra, dmitry Reviewed By: pjain Subscribers: yql, ybase, mtakahara Differential Revision: https://phorge.dev.yugabyte.com/D54934

2026-06-25

[#31166] YSQL: Fix follower read time for parallel queries Summary: #### Problem Below is the log summary of a parallel query with vmodule=pg_session=2,pg_client_session=2 ``` # Leader (PID 589293) launches 2 parallel workers 04:36:25.391 [589293] DEBUG: YB: launching parallel workers 04:36:25.392 [589293] DEBUG: YB: launched 2 parallel workers # pg_session.cc: Leader sets read_time from follower read staleness 04:36:25.391 589293 pg_session.cc:900 Perform options: read_ht: 7281091302930841600 (physical: 1777610181379600) read_from_followers: true, read_time_serial_no: 50 # pg_session.cc: Worker 1 independently computes its own read_time (+46ms) 04:36:25.459 589717 pg_session.cc:900 Perform options: read_ht: 7281091303120982016 (physical: 1777610181426021) read_from_followers: true, read_time_serial_no: 50 # pg_session.cc: Worker 2 independently computes its own read_time (+56ms) 04:36:25.460 589718 pg_session.cc:900 Perform options: read_ht: 7281091303160123392 (physical: 1777610181435577) read_from_followers: true, read_time_serial_no: 50 # pg_client_session.cc: tserver receives all three distinct read_times # through the SAME session (pid 589293), confirming it uses whatever # read_time each worker specified: 04:36:25.393 Session id 6 (pid 589293): read_time={ physical: 1777610181379600 } # leader 04:36:25.460 Session id 6 (pid 589293): read_time={ physical: 1777610181426021 } # worker 1 04:36:25.460 Session id 6 (pid 589293): read_time={ physical: 1777610181435577 } # worker 2 # All workers finish 04:36:25.470 [589293] DEBUG: YB: all 2 parallel workers finished ``` From the log, we can see that all the workers use different read time. This can potentially cause an inconsistent read. #### Root Cause Follower reads computed their read time (now - staleness) in the YSQL backend using the backend's own clock. Each parallel worker is a separate backend, so workers in the same query computed different read times and could return inconsistent results. #### Fix Pick the read time in PgClientSession. When the PgClientSession picks the read point it lowers it by that staleness (SetFollowerReadTime). Every perform and parallel worker that shares the serial number - reads at the same time. Moreover, clock usage is removed from PgTxnManager. This is a step forward in removal of HybridClock from postgres backend since postgres backend is not involved in hybrid time propagation. PgApiImpl keeps a HybridClock only for ANALYZE sampling; see #16034. **Upgrade/Rollback safety:** Only changes proto used for communication between postgres and local tserver proxy. Test Plan: Jenkins ./yb_build.sh release --java-test 'org.yb.pgsql.TestPgFollowerReads#testBankInvariantWithParallelFollowerReads' Fails with the error below without the fix ``` INCONSISTENCY: expected total=100000 but got 100004 (diff=+4) after 611 transfers INCONSISTENCY: expected total=100000 but got 99993 (diff=-7) after 742 transfers INCONSISTENCY: expected total=100000 but got 99988 (diff=-12) after 1065 transfers INCONSISTENCY: expected total=100000 but got 99999 (diff=-1) after 1213 transfers INCONSISTENCY: expected total=100000 but got 100007 (diff=+7) after 1372 transfers INCONSISTENCY: expected total=100000 but got 100007 (diff=+7) after 1645 transfers ``` Reviewers: pjain, amartsinchyk, sanketh, smishra, dmitry Reviewed By: pjain, dmitry Subscribers: mtakahara, ybase, yql Differential Revision: https://phorge.dev.yugabyte.com/D52581

2026-06-23

[#32004] DocDB: Return useful tablet metadata as a response to DeleteTablet Summary: - Extend `DeleteTablet` RPC response with tablet metadata to include the following so callers can verify after delete. Added previous/final data state, tablet_id, table name, hide_only, and per-directory paths with still_present_after. - Populate the response in `TSTabletManager::DeleteTablet` when an optional `DeleteTabletResponsePB*` is provided; existing internal callers unchanged (resp defaults to `nullptr`). - Update yb-ts-cli `delete_tablet` to print structured JSON on successful RPC (message, tablet_id, table_name, and details). **Upgrade/Rollback safety:** This change only adds optional fields to `DeleteTabletResponsePB` and `DeletedDirectoryPB` on the existing `DeleteTablet` admin RPC. It does not change request semantics, on-disk tablet metadata, catalog schema, or any format outside that response. Describe how this change handles upgrade and rollback of YugabyteDB. Only have optional fields on `DeleteTablet` response. Delete behaviour is unchanged. What Test/Preview/AutoFlag is used to guard the feature? No Test/Preview/AutoFlag, behavior is backward compatible by protobuf optional-field rules. Test Plan: Built using: ``` ./yb_build.sh debug ``` Tested output using: ``` $PWD/build/latest/bin/yb-ts-cli --server_address=127.0.0.1:9100 delete_tablet '<tablet_id' 'testing on feature branch' ``` Screenshots of change. Before: {F501795} After: {F501796} Also ran the following test suite: ``` ./yb_build.sh debug --cxx-test delete_table-test ``` Reviewers: mhaddad, bkolagani Reviewed By: mhaddad Subscribers: ybase Differential Revision: https://phorge.dev.yugabyte.com/D54131

[#32116] CDC: Add support to create table bound gRPC streams Summary: ##### Code changes summary Currently when a replication slot is created, retention barriers are setup and cdc_state table entries are written for all the tablets in the DB. In an environment where large number of tables are present and only a small subset is being used for CDC, this leads to unnecessary retention barrier setup. Also the cdc_state table is bloated with unnecessary entries. To prevent this, this diff introduces a mechanism to create gRPC streams that are bound to only specific tables at the time of their creation. To create such a stream, a comma separated list of table ids should be provided to the create_change_data_stream yb-admin command. The syntax is as follows: ``` ./yb-admin create_change_data_stream ysql.<DB-name> EXPLICIT CHANGE NOEXPORT_SNAPSHOT DYNAMIC_TABLES_DISABLED <comma separated table_ids> ``` For example: ``` ./yb-admin create_change_data_stream ysql.yugabyte EXPLICIT CHANGE NOEXPORT_SNAPSHOT DYNAMIC_TABLES_DISABLED 000034e1000030008000000000004000,000034e1000030008000000000004005 CDC Stream ID: 9fbec9b0395a2caacd48676d714ced0d ``` The table_ids are passed to the `CreateCDCStream` rpc by populating the `bound_table_ids` field in the `CDCSDKStreamCreateOptionsPB`. Only these table_ids are written to the stream metadata. The retention barriers are set on the tablets of only these tables, and their entries are written to the cdc_state table. Dynamic table addition is disabled for the table bound streams, meaning that the tables which can be polled using these streams is fixed at the stream creation. Any attmept to create such streams with dynamic tables enabled will fail. Also such streams can only be created for gRPC model. ##### Upgrade / Rollback safety Only proto change made in this diff is in `CDCSDKStreamCreateOptionsPB` which is a part of `CreateCDCStreamRequestPB`. The CreateCDCStream rpc flows from the tserver to the master. Since all the masters are upgraded before the tservers, this change is upgrade safe. Additionally the bound_table_ids field added in `CDCSDKStreamCreateOptionsPB` is an optional field. To make the repeated field optional it has been wrapped in a separate proto called `CDCSDKBoundTableIds`. A table bound stream created before rollback will continue to operate as intended after rollback, i.e post rollback user can use the table bound stream to get the change events from the tables present in the stream metadata. Hence this change is rollback safe. ##### Considerations for colocated tables If a stream is created such that it is bound to subset of colocated tables residing on the tablet, cdcsdk_producer will filter out the change records corresponding to other tables. ##### Considerations for connector NA Test Plan: New tests added: - ./yb_build.sh --cxx-test integration-tests_cdcsdk_ysql-test --gtest_filter 'CDCSDKYsqlTest.TestgRPCStreamBoundToSpecificTables' - ./yb_build.sh --cxx-test integration-tests_cdcsdk_ysql-test --gtest_filter 'CDCSDKYsqlTest.TestgRPCStreamBoundToSpecificColocatedTables' - ./yb_build.sh --cxx-test integration-tests_cdcsdk_ysql-test --gtest_filter 'CDCSDKYsqlTest.TestTableBoundStreamRejectsWithReplicationSlot' - ./yb_build.sh --cxx-test integration-tests_cdcsdk_ysql-test --gtest_filter 'CDCSDKYsqlTest.TestTableBoundStreamDisablesDynamicAddition' - ./yb_build.sh --cxx-test integration-tests_cdcsdk_ysql-test --gtest_filter 'CDCSDKYsqlTest.TestCreationOfgRPCStreamBoundToSpecificTablesViaYBAdmin' - ./yb_build.sh --cxx-test integration-tests_cdcsdk_ysql-test --gtest_filter 'CDCSDKYsqlTest.TestTableBoundStreamYbAdminRejectsTableFromDifferentNamespace' - ./yb_build.sh --cxx-test integration-tests_cdcsdk_ysql-test --gtest_filter 'CDCSDKYsqlTest.TestTableBoundStreamYbAdminRejectsTableFromDifferentNamespaceTestTableBoundStreamYbAdminRejectsDynamicTablesEnabled' Reviewers: xCluster, hsunder, skumar, asrinivasan, devansh.singhal, #db-approvers Reviewed By: asrinivasan, #db-approvers Subscribers: svc_phabricator, ybase Tags: #jenkins-ready Differential Revision: https://phorge.dev.yugabyte.com/D54632

2026-06-18

[BACKPORT 2025.2][#31958] docdb: Vector Index: Skip reverse mapping insertion during backfill Summary: Currently, vector index reverse mapping entries are inserted for all existing vectors whenever a new vector index is created. This approach inserts the same data repeatedly each time a new vector index is created for the same vector column, which is incorrect. Instead, the indexed table should own the reverse mapping and manage its insertion and deletion. Therefore, reverse mapping population should not happen during vector index backfill. This change addresses the backfill aspect and implements the logic to skip reverse mapping population while a vector index is being backfilled. The logic is currently disabled until the remaining parts of the reverse mapping ownership logic are implemented (https://github.com/yugabyte/yugabyte-db/issues/31886). For backward compatibility, `PgVectorIdxOptionsPB` is extended with a new field, `skip_reverse_mapping_backfill`, which is automatically unset in older releases because the field is not present there. This logic is required to ensure that the old approach is used when vector index backfill started before the upgrade (so some reverse mapping entries may have already been added) but had not yet completed by the time of the upgrade. Original commit: 390ba45656e51cb36ebaf821ceb87fe1a00e9d59 / D53921 **Upgrade/Rollback safety:** The change is backward compatible, and the absence of the new field is treated as the old behavior -- this is actually required to allow an in-progress backfill started before the upgrade to complete using the old approach, ensuring that no vectors are lost. Test Plan: yb_build.sh --cxx-test='TEST_F(PgVectorIndexUtilTest, BackfillSkipsReverseMapping)' yb_build.sh --cxx-test='TEST_F(PgVectorIndexUtilTest, BackfillWritesReverseMapping)' Reviewers: sergei, zdrudi Reviewed By: zdrudi Subscribers: ybase Differential Revision: https://phorge.dev.yugabyte.com/D54693

[BACKPORT 2026.1][#31958] docdb: Vector Index: Skip reverse mapping insertion during backfill Summary: Currently, vector index reverse mapping entries are inserted for all existing vectors whenever a new vector index is created. This approach inserts the same data repeatedly each time a new vector index is created for the same vector column, which is incorrect. Instead, the indexed table should own the reverse mapping and manage its insertion and deletion. Therefore, reverse mapping population should not happen during vector index backfill. This change addresses the backfill aspect and implements the logic to skip reverse mapping population while a vector index is being backfilled. The logic is currently disabled until the remaining parts of the reverse mapping ownership logic are implemented (https://github.com/yugabyte/yugabyte-db/issues/31886). For backward compatibility, `PgVectorIdxOptionsPB` is extended with a new field, `skip_reverse_mapping_backfill`, which is automatically unset in older releases because the field is not present there. This logic is required to ensure that the old approach is used when vector index backfill started before the upgrade (so some reverse mapping entries may have already been added) but had not yet completed by the time of the upgrade. Original commit: 390ba45656e51cb36ebaf821ceb87fe1a00e9d59 / D53921 **Upgrade/Rollback safety:** The change is backward compatible, and the absence of the new field is treated as the old behavior -- this is actually required to allow an in-progress backfill started before the upgrade to complete using the old approach, ensuring that no vectors are lost. Test Plan: yb_build.sh --cxx-test='TEST_F(PgVectorIndexUtilTest, BackfillSkipsReverseMapping)' yb_build.sh --cxx-test='TEST_F(PgVectorIndexUtilTest, BackfillWritesReverseMapping)' Reviewers: sergei, zdrudi Reviewed By: zdrudi Subscribers: ybase Differential Revision: https://phorge.dev.yugabyte.com/D54692

2026-06-16

[BACKPORT 2026.1][PLAT-20583][PLAT-19353] Remove Ansible libraries from Yugabyte Anywhere image Summary: Original diffs: 1. https://phorge.dev.yugabyte.com/D52420 (305866f4e87833d4925f6f9c25c3589204b2c3bc) - Remove ansible. 2. https://phorge.dev.yugabyte.com/D52297 (7fc952448ae5e58ee1dab87e75bd5e1b6e013bfd) - Make node agent mandatory. 3. https://phorge.dev.yugabyte.com/D52422 (118e77ca6f22beecf4a83985ca45071c22583756) - Fixes for the changes. 4. https://phorge.dev.yugabyte.com/D52433 (840389244027c372e26ea81f80e3454f38ad6b1d) - UT fix followup. 5. https://phorge.dev.yugabyte.com/D53143 (36519c9313f862f6882e241d45a0a40cdb1dc68f) - ssk key rotation fix. 6. https://phorge.dev.yugabyte.com/D54133 (f67bca88bbb4aa9aaa42f73a10a3260ee52d069a) - Empty optional must be returned if node-agent is not active 7. https://phorge.dev.yugabyte.com/D54120 (d1a6791e0831c8063dfd00103e3c643323436255) - Adjust keep-alive time parameters to avoid GOAWAY received This is a prerequisite before removing ansible because if node-agent is not found, the code flow for configure defaults to ansible. If it is ignored, it can lead of skipping tasks that is very dangerous! Note: Node agent is already mandatory now. [PLAT-20583] Fix for leaked instances: Make node agent mandatory at all applicable call sites to avoid defaulting to the legacy code with ansible Local change was not in the commit. Instance must be terminated for non-onprem. UT fix caused by PLAT-20583 making node agent mandatory. Calls to node manager is skipped for gflags upgrade. This leak fix https://phorge.dev.yugabyte.com/D52422 adds back some calls as destroy is down to NodeManager to terminate VMs. [PLAT-19353] Remove Ansible libraries from Yugabyte Anywhere image Deleted all the references to ansible including roles, python files. A dummy install_ansible_requirements.sh is kept as it is invoked during build (workaround for now). Conflict resolution. Test Plan: Manually tested. Will also wait for itests. Manually tested. Passed UTs locally. itests must pass. Manual tests passed locally - edit, create, resize, vm image upgrade. Reviewers: amalyshev, spothuraju, skhilar, yshchetinin, nbhatia, vkumar, muthu, anijhawan, anabaria Reviewed By: amalyshev, anijhawan Subscribers: nikhil, yugaware Differential Revision: https://phorge.dev.yugabyte.com/D52824

[BACKPORT 2026.1][#30578] YSQL: Reset auto-analyze mutation counts after a manual ANALYZE Summary: Reset auto-analyze mutation counters after user-initiated ANALYZE Original commit: b6bbc7a663df2252865be498931bd4484ac7c224 / #31849 No conflicts on backport ## Problem User-initiated ANALYZE does not reset the auto-analyze service's accumulated mutation count. The next periodic tick can therefore trigger an auto-analyze on a table the user just analyzed, wasting work. ## Solution Add ResetAutoAnalyzeMutationCounters RPC (pg_client.proto / pg_client_service) called from do_analyze_rel after a full ANALYZE. The RPC resets the YCQL service table mutations to 0 for the analyzed table. Reset is gated to match PostgreSQL's semantics for updating its changes_since_analyze counters (fires only when no column list given) and is suppressed for auto-analyze service's internal connections and other internal backends. Auto analyze continues to reset mutations using the existing logic to subtract mutations on its side. Mutations are stored in a separate YCQL table so the mutations update is not transactional. A failed manual ANALYZE can still reset counters to 0. Fixing this is tracked in #32081. ## Race handling Extract mutation-update logic into helper functions in pg_auto_analyze_table that use conditional YCQL writes for better reuse. - ResetPgAutoAnalyzeMutationCounts: sets mutations to 0 (IF EXISTS) - SubtractPgAutoAnalyzeMutationCounts: subtracts snapshot mutations with clamping. Emits two conditional writes per table — one sets to 0 if current < snapshot, another subtracts if current >= snapshot. This prevents the count from going negative when a manual-ANALYZE reset races with auto-analyze post-ANALYZE mutation subtraction. UpdateTableMutationsAfterAnalyze refactored to use SubtractPgAutoAnalyzeMutationCounts instead of building operations directly. ## Wire-format changes New RPC: PgClientService.ResetAutoAnalyzeMutationCounters (pg_client.proto). Additive only — no existing field numbers or messages altered. Upgrade/rollback safety: New RPC is between PG and local tserver so no upgrade/rollback issues. Test Plan: - PgAutoAnalyzeTest.ManualAnalyzeResetsMutationCount: covers ANALYZE, ANALYZE(col), VACUUM ANALYZE, VACUUM ANALYZE(col), asserting reset only when no column list provided. - PgAutoAnalyzeTest.InternalAnalyzeDoesNotResetMutationCount: confirms internal connections do not trigger reset. - PgAutoAnalyzeTest.ManualAnalyzePartitionedTableResetsPartitionMutationCounts: exercises the partitioned-table path. - Existing PgAutoAnalyzeTest cases continue to pass. Reviewers: kfranz, pjain Reviewed By: kfranz Differential Revision: https://phorge.dev.yugabyte.com/D54272

2026-06-15

[PLAT-21159] Allow audit log retention for certain amount of days Summary: Allow keeping audit logs on the DB nodes for compliancy This is a requirement from customers, who don't want to export audit logs to external systems, but want to still be compliant with security requirements. Basically, what it does is - before gzipping postgres (YSQL) or tserser (YCQl) logs we're copying over all the audit line logs from the file to ./audit/[ysql or ycql]/......audit.log file. Now, we zip this new audit log file as well as the original log file. Original log file will be cleaned up, while audit log gzipped file will be kept on the node until the configured amount of days pass - and deleted after that. Number of days are controlled via additional autid logs setting. If the setting is not configured (or is configured to 0) - we keep the old behaviour. Test Plan: Installed older YBA release Created universe. Configured YSQL audit logs via the UI. Upgraded YBA. Re-configured YSQL audit logs via the UI to set the retention inetrval to 1 day. Make sure log rotation script was updated + otel-collector/log_cleanup_env file contains the new setting. Wait for postgres log file to be gzipped. Make sure audit log file gzip was created as well. Wait for 1 day to pass. Make sure the audit log file was deleted after the script run. Reviewers: vbansal, #yba-api-review! Reviewed By: vbansal Subscribers: yugaware Differential Revision: https://phorge.dev.yugabyte.com/D54457

2026-06-12

[BACKPORT 2025.1][#31951] yb-admin: Add a key range to get_table_hash Summary: get_table_hash hashes a whole table. To narrow a detected inconsistency (e.g. during xCluster verification) down to where the data diverged, allow the scan to be restricted to a logical partition-key sub-range. Add optional start_key / end_key arguments (raw partition keys, hex-encoded on the command line -- the same encoding shown as partition_key_start / partition_key_end by list_tablets). start_key is inclusive, end_key is exclusive; an empty bound means unbounded on that side. The range is logical, so it is cluster-independent: each cluster resolves it to whatever tablets it owns, which is what makes it usable for cross-cluster comparison even when tablet boundaries differ. - DumpTabletDataRequestPB gains start_key and end_key. - tablet::DumpTabletData builds each table's encoded bound as [table prefix][encoded partition key]: the table prefix (cotable_id / colocation_id bytes; empty for a non-colocated table) places the bound in this table's slice of the tablet, and the encoded partition key narrows within it. An empty user bound leaves that side at the iterator's natural table boundary. - yb-admin's client skips tablets that do not overlap the requested range and forwards the bounds unchanged to every overlapping tablet. A key range scopes a single table, so it requires a concrete table_id: combined with a colocation parent id (which hashes every table in the tablet) it would be ambiguous, and is rejected with InvalidArgument. Bad input is rejected up front rather than silently hashing the wrong range: the CLI rejects malformed hex and an inverted range (start_key >= end_key), and the server rejects a bound that is not a 2-byte hash for a hash-partitioned table. Builds on #31952 (D53893, landed), whose per-table scoping this composes with: pass a child colocated table id to hash one colocated table over a key range. For #31951. **Upgrade/Rollback safety:** No persistent or on-disk format change, and no AutoFlag is needed. start_key and end_key are optional fields on DumpTabletDataRequestPB, an on-demand admin RPC used only by `yb-admin get_table_hash`; they are not written to disk, the WAL, or sys.catalog, and are absent from any consensus/replication path. Mixed-version behavior is safe: an old yb-admin sends no bounds, so a new tserver scans the full table (unchanged); a new yb-admin's bounds are ignored by an old tserver, which scans the full table rather than erroring -- run a version-matched yb-admin when using start_key/end_key. No state is persisted, so rollback has nothing to undo. Original commit: 208a0f88102d23f40365f4ba3d2c5005eddee88e / D53900 Test Plan: AdminCliTest.TestGetTableXorHashKeyRange (non-colocated YCQL hash table): explicit empty bounds reproduce the full-table totals, and a complementary 0x8000 split partitions the rows so counts sum and hashes XOR back to the full totals. PgLibPqTest.TestGetTableXorHashColocatedKeyRange (colocated, range-only table): derives real mid-data split keys for id=4 and id=8 from the server's own partitioning (a throwaway non-colocated SPLIT AT VALUES ((4), (8)) table), then -- passing the child colocated table id -- splits the table into three disjoint segments [-inf, key(4)) / [key(4), key(8)) / [key(8), +inf) (the middle one specifies both bounds), verifying exact per-segment row counts (3/4/3) and that the segments recombine (counts sum, hashes XOR) to the full totals. Also asserts that a key range against the colocation parent table id is rejected. Verified locally (debug/clang21): AdminCliTest.TestGetTableXorHashKeyRange passes. Relying on CSI for the full suite (incl. PgLibPqTest.TestGetTableXorHashColocatedKeyRange). Reviewers: jhe, #db-approvers Reviewed By: jhe, #db-approvers Subscribers: svc_phabricator Differential Revision: https://phorge.dev.yugabyte.com/D54292

[#27119] DocDB: Table locks: Introduce support for WaitForLockers Summary: **Background** Postgres supports a functionality `WaitForLockers` where the backend actively waits for other backends with active conflicting locks on the desired objects, and returns once the earlier snapshotted backends don't hold conflicting locks anymore. Currently, it is being invoked on the following paths 1. `REINDEX CONCURRENTLY` - YB doesn't support this as of today 2. `DROP INDEX CONCURRENTLY` - again, YB doesn't support this as of today. 3. `CREATE INDEX CONCURRENTLY` - we seem to be executing `WaitForLockers` only when `!IsYugaByteEnabled()` is true. Ideally, we would want to replace the current WaitForYsqlBackends on the concurrent index creation path with WaitForLockers some day (the former is a more generic version of the latter - WaitForYsqlBackends waits for ALL backends with stale catalog version, while WaitForLockers would wait for only backends with current active conflicting locks on desired objects). Created https://github.com/yugabyte/yugabyte-db/issues/31534 to track this effort. 4. `ALTER TABLE DETACH PARTITION CONCURRENTLY` - this seems to be the only active usage of the api. Put up a test case which fails without the functionality - the DML could have a stale view of the table metadata etc, but couldn't produce an data inconsistency issue without the support for WaitForLockers **Solution** **//WaitForLockers//** This revision introduces support for `WaitForLockers` api. For global lock acquires, a working mechanism exists where the host tserver forwards the lock request to the master, and the master then fans it out to all tservers with live ysql lease, and then keeps retrying until the ysql lease is valid or an ack is received. The same is leveraged for `WaitForLockers` as well as follows, 1. Master leader fans out the `WaitForLockers` request to every tserver with a valid ysql lease 2. The tserver then takes a snapshot of the active transactions with conflicting locks on the desired objects, and registers a shared callback that gets invoked on the lock release path for these transactions (which happens after transactions finish). 3. The tserver acks to the master after all such transactions finish. We could have some false positive transactions recorded in step 2. If the backend is on a later catalog version than that calling `WaitForLockers`, then the txn corresponding to the backend is not relevant for us. Yet, we wait on that txn even when we need not do so (since there's no real easy way to distinguish this). Created https://github.com/yugabyte/yugabyte-db/issues/31803 to track this effort. //Failure semantics// - If the tserver loses its ysql lease, then the master stops retrying the req at this tserver since existing backends would be killed and new ones would get the necessary invalidation messages on later lock acquires. - If the master loses it leadership by the time all `WaitForLockers` return from the tservers, it returns an error to the host tserver's client which retries the request against the new master leader. Defined a new wait state `kWaitForLockersMultiple` as with all pggate rpcs which represents that the call is outstanding/being processed at the tserver/docdb side. **//ALTER TABLE ... CONCURRENTLY//** Prior to this revision, `YBCPrepareAlterTableCmd` took `AccessExclusive` locks immaterial of the alter type. The upstream code calculates the locktype that needs to be taken based on the alter type, which wasn't being honored here. Part of the reason might be due to the fact that all of these locks were a no-op back when object locking wasn't supported. This revision refactors the function to take input `lockmode` and lock the tables in that mode, thus //bringing ALTER TABLE concurrency to parity with PostgreSQL//. `lockmode` is computed in `utility.c` ``` * Figure out lock mode, and acquire lock. This also does * basic permissions checks, so that we won't wait for a * lock on (for example) a relation on which we have no * permissions. */ lockmode = AlterTableGetLockLevel(atstmt->cmds); ``` Note that the above analysis of `alter ... concurrently` not being concurrent is true just with table locks enabled, and this revision fixes that behavior. Prior to table locks, ALTER wouldn't wait on DMLs, and that behavior is still preserved when table locking is disabled. **Upgrade / Downgrade section** The new rpcs and usage of the proto messages is protected under the table locks gflag itself which hasn't gone with on by default in any major release yet. So there should not be any issue of upgrade/downgrade safety. **Additional Note** Filed https://github.com/yugabyte/yugabyte-db/issues/31532 to track the discrepancy of observing stale partition hierarchy in YB as compared to PG after the first phase of `ALTER DETACH CONCURRENTLY` commits and before phase 2 finishes. Test Plan: Jenkins Added test cases that assert basic functionality, concurrent `WaitForLockersMultiple` requests, scenarios where `WaitForLockers` times out, master leader loses leadership, etc. Also ported pg isolation test suite for `ALTER DETACH CONCURRENTLY` with some modifications and a follow-up issue. ``` ./yb_build.sh --cxx-test='TEST_F(ObjectLockTest, TestWaitForLockersMultiple) {' ./yb_build.sh --cxx-test='TEST_F(ExternalObjectLockTest, TestWaitForLockers) {' ./yb_build.sh --cxx-test object_lock-test --gtest_filter *WaitForLockersAcrossMasterFailover* ./yb_build.sh --java-test org.yb.pgsql.TestPgRegressIsolationObjectLocking#testPgRegress ``` Reviewers: amitanand, sanketh, pjain, patnaik.balivada, smishra Reviewed By: patnaik.balivada Subscribers: myang, ybase, yql Differential Revision: https://phorge.dev.yugabyte.com/D52015

2026-06-11

[BACKPORT 2025.2][#31594] DocDB: Table Locks: Fix false deadlock caused by re-use of session level txn. Summary: In one of the itests, we saw create index statements running into false deadlocks. It is due to reuse of the session level transaction. The flow is something as below ``` s1 s2 $create index idx1 on test(v1); - phase1, acquires session object lock on relation test in the end associated with session_txn1 - phase2 $analyze test; - waits on session_txn1 - phase3, releases session object lock on relation test - unblocks ... analyze in progress $create index idx1 on test(v1); - ddl txn tries acquiring conflicting object lock on relation test leads to ddl_txn2 -> analyze_txn ``` The above leads to a deadlock. Since the session level txn has a inherent dependency onto the host txn, we end up with a false cycle ``` session_txn1 -> ddl_txn2 -> analyze_txn -> session_txn1 ``` This is due to the earlier edge `analyze_txn -> session_txn1` not being pruned. The wait-for edges in YB are pruned either when the corresponding subtxn rollsback or the txn itself isn't active anymore. This revision fixes the above false deadlock issue by associating the session object locks from different transactions to different subtxns. In specific, - when acquiring a session level object lock, if one doesn't already exist, bump up the active subtxn and acquire the session object locks against it (stored in `subtxn_with_session_object_locks_`). - when acquiring a session level object lock, if an active subtxn with session locks already exists, acquire them against the same subtxn - on release all session locks, rollback to `subtxn_with_session_object_locks_` and bump the active subtxn id The same session level txn is also used for acquiring session advisory locks, and they need to honored despite the above logic of rolling back specific subtxns. The above logic takes care of this since we increment active subtxn when acquiring a session object lock for the first time, ensuring that all active advisory locks associated with the current subtxn (before the increment) would still remain active. Additionally an `SCHECK` is introduced on the acquire session advisory lock path which ensures that we don't serve any session advisory lock requests when the session level txn is holding active session object locks. This is necessary as release all session object locks rollsback the subtxn. This SCHECK itself is expected to never fail since session advisory locks are user level and cannot be issue in the middle of execution of a DDL (like CREATE INDEX) which use session object locks. **Upgrade/Downgrade safety** Added new field to the proto message which is used for ysql <-> local tserver communication alone. No upgrade/downgrade impact. Additionally, the usage of the feature (table locking) is disabled by default in all existing major releases. Original commit: f37604b92442062f18f1f39e1525e86a67925dc5 / D53287 Test Plan: Jenkins ./yb_build.sh --cxx-test pg_object_locks-test --gtest_filter PgObjectLocksTest.ConsecutiveCreateIndexDontDeadlock Reviewers: amitanand, #db-approvers, hsunder Reviewed By: amitanand, #db-approvers, hsunder Subscribers: svc_phabricator, ybase, yql Differential Revision: https://phorge.dev.yugabyte.com/D53950

[BACKPORT 2025.2][#29647] YSQL: Avoid master query on PG startup for colocation info Summary: Original commit: c74b3d13b6d3 / D53350 Every PG backend startup queries the master to learn whether its target database is colocated (added a while ago in f0082094c4583989665b72e78180d5874600090e). This diff introduces a tserver-side cache by db oid for this information to avoid this RPC. DB colocation information does not change once the db is created. 1. When multiple backends look up this information at the same time, only one outstanding query is made to the master and the remaining backends use the result of that query (both in success & failure cases). 2. If a query to the master for this info fails, there is no negative cache of the failed result. 3. When a db is dropped, heartbeats to the tserver inform of this event - this is currently used to keep the db oid -> catalog version map accurate and remove entries from it. The same path is used to also keep this cache up to date for dropped dbs. **Upgrade/Rollback safety:** Test Plan: No new test is introduced. A manual test confirmed the behavior for regular hits/misses/drop db. Jenkins tests confirm that no breakage happened. Reviewers: myang, zdrudi Reviewed By: zdrudi Differential Revision: https://phorge.dev.yugabyte.com/D53795

2026-06-10

[BACKPORT 2024.2][#31951] yb-admin: Add a key range to get_table_hash Summary: Backport note: two conflicts, both from newer master code absent on 2024.2. In yb-admin_cli.cc, dropped the `unsafe_release_object_locks_global` command (its args, action, and REGISTER_COMMAND_HIDDEN) -- pre-existing object-locking master code, not part of this change, and ReleaseObjectLocksGlobal does not exist on this branch. Kept this change's DecodeHexPartitionKey helper, the new includes, and the key-range args. The yb-admin-test.cc conflict was this change's new tests landing next to branch-specific context; took the tests as-is. No other changes. get_table_hash hashes a whole table. To narrow a detected inconsistency (e.g. during xCluster verification) down to where the data diverged, allow the scan to be restricted to a logical partition-key sub-range. Add optional start_key / end_key arguments (raw partition keys, hex-encoded on the command line -- the same encoding shown as partition_key_start / partition_key_end by list_tablets). start_key is inclusive, end_key is exclusive; an empty bound means unbounded on that side. The range is logical, so it is cluster-independent: each cluster resolves it to whatever tablets it owns, which is what makes it usable for cross-cluster comparison even when tablet boundaries differ. - DumpTabletDataRequestPB gains start_key and end_key. - tablet::DumpTabletData builds each table's encoded bound as [table prefix][encoded partition key]: the table prefix (cotable_id / colocation_id bytes; empty for a non-colocated table) places the bound in this table's slice of the tablet, and the encoded partition key narrows within it. An empty user bound leaves that side at the iterator's natural table boundary. - yb-admin's client skips tablets that do not overlap the requested range and forwards the bounds unchanged to every overlapping tablet. A key range scopes a single table, so it requires a concrete table_id: combined with a colocation parent id (which hashes every table in the tablet) it would be ambiguous, and is rejected with InvalidArgument. Bad input is rejected up front rather than silently hashing the wrong range: the CLI rejects malformed hex and an inverted range (start_key >= end_key), and the server rejects a bound that is not a 2-byte hash for a hash-partitioned table. Builds on #31952 (D53893, landed), whose per-table scoping this composes with: pass a child colocated table id to hash one colocated table over a key range. For #31951. **Upgrade/Rollback safety:** No persistent or on-disk format change, and no AutoFlag is needed. start_key and end_key are optional fields on DumpTabletDataRequestPB, an on-demand admin RPC used only by `yb-admin get_table_hash`; they are not written to disk, the WAL, or sys.catalog, and are absent from any consensus/replication path. Mixed-version behavior is safe: an old yb-admin sends no bounds, so a new tserver scans the full table (unchanged); a new yb-admin's bounds are ignored by an old tserver, which scans the full table rather than erroring -- run a version-matched yb-admin when using start_key/end_key. No state is persisted, so rollback has nothing to undo. Original commit: 208a0f88102d23f40365f4ba3d2c5005eddee88e / D53900 Test Plan: AdminCliTest.TestGetTableXorHashKeyRange (non-colocated YCQL hash table): explicit empty bounds reproduce the full-table totals, and a complementary 0x8000 split partitions the rows so counts sum and hashes XOR back to the full totals. PgLibPqTest.TestGetTableXorHashColocatedKeyRange (colocated, range-only table): derives real mid-data split keys for id=4 and id=8 from the server's own partitioning (a throwaway non-colocated SPLIT AT VALUES ((4), (8)) table), then -- passing the child colocated table id -- splits the table into three disjoint segments [-inf, key(4)) / [key(4), key(8)) / [key(8), +inf) (the middle one specifies both bounds), verifying exact per-segment row counts (3/4/3) and that the segments recombine (counts sum, hashes XOR) to the full totals. Also asserts that a key range against the colocation parent table id is rejected. Verified locally (debug/clang21): AdminCliTest.TestGetTableXorHashKeyRange passes. Relying on CSI for the full suite (incl. PgLibPqTest.TestGetTableXorHashColocatedKeyRange). Reviewers: jhe Reviewed By: jhe Differential Revision: https://phorge.dev.yugabyte.com/D54293

2026-06-09

[BACKPORT 2025.2][#31951] yb-admin: Add a key range to get_table_hash Summary: get_table_hash hashes a whole table. To narrow a detected inconsistency (e.g. during xCluster verification) down to where the data diverged, allow the scan to be restricted to a logical partition-key sub-range. Add optional start_key / end_key arguments (raw partition keys, hex-encoded on the command line -- the same encoding shown as partition_key_start / partition_key_end by list_tablets). start_key is inclusive, end_key is exclusive; an empty bound means unbounded on that side. The range is logical, so it is cluster-independent: each cluster resolves it to whatever tablets it owns, which is what makes it usable for cross-cluster comparison even when tablet boundaries differ. - DumpTabletDataRequestPB gains start_key and end_key. - tablet::DumpTabletData builds each table's encoded bound as [table prefix][encoded partition key]: the table prefix (cotable_id / colocation_id bytes; empty for a non-colocated table) places the bound in this table's slice of the tablet, and the encoded partition key narrows within it. An empty user bound leaves that side at the iterator's natural table boundary. - yb-admin's client skips tablets that do not overlap the requested range and forwards the bounds unchanged to every overlapping tablet. A key range scopes a single table, so it requires a concrete table_id: combined with a colocation parent id (which hashes every table in the tablet) it would be ambiguous, and is rejected with InvalidArgument. Bad input is rejected up front rather than silently hashing the wrong range: the CLI rejects malformed hex and an inverted range (start_key >= end_key), and the server rejects a bound that is not a 2-byte hash for a hash-partitioned table. Builds on #31952 (D53893, landed), whose per-table scoping this composes with: pass a child colocated table id to hash one colocated table over a key range. For #31951. **Upgrade/Rollback safety:** No persistent or on-disk format change, and no AutoFlag is needed. start_key and end_key are optional fields on DumpTabletDataRequestPB, an on-demand admin RPC used only by `yb-admin get_table_hash`; they are not written to disk, the WAL, or sys.catalog, and are absent from any consensus/replication path. Mixed-version behavior is safe: an old yb-admin sends no bounds, so a new tserver scans the full table (unchanged); a new yb-admin's bounds are ignored by an old tserver, which scans the full table rather than erroring -- run a version-matched yb-admin when using start_key/end_key. No state is persisted, so rollback has nothing to undo. Original commit: 208a0f88102d23f40365f4ba3d2c5005eddee88e / D53900 Test Plan: AdminCliTest.TestGetTableXorHashKeyRange (non-colocated YCQL hash table): explicit empty bounds reproduce the full-table totals, and a complementary 0x8000 split partitions the rows so counts sum and hashes XOR back to the full totals. PgLibPqTest.TestGetTableXorHashColocatedKeyRange (colocated, range-only table): derives real mid-data split keys for id=4 and id=8 from the server's own partitioning (a throwaway non-colocated SPLIT AT VALUES ((4), (8)) table), then -- passing the child colocated table id -- splits the table into three disjoint segments [-inf, key(4)) / [key(4), key(8)) / [key(8), +inf) (the middle one specifies both bounds), verifying exact per-segment row counts (3/4/3) and that the segments recombine (counts sum, hashes XOR) to the full totals. Also asserts that a key range against the colocation parent table id is rejected. Verified locally (debug/clang21): AdminCliTest.TestGetTableXorHashKeyRange passes. Relying on CSI for the full suite (incl. PgLibPqTest.TestGetTableXorHashColocatedKeyRange). Reviewers: jhe Reviewed By: jhe Differential Revision: https://phorge.dev.yugabyte.com/D54291

[BACKPORT 2026.1][#31951] yb-admin: Add a key range to get_table_hash Summary: get_table_hash hashes a whole table. To narrow a detected inconsistency (e.g. during xCluster verification) down to where the data diverged, allow the scan to be restricted to a logical partition-key sub-range. Add optional start_key / end_key arguments (raw partition keys, hex-encoded on the command line -- the same encoding shown as partition_key_start / partition_key_end by list_tablets). start_key is inclusive, end_key is exclusive; an empty bound means unbounded on that side. The range is logical, so it is cluster-independent: each cluster resolves it to whatever tablets it owns, which is what makes it usable for cross-cluster comparison even when tablet boundaries differ. - DumpTabletDataRequestPB gains start_key and end_key. - tablet::DumpTabletData builds each table's encoded bound as [table prefix][encoded partition key]: the table prefix (cotable_id / colocation_id bytes; empty for a non-colocated table) places the bound in this table's slice of the tablet, and the encoded partition key narrows within it. An empty user bound leaves that side at the iterator's natural table boundary. - yb-admin's client skips tablets that do not overlap the requested range and forwards the bounds unchanged to every overlapping tablet. A key range scopes a single table, so it requires a concrete table_id: combined with a colocation parent id (which hashes every table in the tablet) it would be ambiguous, and is rejected with InvalidArgument. Bad input is rejected up front rather than silently hashing the wrong range: the CLI rejects malformed hex and an inverted range (start_key >= end_key), and the server rejects a bound that is not a 2-byte hash for a hash-partitioned table. Builds on #31952 (D53893, landed), whose per-table scoping this composes with: pass a child colocated table id to hash one colocated table over a key range. For #31951. **Upgrade/Rollback safety:** No persistent or on-disk format change, and no AutoFlag is needed. start_key and end_key are optional fields on DumpTabletDataRequestPB, an on-demand admin RPC used only by `yb-admin get_table_hash`; they are not written to disk, the WAL, or sys.catalog, and are absent from any consensus/replication path. Mixed-version behavior is safe: an old yb-admin sends no bounds, so a new tserver scans the full table (unchanged); a new yb-admin's bounds are ignored by an old tserver, which scans the full table rather than erroring -- run a version-matched yb-admin when using start_key/end_key. No state is persisted, so rollback has nothing to undo. Original commit: 208a0f88102d23f40365f4ba3d2c5005eddee88e / D53900 Test Plan: AdminCliTest.TestGetTableXorHashKeyRange (non-colocated YCQL hash table): explicit empty bounds reproduce the full-table totals, and a complementary 0x8000 split partitions the rows so counts sum and hashes XOR back to the full totals. PgLibPqTest.TestGetTableXorHashColocatedKeyRange (colocated, range-only table): derives real mid-data split keys for id=4 and id=8 from the server's own partitioning (a throwaway non-colocated SPLIT AT VALUES ((4), (8)) table), then -- passing the child colocated table id -- splits the table into three disjoint segments [-inf, key(4)) / [key(4), key(8)) / [key(8), +inf) (the middle one specifies both bounds), verifying exact per-segment row counts (3/4/3) and that the segments recombine (counts sum, hashes XOR) to the full totals. Also asserts that a key range against the colocation parent table id is rejected. Verified locally (debug/clang21): AdminCliTest.TestGetTableXorHashKeyRange passes. Relying on CSI for the full suite (incl. PgLibPqTest.TestGetTableXorHashColocatedKeyRange). Reviewers: jhe Reviewed By: jhe Differential Revision: https://phorge.dev.yugabyte.com/D54289

2026-06-08

[BACKPORT 2025.1][#31952] yb-admin: Hash a single colocated table in get_table_hash Summary: Backport note: one conflict, in tablet_dump_helper.cc. Dropped the vector-index skip block (`if (table_info->IsVectorIndex()) continue;`) -- that is pre-existing master code, not part of this change, and `IsVectorIndex` does not exist on this branch. Kept the `target_table_found` assignment that this change adds. No other conflicts. get_table_hash hashes an entire tablet. For a colocated tablet (which hosts multiple colocated tables) the command hashes all of them and the requested table_id is effectively ignored -- so a detected inconsistency cannot be narrowed to the specific colocated table that diverged. Passing any colocated table's id returns the whole-tablet result; that is a bug. Scope the hash by the table_id the command is invoked for: - a colocation parent table id hashes every table in the tablet (for a colocated database, all the colocated tables sharing it) -- the previous whole-tablet behavior, now requested explicitly; - any other table id hashes only that single (colocated or non-colocated) table. - DumpTabletDataRequestPB gains table_id; the yb-admin client always forwards it. - tablet_service derives the scope: an unset or colocation-parent table id hashes all tables, otherwise the single requested table. tablet::DumpTabletData skips colocated tables other than the target and errors if the target is not in the tablet. Per-table iterator scoping (via cotable_id) already existed, so this only adds filtering on top of it. Non-colocated tables are unaffected (one table per tablet). This changes behavior for callers that passed a colocated child table id and relied on getting the whole-tablet hash. That behavior was a bug and the tool is new, so we fix it directly (no opt-in flag) and backport to release branches. To request a whole-tablet hash, pass the colocation parent table id, visible in list_tables when system tables are included. For #31952. Prerequisite for the key-range work in #31951. **Upgrade/Rollback safety:** No persistent or on-disk format change, and no AutoFlag is needed. table_id is an optional field on DumpTabletDataRequestPB, an on-demand admin RPC used only by `yb-admin get_table_hash`; it is not written to disk, the WAL, or sys.catalog, and is absent from any consensus/replication path. Mixed-version behavior is safe: an old yb-admin sends no table_id, so a new tserver hashes the whole tablet (unchanged); a new yb-admin's table_id is ignored by an old tserver, which also hashes the whole tablet -- run a version-matched yb-admin to scope to a single colocated table. No state is persisted, so rollback has nothing to undo. Original commit: 048f6f127587cda08991d74ebe0e54f7d30c9aa7 / D53893 Test Plan: AdminCliTestWithYSQL.TestGetTableXorHashColocated: a colocated database with two tables; a child table id hashes just that table (distinct non-zero hashes and correct per-table row counts), and the colocation parent table id hashes the whole tablet (row counts sum and per-table hashes XOR back to the whole-tablet hash). Relying on CSI to build and run the test. --- _automated · Claude Code (Opus 4.8)_ Reviewers: jhe, #db-approvers Reviewed By: jhe, #db-approvers Subscribers: svc_phabricator Differential Revision: https://phorge.dev.yugabyte.com/D54193

[#30578] YSQL: Reset auto-analyze mutation counts after a manual ANALYZE (#31849) Reset auto-analyze mutation counters after user-initiated ANALYZE ## Problem User-initiated ANALYZE does not reset the auto-analyze service's accumulated mutation count. The next periodic tick can therefore trigger an auto-analyze on a table the user just analyzed, wasting work. ## Solution Add ResetAutoAnalyzeMutationCounters RPC (pg_client.proto / pg_client_service) called from do_analyze_rel after a full ANALYZE. The RPC resets the YCQL service table mutations to 0 for the analyzed table. Reset is gated to match PostgreSQL's semantics for updating its changes_since_analyze counters (fires only when no column list given) and is suppressed for auto-analyze service's internal connections and other internal backends. Auto analyze continues to reset mutations using the existing logic to subtract mutations on its side. Mutations are stored in a separate YCQL table so the mutations update is not transactional. A failed manual ANALYZE can still reset counters to 0. Fixing this is tracked in #32081. ## Race handling Extract mutation-update logic into helper functions in pg_auto_analyze_table that use conditional YCQL writes for better reuse. - ResetPgAutoAnalyzeMutationCounts: sets mutations to 0 (IF EXISTS) - SubtractPgAutoAnalyzeMutationCounts: subtracts snapshot mutations with clamping. Emits two conditional writes per table — one sets to 0 if current < snapshot, another subtracts if current >= snapshot. This prevents the count from going negative when a manual-ANALYZE reset races with auto-analyze post-ANALYZE mutation subtraction. UpdateTableMutationsAfterAnalyze refactored to use SubtractPgAutoAnalyzeMutationCounts instead of building operations directly. ## Wire-format changes New RPC: PgClientService.ResetAutoAnalyzeMutationCounters (pg_client.proto). Additive only — no existing field numbers or messages altered. Upgrade/rollback safety: New RPC is between PG and local tserver so no upgrade/rollback issues. ## Test plan - PgAutoAnalyzeTest.ManualAnalyzeResetsMutationCount: covers ANALYZE, ANALYZE(col), VACUUM ANALYZE, VACUUM ANALYZE(col), asserting reset only when no column list provided. - PgAutoAnalyzeTest.InternalAnalyzeDoesNotResetMutationCount: confirms internal connections do not trigger reset. - PgAutoAnalyzeTest.ManualAnalyzePartitionedTableResetsPartitionMutationCounts: exercises the partitioned-table path. - Existing PgAutoAnalyzeTest cases continue to pass.

2026-06-06

[BACKPORT 2024.2][#31952] yb-admin: Hash a single colocated table in get_table_hash Summary: Backport note: three conflicts, all resolved to keep only what this change adds. - tablet_dump_helper.cc: dropped the vector-index skip block (pre-existing master code; `IsVectorIndex` does not exist on this branch); kept the `target_table_found` assignment this change adds. - tablet_service.cc: added only `#include "yb/common/colocated_util.h"` (for IsColocationParentTableId). The adjacent `pg_types.h` include in the master hunk was pre-existing master context, absent here and not needed by this change. - yb-admin-test.cc: the conflict bundled in the unrelated, pre-existing TestPartitionRangeFormat test (not on this branch) -- excluded it. Added the AdminCliTestWithYSQL fixture (pre-existing on master, absent here and required by the new test), the GetColocationParentTableId helper, and TestGetTableXorHashColocated. get_table_hash hashes an entire tablet. For a colocated tablet (which hosts multiple colocated tables) the command hashes all of them and the requested table_id is effectively ignored -- so a detected inconsistency cannot be narrowed to the specific colocated table that diverged. Passing any colocated table's id returns the whole-tablet result; that is a bug. Scope the hash by the table_id the command is invoked for: - a colocation parent table id hashes every table in the tablet (for a colocated database, all the colocated tables sharing it) -- the previous whole-tablet behavior, now requested explicitly; - any other table id hashes only that single (colocated or non-colocated) table. - DumpTabletDataRequestPB gains table_id; the yb-admin client always forwards it. - tablet_service derives the scope: an unset or colocation-parent table id hashes all tables, otherwise the single requested table. tablet::DumpTabletData skips colocated tables other than the target and errors if the target is not in the tablet. Per-table iterator scoping (via cotable_id) already existed, so this only adds filtering on top of it. Non-colocated tables are unaffected (one table per tablet). This changes behavior for callers that passed a colocated child table id and relied on getting the whole-tablet hash. That behavior was a bug and the tool is new, so we fix it directly (no opt-in flag) and backport to release branches. To request a whole-tablet hash, pass the colocation parent table id, visible in list_tables when system tables are included. For #31952. Prerequisite for the key-range work in #31951. **Upgrade/Rollback safety:** No persistent or on-disk format change, and no AutoFlag is needed. table_id is an optional field on DumpTabletDataRequestPB, an on-demand admin RPC used only by `yb-admin get_table_hash`; it is not written to disk, the WAL, or sys.catalog, and is absent from any consensus/replication path. Mixed-version behavior is safe: an old yb-admin sends no table_id, so a new tserver hashes the whole tablet (unchanged); a new yb-admin's table_id is ignored by an old tserver, which also hashes the whole tablet -- run a version-matched yb-admin to scope to a single colocated table. No state is persisted, so rollback has nothing to undo. Original commit: 048f6f127587cda08991d74ebe0e54f7d30c9aa7 / D53893 Test Plan: AdminCliTestWithYSQL.TestGetTableXorHashColocated: a colocated database with two tables; a child table id hashes just that table (distinct non-zero hashes and correct per-table row counts), and the colocation parent table id hashes the whole tablet (row counts sum and per-table hashes XOR back to the whole-tablet hash). Relying on CSI to build and run the test. --- _automated · Claude Code (Opus 4.8)_ Reviewers: jhe Reviewed By: jhe Differential Revision: https://phorge.dev.yugabyte.com/D54194