Proto commits in yugabyte/yugabyte-db

These commits are when the Protocol Buffers files have changed: (only the last 100 relevant commits are shown)

Commit:85defa7
Author:Sergei Politov
Committer:GitHub

[BACKPORT 2025.2][#36288] DocDB: Keep the vectors of rows whose DELETE or UPDATE is rolled back to a savepoint (#38774) ## Summary A DELETE or UPDATE of a row with a vector puts the ids of the vectors it removes into a delete list record of its batch, and the transaction apply tombstones them in the vector reverse mapping. The record had no subtransaction id, so it was applied even when its subtransaction was rolled back, and the index lost the vector of a live row for good. Besides `ROLLBACK TO SAVEPOINT`, READ COMMITTED statement retries, plpgsql `EXCEPTION` blocks and client autosave roll back subtransactions too. The record now starts with a new value type for vector index records, `kVectorIndexRecord`, followed by the record type `VectorIndexRecordType::kDeleteList` and the subtransaction id of its batch, and the apply skips the records of rolled back subtransactions. A later record format takes a new record type rather than a new value type. The readers that only skip these records (transaction loading, lock status and CDC) recognize both the old and the new record through `dockv::IsVectorIndexRecord`. Also moves the subtransaction id encoding of intent values to `EncodeSubTransactionId`, shared by the intents and the external intents writers, which writes the same bytes. ## Upgrade/Rollback safety An older binary does not know the new record: its apply looks the record up as an intent key, does not find it and skips it, so the vectors of the rows deleted by the transaction stay in the index, and its transaction loader logs a decode failure (both `DFATAL`). So the leader writes the new record only after the new AutoFlag `vector_index_delete_list_with_subtransaction` (`kLocalPersisted`) is promoted, and replicates its choice in the write batch (`delete_vector_ids_with_subtransaction`, a new field of `KeyValueWriteBatchPB`), so all replicas and WAL replay write the same record. Promoting the AutoFlag is the point of no return: after it, an older binary leaves the vectors of the rows deleted by transactions that hold the new record. Until the promotion, rolled back subtransactions still lose vectors. `yb-ts-cli verify_vector_indexes` detects the vectors lost before it, and rebuilding the index or rewriting the affected rows restores them. Original commit: 4c1bd14ecceede59962480a173cd36116a8cb6c7 / #37985 ## Test plan - [x] `./yb_build.sh debug --cxx-test pgwrapper_pg_vector_index-test --gtest_filter PgVectorIndexUtilTest.RolledBackSubtransactionKeepsVectors`: without the fix rows 1, 3, 20 and 1000 are missing from the index search results. - [x] `LoadTransactionWithDeleteList`, `DeleteListsWithBothEncodings`, `SameTxnReplacedVector` and `SameTxnDeletedVector` of `PgVectorIndexUtilTest` pass. - [ ] Jenkins.

Commit:37138c6
Author:Suranjan Kumar
Committer:Suranjan Kumar

[#38838] CDC: Store the images of the CDC table as serialized protobuf A before or after image is now a bytea column holding a serialized CdcTableImagePB: for each column its attribute number, its type OID and its QLValuePB, with no value for NULL. The poller serializes the values it already has without converting them, and a reader parses the message once and converts the QLValuePBs the way it converts the values that the CDC service sends. With 8 writers, the poller's share of the tserver's CPU is under 2%. The tests read the images by parsing them. An image is not readable from SQL. Nothing in this design needs it. The tests turn packed rows off: with them, the CDC service reports the update of a row as an insert without a before image, so TablePublication fails. LeaderChange turns off load balancing. The test reads the tablet's leader once and expects the poller there, and the load balancer moved leaders while the cluster settled, which failed it about half the time. ## Upgrade/Rollback safety New messages in cdc_consumer.proto, used only by the CDC table, which exists only behind TEST_cdcsdk_enable_cdc_table. The table that a build creates has the new column types, so a database whose table an earlier build created must create it again. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Commit:686de8e
Author:Suranjan Kumar
Committer:Suranjan Kumar

[#38838] CDC: Stream a HYBRID_TIME replication slot from the CDC table in the walsender The CDC table pollers write the changes of a slot into public.pg_yb_cdc_changes. With yb_use_table_based_cdc on, a walsender reads that table with SPI and hands the existing decoder the same record batches that the CDC service produces, instead of calling the service. The setting is off by default and needs a HYBRID_TIME slot, whose LSN is the commit hybrid time, so a commit time is the position. Rows with one commit_time form one transaction. YBCPgParseCdcTableImage parses an image once, and the decoder converts its QLValuePBs per record as it does for the values of the CDC service. The catalog is not pinned in this mode, so records are decoded against the current catalog. See the header of yb_virtual_wal_table_reader.c for what the reader assumes of the table. The table is hash partitioned on bucket_id and the pollers write it with raw DocDB operations, which would not maintain a secondary index, so the primary key is the only fast access path. A query reads a window of commit times with all 256 buckets listed, so that it seeks to the range in each bucket. The window targets as many rows as sort in half of work_mem, from the size of the first row, between 1000 and 20000, growing while queries return few rows and shrinking while they return many. A query gives back only whole commit groups. The pollers write the table while it is read, so a read can fail with a restart-read error. When the reader began the transaction, it aborts it and reads the window again in a new one, up to 20 times. A walsender that reads the table takes no object locks: YbSetObjectLockingForCurrTxn passes "skip object locks" to pggate for it. They would only hold back concurrent DDLs, and releasing them cost a synchronous RPC at the end of every replay subtransaction. yb_walsender_records_per_flush (default 1, flush after every record) is the number of records WalSndLoop reads before it flushes and checks for replies, for any walsender. While it is above 1, WalSndWriteData holds output back up to 64KB or 5 ms, and the loop reads on only while the next record is already in hand (YBCHasCachedRecord) and flushes before it can sleep. Streaming once all changes of 8 writers over 60 s were in the table, at 64 records per flush, every change streamed once: a table of three columns at 114K changes/s and 7.5 us of walsender CPU per change; a table of 19 columns at 41K changes/s and 20.9 us. The walsender persists the slot's position as the client confirms it. It has no virtual WAL session, so UpdateAndPersistLSN takes table_based, with which the CDC service writes the slot's cdc_state entry directly: restart_lsn as given, one past the commit time of the last transaction the client flushed, and record_id_commit_time just below it. A restarted walsender reads the table from there, and the slot's catalog history retention moves with it. Not handled yet, to come on top of a stable stack: - A table whose relfilenode differs from its OID, as after a rewriting ALTER TABLE: the table id in the CDC table is built from the relfilenode, so the reader does not recognize the table and skips its changes without an error. - Adding a table to or dropping one from the publication while streaming. - Tablet splits. To explore: reading with yb_read_after_commit_visibility=relaxed, which clamps the uncertainty window and so would remove the restart-read errors that the retry handles. Every row below the safe time was written seconds before the reader fetches it, so the relaxed visibility would miss none of them. New GUCs: yb_use_table_based_cdc (off) and yb_walsender_records_per_flush (1). ## Upgrade/Rollback safety UpdateAndPersistLSNRequestPB gets table_based, sent only by a walsender with yb_use_table_based_cdc. A tserver without it treats the call as one for a virtual WAL and fails it, so table mode needs the tservers upgraded first. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Commit:9edf512
Author:Suranjan Kumar
Committer:Suranjan Kumar

[#38838] CDC: Compute a safe time for each CDC table stream The pollers of the tablets of a CDC table stream write the CDC table at different rates, so a reader of the table needs a time up to which every tablet of the stream has all its changes in it. This computes that time by reusing the xCluster safe time instead of new code: - After a CDC table poller applies a GetChanges response, it raises its safe time to the response's safe time, and starts from the one in cdc_state. The consumer publishes these to the xCluster safe time table as it does for xCluster pollers. - The master's XClusterSafeTimeService adds the tablets of the CDC table registry, including the sys catalog when it is polled, and takes the minimum over them. As for xCluster, a tablet that has not published yet makes it invalid and it never goes back. The xCluster background task runs the service while the registry has streams. - The tservers get it in the heartbeat. A new PgClient call, GetXClusterSafeTime, returns it to Postgres through YBCGetXClusterSafeTime. The safe time is keyed by the stream id, not the namespace id. Under a namespace id, xCluster would hold reads in the source database at the default yb_xcluster_consistency_level=database to the safe time, and hold back its compaction. Known limits: yb-admin get_xcluster_safe_time fails with "Keyspace identifier not found" on the stream-id key; a long-running transaction holds back the safe time of the whole stream; the registry still has one stream per database, and does not keep a split tablet's parent until it is drained. The safe time lags commit by the 1 s steps of publishing, computing and the heartbeat. ## Upgrade/Rollback safety GetXClusterSafeTime is a new PgClient RPC between Postgres and its local tserver. The safe times of CDC table streams are written only when CDC table pollers exist, which is only behind TEST_cdcsdk_enable_cdc_table. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Commit:3d2a137
Author:Piyush Jain
Committer:GitHub

[#30797,#37091] YSQL: Set index birth_time for indexes built nonconcurrently (#38639) ## Summary Re-lands #37103, which #38235 reverted, together with the fix for the limitation that got it reverted (#37091). The first commit is the original change, cherry-picked unchanged; the second is the fix. ### Index birth_time for nonconcurrent builds (#30797, first commit) A `CREATE INDEX` run inside a transaction block is built nonconcurrently and skips the backfill that sets the index `birth_time`, so `birth_time` stays 0. The DocDB read guard then does not reject a snapshot read whose read time precedes the index commit, the index rows (written at the commit) are not visible to that read, and it silently returns an incomplete result. This is the data loss seen in the Jepsen si.append-table test. On a successful nonconcurrent index create, set the index `birth_time` to the master clock time (>= the DDL commit time) and propagate it to the tablets with a new `MarkNonConcurrentIndexDone` tserver admin RPC. To reject reads until `birth_time` is final, `IndexInfoPB` gets a `birth_time_state`, set by the master at index creation: - `BIRTH_TIME_FINAL` (default): `birth_time`, if set, is final. - `BIRTH_TIME_PENDING_BACKFILL`: online (backfilled) index, until the backfill completes. - `BIRTH_TIME_PENDING_DDL_COMMIT`: nonconcurrent index created in a DDL transaction, until the transaction commits. The read guard returns two different errors: - While the state is pending on the tablet: `LeaderNotReadyToServe`. The tserver's client retries it until the RPC deadline, so the read waits until the tablet is finalized. - Once `birth_time` is set, for a read whose read time is before it: `SnapshotTooOld` (SQLSTATE 72000). The query layer does not retry it, so the application has to retry the transaction. Details: - Online indexes: the state is cleared together with setting `birth_time` when the backfill completes. The backfill path used `retain_delete_markers` as a proxy for this; both are checked now. - Nonconcurrent indexes: the DDL transaction verifier (`HandleSuccessfulYsqlDdlTxn`) sets `birth_time`, clears the state and posts `AsyncMarkNonConcurrentIndexDone` to each tablet, which replicates a ChangeMetadata op with the new `mark_non_concurrent_index_done` field. Each finalize path only clears its own pending state. A finalize for an index already removed from the tablet (e.g. a `DROP INDEX` on a colocated table racing with the RPC) is skipped instead of failing the replicated op. - `birth_time` >= commit time: the transaction coordinator picks the commit time, so the master's clock is moved past it first. `ReportYsqlDdlTxnStatus` now carries the reporting tserver's hybrid time, and when the master polls the transaction instead, its clock is updated from the `GetTransactionStatus` response. - Recovery of a lost finalize RPC: the master does not resend it. The tablet validator also checks indexes whose local state is not final and finalizes the tablet metadata once the master reports the index final. It asks only once per tablet open and caches a "pending" answer for an hour; the same gap already exists for BackfillDone, and both are tracked in #37622. - xCluster targets: the birth time is not propagated yet (#33155). The target re-creates the index through DDL replication, so it sets the pending state and clears it on commit, without stamping a local `birth_time`. ### Let a transaction read an index it created (#37091, second commit) With the above alone, a statement that reads a nonconcurrently built index in the transaction that created it (index scan, foreign key check against a new UNIQUE constraint, `INSERT ... ON CONFLICT` on a new unique index, `yb_index_check()`, an extension script) is retried until the RPC deadline: the index only becomes final once that same transaction commits. This broke `DocumentDBTest.*`, `BasicDocumentDBTests#testBasicOperations` and `PgPartmanTest.*/FlagOn`. The creating transaction wrote the index, so the index is complete for it. Its reads are now served: - The master stores the creating transaction's id in the new `IndexInfoPB.birth_time_pending_txn_id` together with `BIRTH_TIME_PENDING_DDL_COMMIT`, and clears both at finalize; the tablet does the same. - The read guard does not reject a read of a pending index whose transaction id matches. Every other reader is treated as before. - pggate no longer sends the request to an index with `skip_intents`, so that it carries the transaction's metadata. With the new-relation fastpath, reads of a table created in the transaction, and until now of its indexes, are sent without it. The index read still happens at in_txn_limit, so it sees the same rows; the cost is an intents lookup that finds nothing. The three regress reads that the first commit comments out (`yb.orig.ddl_txn_{triggers,event_triggers,indices}`) are restored. Fixes #30797 Fixes #37091 ## Upgrade/Rollback safety - New AutoFlag `enable_ysql_index_birth_time_state` (kLocalPersisted). The master sets a non-final `birth_time_state` only when it is on; while it is off, behavior is unchanged (every index is final). It is an AutoFlag so that no index is left pending forever: an old master that becomes leader before the CREATE INDEX transaction is verified would not finalize the index, and after a rollback, indexes that were still pending would stay pending with nothing to finalize them on re-upgrade. - The new fields (`IndexInfoPB.birth_time_state`, `IndexInfoPB.birth_time_pending_txn_id`, `IndexStatusPB.birth_time_state`, `ChangeMetadataRequestPB.mark_non_concurrent_index_done`) and the new RPC are only used for indexes with a non-final state, which exist only once the AutoFlag is promoted, i.e. after all tservers run the new version. Older tservers never receive them. - A missing `IndexInfoPB.birth_time_state` means final, so indexes created before the upgrade keep the current behavior. An older binary that reads persisted metadata containing these fields ignores them, and only loses the new read guard. A `GetBackfillStatus` response without `IndexStatusPB.birth_time_state` (from an older master) is treated as not final, so the tablet validator does not finalize anything from it. - New optional `ReportYsqlDdlTxnStatusRequestPB.propagated_hybrid_time`: an older master ignores it and an older tserver does not send it, which is the behavior before this change in both cases. ## Test plan Release build. - [x] `./yb_build.sh release --cxx-test pg_concurrent_ddls-test`. New: `PgReadIndexInCreatingTxnTest.{NewIndexOnExistingTable,NewTableWithUniqueConstraint,Rewrites}`, each for READ COMMITTED, REPEATABLE READ and SERIALIZABLE, with the new-relation fastpath in transaction blocks on and off, with and without a forced parallel plan, and in a normal and a colocated database (fastpath at its default there). `PgConcurrentDDLsTest.PendingIndexRejectsStaleRead` now also reads through the index from the creating transaction before it commits. The foreign key part of `NewTableWithUniqueConstraint` is skipped for REPEATABLE READ with the fastpath on, where the unrelated #38492 fails it. - [x] `./yb_build.sh release --java-test 'org.yb.pgsql.TestDdlTransactionBlocks'` (runs the three restored regress reads) - [x] `./yb_build.sh release --cxx-test pg_vector_index-test --gtest_filter '*SearchInIndexCreatingTransaction*:*ReadBeforeIndexCreation*'` (after merging master, which made the read guard cover vector indexes) - [x] `./yb_build.sh release --cxx-test index_birth_time_upgrade-test` - [x] `./yb_build.sh release --cxx-test pg_partman-test --gtest_filter 'PgPartmanTest.*'` (FlagOn and FlagOff; before the merge of master, not repeated since) - [x] `./yb_build.sh release --cxx-test pg_skip_intents_metrics-test` - [x] On an earlier base of this branch: without the read guard exemption, the `PgReadIndexInCreatingTxnTest.NewIndexOnExistingTable` cases time out on the first read through the new index. - [ ] `DocumentDBTest.*`, `BasicDocumentDBTests#testBasicOperations`: not run locally (the DocumentDB extension is not built in the local environment); left to CI. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>

The documentation is generated from this commit.

Commit:eb964e3
Author:Hari Krishna Sunder

Rust master layer 4: the master leader follows the preferred zones With sys_catalog_respect_affinity_task (default true), the Rust leader moves sys-catalog leadership to a master in the cluster's preferred zones, as the C++ master did (SysCatalogRespectLeaderAffinity, GetMoreEligibleSysCatalogLeaders): priority-ordered affinitized leaders, wildcards and the leader blacklist; a down preferred master is skipped. It runs every catalog_manager_bg_task_wait_ms (cluster-balancer master_affinity.rs) and steps down through a new host method, SysCatalogLeaderStepDown, which also honors TEST_crash_server_on_sys_catalog_leader_affinity_move. Tests: sys_catalog_respect_affinity-test 11/11 (6 failed before), rust_master-itest 13/13, master_failover-itest 15/15, rust_checks, 15 new Rust unit tests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015zgRgy1Vf959gw53qK3oNw

Commit:60706d2
Author:Hari Krishna Sunder

Rust master layer 4: close the gaps left after the C++ master's deletion - A YSQL database whose creation failed frees its name, as in C++, so CREATE DATABASE can be retried; its id stays taken, and an id collision reports YB_PG_DUPLICATE_DATABASE. - backfill_aborted is published when a backfill fails. - FlushTables fails with InternalError when a tserver's flush RPC fails, instead of waiting for the client's deadline. - A clone to a time during a YSQL major catalog upgrade is refused again: clone.rs looked the catalog config row up under the wrong id. - Old master flags that nothing read now take effect with their C++ meaning: qos_max_db_count, enable_tablet_split_of_pitr_tables, enable_transactional_ddl_gc, master_failover_catchup_timeout_ms, cdc_parent_tablet_deletion_task_retry_secs, max_concurrent_alter_table_rpcs(_per_tserver), retrying_rpc_max_jitter_ms, the index backfill retry, alter-wait and split-wait flags, ysql_backends_catalog_version_job_expiration_sec, catalog_manager_inject_latency_in_delete_table_ms and master_slow_get_registration_probability. Flags with no Rust mechanism are listed in cpp-host.md ("Flags without effect"). - Rust unit tests replace the deleted C++ ones for xCluster safe time, the clone state machine and affinitized-leader balancing. Tests: rust_checks; master-test (72, MasterTest.TestMultipleNamespacesWithSameName restored), flush_manager-test, rust_master_mini_cluster-test, rust_master-itest, PgLibPqTest.Simple, pg_index_backfill-test (BackendKill, WaitForSplitsToComplete), cdcsdk_tablet_split-test TestTabletSplitBeforeBootstrapGetCheckpoint, PgOidCollisionCreateDatabaseTest. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015zgRgy1Vf959gw53qK3oNw

Commit:1a99ac5
Author:Hari Krishna Sunder

Rust master layer 4: close the master gaps the test ports found Once yb-master is the Rust binary there is no C++ master to fall back to, so master behavior that tests exercise and Rust lacked is ported here. Most of it was found by running the ported tests on Rust masters. New-cluster setup and the shipped snapshot: - master_auto_run_initdb: a Rust leader runs initdb for a new cluster (MajorUpgrade). - create_initial_sys_catalog_snapshot: the host writes the initial sys-catalog snapshot after initdb; Rust and C++ masters both restore it. - A leader that does not run initdb makes the YSQL catalog tables transactional, as C++ MakeYsqlSysCatalogTablesTransactional does (host method MakeSysCatalogTablesTransactional). - Call-home runs on Rust masters; its collectors read through the master's own RPCs. Fixes to behavior users see: - A restarted master fetches the universe (encryption) keys from the other masters, as C++ does; it never had them before. - A YSQL ALTER records its DDL transaction with the write, so AlterSchema always carries it and concurrent transactions are aborted. - Placement-local transaction status tables are listed to tservers from creation, so geo-partitioned transactions use them. - Tablet replica limits are enforced on create, split and clone. - Transaction status tables can be altered (placement), bumping the transaction tables version. - Bidirectional xCluster index replication (C++ AddBiDirectionalIndexToXClusterTargetTask). - The target-side xCluster WAL anchor deletion. - A leaderless tablet report starts a hinted election, as in C++: split children (whose first reports carry the parent's leader) and a restarted cluster's tablets get leaders without waiting for Raft's election timeout. - catalog_manager_wait_for_new_tablets_to_elect_leader is honored. - emergency_repair_mode: no catalog load or leader duty; DumpSysCatalogEntries and WriteSysCatalogEntry are served. - Shutdown stops the Rust runtimes before the host, so pending calls are answered and the messenger does not wait forever. - RPC metrics for Rust services, load_balancer_duration, split gauges, tserver lease info in ListTabletServers, deleted-database tombstone cleanup at term start. Test hooks: 34 C++ master TEST_ flags and about 10 sync points that tests set are now read by the Rust master. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015zgRgy1Vf959gw53qK3oNw

Commit:3300795
Author:Hideaki Kimura
Committer:Hideaki Kimura

[#32163] DocDB: Retain and replay APPLYING ops in the WAL until intents removal is flushed Summary: A committed transaction's UPDATE_TRANSACTION_OP(APPLYING) can be re-delivered, e.g. after a status tablet leader change resets the coordinator's in-memory applied acks. On a live node the duplicate is a no-op, because the participant's in-memory local_commit_time marks the transaction as already applied. That state is lost on an ungraceful restart, while duplicates can keep arriving until the transaction's intents are durably removed. If a duplicate is above the regular flushed frontier at the moment of the crash, bootstrap replays it and re-applies the transaction. If a packed-row compaction has meanwhile packed a newer value below the transaction's commit hybrid time, the re-applied value shadows the newer one, causing silent data loss. Fix: guarantee that whenever a duplicate APPLYING can replay at bootstrap, the transaction's original APPLYING replays before it. Replaying the original re-establishes local_commit_time, so the duplicate becomes a recognized no-op. A new consensus frontier field, txn_apply_op_index, holds a transaction APPLYING Raft index watermark in each RocksDB. In the regular RocksDB it is the highest APPLYING index applied there, recorded by the apply write. In the intents RocksDB it is the highest APPLYING index up to which every applied transaction has its intents removal written, recorded by the removal write. The transaction participant tracks applied transactions until their intents removal is written, to compute this value. While the flushed regular value is above the flushed intents value, WAL GC keeps and tablet bootstrap replays APPLYING operations from the intents value on. Otherwise nothing extra is retained. On a tablet with no flushed intents removal yet (a new tablet, or the first start after upgrade), this means from the start of the log, until the first removal is flushed. Bootstrap still chooses its first segment by the flushed OpIds. While there is a value to replay from, the first operation of that segment must also be at or below it by Raft index, and the segment is replayed from its beginning. Intents can also leave the disk without a flushed removal write: the intents SST file cleanup deletes the SST and a crash loses the unflushed removal, or the removal finds nothing left to delete. So when the regular value is ahead and the intents RocksDB has nothing to flush, WAL GC records the participant's current value in the intents flushed frontier directly, and the retention ends without waiting for the next removal on the tablet. The rule reads the flushed frontiers in memory, so it depends on #38765, which makes them match the MANIFEST after a group commit. The transaction intents removal write (Tablet::RemoveIntents) no longer applies the min-running-hybrid-time SST file filter, since a removal that skips the files holding the transaction's reverse index would end the tracking with those intents still on disk; the apply, read, and advisory-lock paths keep the filter. Applies are not recorded in the regular RocksDB while a CDC checkpoint is active on the tablet, and transactions whose intents cleanup is handed over to CDC intent SST file cleanup are not tracked. Both keep the current behavior. On truncate/restore the tracking is cleared, since the intents are deleted together with the DB. **Upgrade/Rollback safety:** The only persistent format change is the new optional ConsensusFrontierPB field. An absent field falls back to the current behavior and old binaries ignore the unknown field, so the change is upgrade and rollback safe without an AutoFlag. The fix is guarded by the runtime flag enable_wal_retention_for_unremoved_intents (default true). Test Plan: New regression tests for the duplicate-APPLYING re-apply, the WAL retention and its release, plus a unit test for the new frontier field and the retention rule: ./yb_build.sh fastdebug --cxx-test pgwrapper_pg_mini-test --gtest_filter PgMiniTestSingleNode.ReApplyAfterRepackPreventedOnBootstrap ./yb_build.sh fastdebug --cxx-test pgwrapper_pg_mini-test --gtest_filter PgMiniTestSingleNode.ReApplyAfterRepackPreventedWithAdvancedIntentsFlushedOpId ./yb_build.sh fastdebug --cxx-test pgwrapper_pg_mini-test --gtest_filter PgMiniTestSingleNode.ReApplyAfterRepackWithoutWalRetention ./yb_build.sh fastdebug --cxx-test pgwrapper_pg_mini-test --gtest_filter PgMiniTestSingleNode.ReApplyAfterRepackPreventedWithUnflushedIntentsRemoval ./yb_build.sh fastdebug --cxx-test pgwrapper_pg_mini-test --gtest_filter PgMiniTestSingleNode.WalGcReleasedAfterLastDistributedTxn ./yb_build.sh fastdebug --cxx-test pgwrapper_pg_mini-test --gtest_filter PgMiniTestSingleNode.WalGcReleasedAfterLastDistributedTxnWithoutFlushedOpIdAdvance ./yb_build.sh fastdebug --cxx-test pgwrapper_pg_mini-test --gtest_filter PgMiniTestSingleNode.WalGcReleasedAfterLastDistributedTxnAndCrash ./yb_build.sh fastdebug --cxx-test pgwrapper_pg_mini-test --gtest_filter PgMiniTestSingleNode.WalGcReleasedAfterRolledBackLastDistributedTxn ./yb_build.sh fastdebug --cxx-test docdb_consensus_frontier-test --gtest_filter ConsensusFrontierTest.TxnApplyOpIndex Existing bootstrap tests: ./yb_build.sh fastdebug --cxx-test tablet_tablet_bootstrap-test

Commit:87a0b7d
Author:myang2021
Committer:GitHub

[BACKPORT 2026.1][#34309] YSQL: Send a connection away from a fill of its own session (#37789) (#38655) ## Summary Problem: Two TODO(#34309) markers left by 510da0f5d37, independent of each other. A connection start's prefetch asks the tserver not to queue it behind a fill already under way, by setting send_away_if_fill_in_flight on the request. AnotherFillInFlight, the check behind that ask, counted only fills whose session id differs from the asker's, since the session id is the only handle on "another connection" a Perform carries. Parallel workers adopt their leader's session id and each run a connection start of their own, so the workers of one query -- siblings -- all prefetch under a single session id. A sibling's fill therefore carries the asking worker's own session id, and AnotherFillInFlight skipped it as the asker's own rather than counting it. A prefetch is several paged rounds, so the second worker waits out the rounds the first still has in flight instead of reading the version this node already holds -- the wait 510da0f5d37 exists to remove. Fix: Count every fill in the key group, the asking session's own included. AnotherFillInFlight loses its session id argument and becomes FillInFlight, a plain lookup of the key group: [[nodiscard]] bool AnotherFillInFlight(KeyGroup group, uint64_t session_id) REQUIRES(mutex_) { auto it = fills_in_flight_.find(group); if (it == fills_in_flight_.end()) { return false; } return std::ranges::any_of( it->second, [session_id](const auto& e) { return e.first != session_id; }); } becomes [[nodiscard]] bool FillInFlight(KeyGroup group) REQUIRES(mutex_) { return fills_in_flight_.contains(group); } Skipping the asker's own session was meant to stop a connection being sent away from a fill of its own. Outside a parallel query it almost never had anything to skip. send_away_if_fill_in_flight is set only when the master leader is not shedding, and setting it and starting at the stale read are exclusive branches, so a connection that asks begins at the trust_cache attempt. Its opening round is the one request that carries the ask, and it reaches this check before the connection has filled anything. A send-away clears send_away_if_fill_in_flight, and the connection then either reads the older version it was sent to or, where that could not be prepared, runs trust_cache a second time -- an attempt that no longer reaches this check. A connection to template1 is the exception. The auth prefetch fills under Template1DbOid with an ordinary token, and that token lives in its setter until the request object goes away, which is past the response that lets the backend send its opening round. The skip was hiding a fill of that connection's own, and counting it costs such a connection a stale attempt or a rerun, not a wrong read. Siblings are the case the skip decided wrongly, and the one this change is for. With the session filter gone nothing reads the session level of fills_in_flight_, so it becomes a count per key group, and session_id goes from FillToken, NoteFillStarted, NoteFillFinished, TakeFillToken and Get. What ties a caller to another's claim is sharing its token slot, which is what parallel workers do: prefetch_fill_token_ is a PgClientSession member and a worker adopts its leader's. Descriptions of the check that still gave the old rule now match it: the FillInFlight comment, the pg_response_cache_send_aways help text an operator reads, the YBCStartSysTablePrefetching contract and the send-away branch in YbRunWithPrefetcher. AConnectionIsNotSentAwayFromAFillOfItsOwn pinned that skip, asserting that a second round of one session is not sent away while the first is still filling. It becomes ASiblingIsSentAwayFromAFillOfTheSameSession, which asserts the opposite and reads those two requests as two siblings rather than one connection asking twice -- pggate asks on the opening round only. Their keys differ, so only FillInFlight can turn the second away, and what makes them siblings is now the token slot they share. AGroupStaysBusyAcrossOverlappingRounds reached the skip the same way, with SendAway::kTrue on its later round; that round now carries SendAway::kFalse, which is what pggate sends. FillInFlight counts the whole key group, so it already covers most callers that ask for the very entry being filled. The group can still be uncounted while that entry fills: the entry is marked as filling a moment before the setter takes the token that counts the group, and siblings share one token, so one sibling's completing round can release it while another's round is still filling -- a round's response is stored after the session lock is dropped, so the two overlap. A caller landing in either gap is turned away on the entry itself, which is what ACallerOnTheEntryBeingFilledIsSentAway covers. This change leaves that send-away alone, but the test no longer reached it: the caller asked on an entry its own session was filling, and FillInFlight now catches that first. Its setup is rebuilt on the token-release gap. The other TODO is unrelated to the check above. ComesUpWhenSentAwayWithNoUsableOlderVersion could not tell a connection that never failed from one whose retry succeeded. It now connects through PGConnSettings::should_stop, which ends the retry loop after the first attempt. ## Upgrade/Rollback safety No wire-format change. The only `.proto` edit is a comment on the existing `CachingInfoPB.send_away_if_fill_in_flight` (field 7); no field is added, removed, renumbered, or reinterpreted. That flag travels only between a node's own PG backends and its tserver, which come from the same binary, and the response cache it drives is local to that tserver. A mixed-version cluster therefore sees no interaction here: each node answers the flag with whatever check its own build carries. Rollback is a plain revert. The send-away decision is taken per request and leaves no persistent state, so there is nothing to migrate in either direction. Original commit: ad6f5e16387572dc8fd558b9f96990950c46082d / #37789 ## Test plan ``` ./yb_build.sh release --skip-pg-parquet --sj --cxx-test pg_response_cache-test ./yb_build.sh release --skip-pg-parquet --sj --cxx-test pg_catalog_perf-test \ --gtest_filter 'PgStaleCatalogPrefetch*:PgUnbridgeable*:PgNarrowMessageQueue*' build-support/lint.sh --rev origin/master ``` ## Merge conflicts ### 2026.2 --> 2026.1 No conflicts. ### master --> 2026.2 No conflicts. Verified on 2026.1 (release build): `pg_response_cache-test` 18/18 passed; `pg_catalog_perf-test --gtest_filter 'PgStaleCatalogPrefetch*:PgUnbridgeable*:PgNarrowMessageQueue*'` 16/16 passed.

Commit:520a54c
Author:myang2021
Committer:GitHub

[BACKPORT 2026.2][#34309] YSQL: Send a connection away from a fill of its own session (#37789) (#38654) ## Summary Problem: Two TODO(#34309) markers left by 510da0f5d37, independent of each other. A connection start's prefetch asks the tserver not to queue it behind a fill already under way, by setting send_away_if_fill_in_flight on the request. AnotherFillInFlight, the check behind that ask, counted only fills whose session id differs from the asker's, since the session id is the only handle on "another connection" a Perform carries. Parallel workers adopt their leader's session id and each run a connection start of their own, so the workers of one query -- siblings -- all prefetch under a single session id. A sibling's fill therefore carries the asking worker's own session id, and AnotherFillInFlight skipped it as the asker's own rather than counting it. A prefetch is several paged rounds, so the second worker waits out the rounds the first still has in flight instead of reading the version this node already holds -- the wait 510da0f5d37 exists to remove. Fix: Count every fill in the key group, the asking session's own included. AnotherFillInFlight loses its session id argument and becomes FillInFlight, a plain lookup of the key group: [[nodiscard]] bool AnotherFillInFlight(KeyGroup group, uint64_t session_id) REQUIRES(mutex_) { auto it = fills_in_flight_.find(group); if (it == fills_in_flight_.end()) { return false; } return std::ranges::any_of( it->second, [session_id](const auto& e) { return e.first != session_id; }); } becomes [[nodiscard]] bool FillInFlight(KeyGroup group) REQUIRES(mutex_) { return fills_in_flight_.contains(group); } Skipping the asker's own session was meant to stop a connection being sent away from a fill of its own. Outside a parallel query it almost never had anything to skip. send_away_if_fill_in_flight is set only when the master leader is not shedding, and setting it and starting at the stale read are exclusive branches, so a connection that asks begins at the trust_cache attempt. Its opening round is the one request that carries the ask, and it reaches this check before the connection has filled anything. A send-away clears send_away_if_fill_in_flight, and the connection then either reads the older version it was sent to or, where that could not be prepared, runs trust_cache a second time -- an attempt that no longer reaches this check. A connection to template1 is the exception. The auth prefetch fills under Template1DbOid with an ordinary token, and that token lives in its setter until the request object goes away, which is past the response that lets the backend send its opening round. The skip was hiding a fill of that connection's own, and counting it costs such a connection a stale attempt or a rerun, not a wrong read. Siblings are the case the skip decided wrongly, and the one this change is for. With the session filter gone nothing reads the session level of fills_in_flight_, so it becomes a count per key group, and session_id goes from FillToken, NoteFillStarted, NoteFillFinished, TakeFillToken and Get. What ties a caller to another's claim is sharing its token slot, which is what parallel workers do: prefetch_fill_token_ is a PgClientSession member and a worker adopts its leader's. Descriptions of the check that still gave the old rule now match it: the FillInFlight comment, the pg_response_cache_send_aways help text an operator reads, the YBCStartSysTablePrefetching contract and the send-away branch in YbRunWithPrefetcher. AConnectionIsNotSentAwayFromAFillOfItsOwn pinned that skip, asserting that a second round of one session is not sent away while the first is still filling. It becomes ASiblingIsSentAwayFromAFillOfTheSameSession, which asserts the opposite and reads those two requests as two siblings rather than one connection asking twice -- pggate asks on the opening round only. Their keys differ, so only FillInFlight can turn the second away, and what makes them siblings is now the token slot they share. AGroupStaysBusyAcrossOverlappingRounds reached the skip the same way, with SendAway::kTrue on its later round; that round now carries SendAway::kFalse, which is what pggate sends. FillInFlight counts the whole key group, so it already covers most callers that ask for the very entry being filled. The group can still be uncounted while that entry fills: the entry is marked as filling a moment before the setter takes the token that counts the group, and siblings share one token, so one sibling's completing round can release it while another's round is still filling -- a round's response is stored after the session lock is dropped, so the two overlap. A caller landing in either gap is turned away on the entry itself, which is what ACallerOnTheEntryBeingFilledIsSentAway covers. This change leaves that send-away alone, but the test no longer reached it: the caller asked on an entry its own session was filling, and FillInFlight now catches that first. Its setup is rebuilt on the token-release gap. The other TODO is unrelated to the check above. ComesUpWhenSentAwayWithNoUsableOlderVersion could not tell a connection that never failed from one whose retry succeeded. It now connects through PGConnSettings::should_stop, which ends the retry loop after the first attempt. ## Upgrade/Rollback safety No wire-format change. The only `.proto` edit is a comment on the existing `CachingInfoPB.send_away_if_fill_in_flight` (field 7); no field is added, removed, renumbered, or reinterpreted. That flag travels only between a node's own PG backends and its tserver, which come from the same binary, and the response cache it drives is local to that tserver. A mixed-version cluster therefore sees no interaction here: each node answers the flag with whatever check its own build carries. Rollback is a plain revert. The send-away decision is taken per request and leaves no persistent state, so there is nothing to migrate in either direction. Original commit: ad6f5e16387572dc8fd558b9f96990950c46082d / #37789 ## Test plan ``` ./yb_build.sh release --skip-pg-parquet --sj --cxx-test pg_response_cache-test ./yb_build.sh release --skip-pg-parquet --sj --cxx-test pg_catalog_perf-test \ --gtest_filter 'PgStaleCatalogPrefetch*:PgUnbridgeable*:PgNarrowMessageQueue*' build-support/lint.sh --rev origin/master ```

Commit:3e04d8c
Author:Samson Shaji
Committer:GitHub

[DB-21916] DocDB: Report each replica's storage tier and migration state in the tablet report (#36848) ## Summary Builds on #35107 (DB-21917), landed on master as 5afa7d2cb91. The master's placement logic needs to know, per replica, which tier its data is on, which tier it was told to move to, and whether that migration is still running, finished, or gave up. Nothing carried that upward: `AlterTabletTier` only starts the migration, and its outcome was visible only through `GetTabletTierInfo`. - `ReportedTabletPB` gains `current_storage_tier`, `target_storage_tier` and `tier_migration_state`, filled only when the tserver runs with `--enable_tiered_storage`. - A tablet re-reports itself when its target or its migration state changes, via a new `tier_state_changed` hook that marks it dirty. One migration costs two reports: the target recorded, then the reconciler's `DONE` or `FAILED`. Nothing is sent on a timer. - The master keeps the three values on `TabletReplica` and shows them per replica on the table page. A report that rebuilds every replica's entry applies the tier fields to the sender only; the other replicas keep what they last reported for themselves. - An empty tablet reports the tier of the disk its next flush lands on, so `GetTierInfo` agrees with the reconciler's verdict for it. - `TierMigrationStatusPB` moves from `tserver_admin.proto` to `tserver_types.proto` unchanged, so `master_heartbeat.proto` can use it without depending on the admin service. ## Upgrade/Rollback safety `ReportedTabletPB` gains three optional fields; `TierMigrationStatusPB` keeps its name, package and field numbers. - Old master, new tserver: the fields arrive as unknown fields and are ignored. - New master, old tserver: the fields are absent, so `TabletReplica` keeps empty tiers and `NONE`, the same as a tserver with the feature off. - Rollback: the master stops reading the fields; the tserver stops sending them. Nothing is persisted on either side. - `StateChangeReason::TIER_MIGRATION_STATE_CHANGED` is a new in-memory enumerator used only to label the dirty mark in logs. ## Test plan - [x] `ts_tablet_manager-test`: `TierMigrationReconcilerTest.ReportCarriesTierState` (born on ssd with no target; `AlterTabletTier` re-reports target, tier and `IN_PROGRESS`; a repeated `AlterTabletTier` re-reports nothing; the reconciler's verdict re-reports `DONE`, a repeated verdict nothing), `TierMigrationReconcilerDisabledTest.ReportCarriesNoTierFields`, plus the existing `TierMigrationReconcilerTest.*` and `TsTabletManagerTieredDriveTest.*` - [x] `master_heartbeat-itest`: `TierStateHeartbeatTest.TierStateReachesMaster` (every replica reports ssd / no target / `NONE`; after `AlterTabletTier` on one replica the master sees its target, tier and `DONE` while the others keep no target; a new-term report from that replica rebuilds the replica map and the others still keep no target) - [x] `tablet-tier-migration-test`: all, for the `GetTierInfo` change - [x] `master_path_handlers-itest`: `TabletServersListStorageTiers` --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>

Commit:1f403af
Author:Sergei Politov
Committer:GitHub

[BACKPORT 2026.1][#36288] DocDB: Keep the vectors of rows whose DELETE or UPDATE is rolled back to a savepoint (#38699) ## Summary A DELETE or UPDATE of a row with a vector puts the ids of the vectors it removes into a delete list record of its batch, and the transaction apply tombstones them in the vector reverse mapping. The record had no subtransaction id, so it was applied even when its subtransaction was rolled back, and the index lost the vector of a live row for good. Besides `ROLLBACK TO SAVEPOINT`, READ COMMITTED statement retries, plpgsql `EXCEPTION` blocks and client autosave roll back subtransactions too. The record now starts with a new value type for vector index records, `kVectorIndexRecord`, followed by the record type `VectorIndexRecordType::kDeleteList` and the subtransaction id of its batch, and the apply skips the records of rolled back subtransactions. A later record format takes a new record type rather than a new value type. The readers that only skip these records (transaction loading, lock status and CDC) recognize both the old and the new record through `dockv::IsVectorIndexRecord`. Also moves the subtransaction id encoding of intent values to `EncodeSubTransactionId`, shared by the intents and the external intents writers, which writes the same bytes. ## Upgrade/Rollback safety An older binary does not know the new record: its apply looks the record up as an intent key, does not find it and skips it, so the vectors of the rows deleted by the transaction stay in the index, and its transaction loader logs a decode failure (both `DFATAL`). So the leader writes the new record only after the new AutoFlag `vector_index_delete_list_with_subtransaction` (`kLocalPersisted`) is promoted, and replicates its choice in the write batch (`delete_vector_ids_with_subtransaction`, a new field of `KeyValueWriteBatchPB`), so all replicas and WAL replay write the same record. Promoting the AutoFlag is the point of no return: after it, an older binary leaves the vectors of the rows deleted by transactions that hold the new record. Until the promotion, rolled back subtransactions still lose vectors. `yb-ts-cli verify_vector_indexes` detects the vectors lost before it, and rebuilding the index or rewriting the affected rows restores them. Original commit: 4c1bd14ecceede59962480a173cd36116a8cb6c7 / #37985 ## Test plan - [x] `./yb_build.sh debug --cxx-test pgwrapper_pg_vector_index-test --gtest_filter PgVectorIndexUtilTest.RolledBackSubtransactionKeepsVectors`: without the fix rows 1, 3, 20 and 1000 are missing from the index search results. - [x] `LoadTransactionWithDeleteList`, `DeleteListsWithBothEncodings`, `SameTxnReplacedVector` and `SameTxnDeletedVector` of `PgVectorIndexUtilTest` pass. - [ ] Jenkins.

Commit:7c65577
Author:Piyush Jain
Committer:GitHub

[#35007,#36262,#36877,#31142,#29272] YSQL: Fix txn retries dropping their re-acquired table locks (#36020) ## Summary Fixes #35007, Fixes #36262, Fixes #36877, Fixes #31142, Fixes #29272 A REPEATABLE READ or SERIALIZABLE transaction (or an autocommit statement) whose first statement fails with a read restart, kConflict, kDeadlock or kAborted is transparently retried from scratch. pggate restarted or recreated the transaction only locally, and postgres re-acquired the statement's table locks (`YBAcquireExecutorLocksForRetry`) through the object lock fastpath, which records them in the session's shared memory without an RPC. The tserver applied the restart or replacement with the next RPC by releasing all of the session's fastpath locks (`PgClientSession::RestartTransaction` / `DoBeginTransactionIfNecessary` -> `TServerUnlockAll`), including the ones just re-acquired. The transaction kept running with no table locks while postgres believed it held them. Consequences: - A conflicting DDL is not blocked (#35007: rows violating a concurrently added CHECK constraint were committed). - A later stronger table lock from the lockless txn queues behind a waiting DDL, forming a deadlock that clears only after ~30s (#36262). Fix: `PgTxnManager::RestartTransaction` and `RecreateTransaction`, called only by query layer retries, now send a new `RestartTransactionForRetry` RPC right away. For a conflict retry its tserver handler runs `SetupSession`, which replaces the txn on the bumped txn serial no; for a read restart retry (`REPLACE_DIST_TRANSACTION_ONLY`) it aborts the txn, and the retry's next request starts a new one. Either way the old txn's object locks are released before the retry re-acquires any. A read restart retry no longer creates a restarted transaction: the retry takes a new snapshot anyway, so the restart read time was never used. The new transaction keeps the replaced one's priority and starts from a fresh read point. `need_restart_`, `ReadTimeOptionsPB.restart_transaction` and the `RESTART` read time manipulation are gone. Read restart retries: they used to converge only because the restart rode on the retry's first RPC (`UpdateReadTime`'s restart branch preempts the read point reset on a new read time serial no). That left later snapshots of a retried statement, and catalog reads that consumed the restart, with fresh uncertainty windows (#31142, #29272). Now the `RestartTransactionForRetry` response returns the failed read point's global limit and per-tablet local limits; for the rest of the retried statement pggate sends the lowest of each as `global_limit_cap_ht` and `local_limit_caps`, and the tserver caps every read point it picks at them (`ConsistentReadPoint::CapGlobalLimit` / `CapLocalLimit`), picking the read time itself rather than leaving it to DocDB. Clamped reads (catalog snapshots) and read points reset for the storage layer to pick (non-transactional writes) are not capped. The caps end with the plain transaction, so a transaction a procedure starts after a COMMIT gets full uncertainty windows. Without the local limits, a tablet with a steady stream of writes would keep restarting the retry until the clock passed the global limit, up to the max clock skew. A retry reads at the current time, which hybrid time propagation from the restarting read already puts past the conflicting value, and its window can only shrink, so the retry converges. The caps apply to every snapshot of the retried query until its transaction ends, so in READ COMMITTED a later statement of a retried CALL, DO block or function that starts after the cap has passed reads with no uncertainty window. That is acceptable: once a query is issued, nothing reaches it except through the database. If another transaction commits a write and then unblocks the query (a lock or advisory lock release, a wait queue wakeup), the release carries its hybrid time, so the query's later reads are past that write and see it without a window. If it unblocks the query and then commits, or commits without the query depending on it, the write is concurrent with the query, which can't have been told about it; read-after-commit visibility applies to queries issued after the commit, and those start uncapped. The exception is a side channel inside the query that bypasses the hybrid clock (e.g. dblink or an FDW to another system), and there the exposure is bounded by the max clock skew. READ COMMITTED statement retries use the same path (`FETCH_GLOBAL_LOCAL_LIMITS_ONLY`) instead of the `RESTART` read time manipulation. `YbResetTransactionReadPoint` therefore always resets, including for catalog snapshots, which fixes the stale relcache after an RC read restart retry (#36877). Parallel workers are skipped for the whole read restart retry (REPEATABLE READ too, not only READ COMMITTED), since they don't carry the caps. `TEST_inject_read_restart_with_probability` now injects the tablet's current time as the restart time, like a real conflicting write, instead of a random point that could be ahead of every clock. ## Upgrade/Rollback safety Adds `PgClientService.RestartTransactionForRetry` and the `global_limit_cap_ht` / `local_limit_caps` read time options to `pg_client.proto`, and removes `ReadTimeOptionsPB.restart_transaction` and `ReadTimeManipulation.RESTART`. These are only exchanged between postgres and its local tserver, which always run the same version, so there is no mixed-version exposure and rollback is unaffected. No gflags or on-disk changes. ## Test plan - [x] `pg_object_locks-test` retry tests (`PgObjectLocksRetryTest.*`, `PgObjectLocksRetryReadTimeTest.*`), with the object lock fastpath on and off: table locks survive read restart retries (REPEATABLE READ and READ COMMITTED, in a txn block and autocommit, READ COMMITTED after an earlier statement in the block, and read-only autocommit statements without a distributed txn); per-tablet local limits stop a tablet from restarting the retry again; the caps end at a COMMIT inside a procedure; clamped catalog reads and non-transactional writes in a retry need no further retries; read restart retries converge with a restart injected on every attempt (REPEATABLE READ txn restart, READ COMMITTED statement retry). The REPEATABLE READ lock cases failed with the eager RPC disabled, and the convergence cases failed without the cap. - [x] `PgObjectLocksRetryTest.IndexStaysConsistentUnderRetriedWrites` (stress, fastpath on and off): two writers, one per query protocol, update the same rows at random isolation levels (REPEATABLE READ, READ COMMITTED, SERIALIZABLE), in autocommit or a txn block, with injected read restarts and wait queue conflicts, while another session builds an index online, runs `yb_index_check` and drops it 20 times; all index checks pass. Passed 20/20 iterations with `-n 20 --tp 1` in both release and fastdebug. - [x] Isolation spec `yb.orig.conflict_retry_retakes_object_locks` (in `yb_isolation_object_locking_schedule`, wait queues on): table locks survive conflict retries (REPEATABLE READ and READ COMMITTED, in a txn block and autocommit, READ COMMITTED after an earlier statement), an advisory lock and a global ACCESS EXCLUSIVE lock survive a conflict retry. On master (375b06827e) the DDL does not wait in the REPEATABLE READ (txn block and autocommit) and READ COMMITTED autocommit cases. - [x] `txn_error_injection-test`, `PgMiniTest` `*Restart*:*Retry*:*Conflict*` (also both in legacy DDL mode: object locking, transactional DDL and concurrent DDL off), `PgObjectLocksFastpathTest.*`, release build. - [x] #36877 repro (1 node, READ COMMITTED writers doing UPDATE and DELETE+INSERT while another session loops ADD/RENAME/DROP COLUMN, `TEST_inject_read_restart_with_probability=0.3`): master gave 52 `Invalid column number` errors in 60s; this change gave 0 in 30s with 4,701 read restarts retried. Jenkins: no brass, urgent cloud.mekodata.ai://efcaf569-041e-4415-a0da-a4023c93197c;adlc/proj/yb_concurrency_control meko_conversation_id:b5b3be22c2e04f42a438f079c25f9aea --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

Commit:dc84e63
Author:Sergei Politov
Committer:GitHub

[#36087] DocDB: Clustered HNSW: search only the tenant of the query (#37540) ## Summary Fourth part of clustered HNSW (#35282), split out of #35285. Since #36089 and #36086 the vectors of each tenant form a separate graph cluster with its own entry point, in memory and in saved chunks. DocDB passed no tenant, though, so every search ran one graph search per tenant of each chunk, at a cost that grows with the number of tenants. - For an index with tenants, pggate encodes the tenant of a vector index search when the query compares each tenant column with a single value: the ybctid prefix of those columns, without the closing range group end. It sends the tenant in the new `PgVectorReadOptionsPB.tenant`. - `DocVectorIndex` searches only that tenant: chunks start from its entry vector, and the insert registry skips the vectors of other tenants. - A search of an index with tenants fails with InvalidArgument when the query selects no tenant, e.g. with an IN list of tenants (#35388). The hnswlib and YbHnsw searches over every tenant cluster are removed. - Postgres passes key values from the pushed down conditions, with parameters and pushable stable functions already evaluated, but only for the key types and collations that routing supports (#34735), and not for index-only scans. A tenant equality that is not pushed down, or on a numeric, float, `CHAR(n)` or non-C collation text column, gives no tenant yet, so the search fails. Complete Postgres support is left to a follow-up. - `filter_accepted` and `filter_removed` of the vector index metrics instantiated `filter_checked`, so all three counted into one metric. The new tests read them. Postgres still checks the tenant equalities on every candidate row. Leaving them out of the pushed down quals is a separate change. ## Upgrade/Rollback safety `PgVectorReadOptionsPB.tenant` is a new optional field, used only by indexes created with the experimental `vector_index_tenant_hash_group_and_range_components`, which must stay 0 until the upgrade is finalized, as before. For such an index, a tserver of this release fails the searches of an older pggate, which sends no tenant; a tserver of an older release, also after a rollback, ignores the tenant and searches all tenants. Indexes without tenants are unaffected. ## Test plan - [x] `./yb_build.sh --cxx-test pgwrapper_pg_vector_index-test --gtest_filter PgVectorIndexPkRoutingTest.TenantsHashKey`, and the same for `TenantsRangeKey`, `TenantsRangeDescKey`, `TenantsBackfill`, `TenantsColocated` and `TenantsSplit`: a query with a single tenant, also as a parameter of a generic plan, finds the rows of every tenant, though the query vector is closest to tenant 0, and the filter checks only vectors of that tenant (it rejects some without the tenant in the request). A tenant without rows finds none, equalities on more key columns than the tenant has work, queries without a tenant equality or with an IN list of tenants fail, and the vector of a deleted row is removed by the filter. Checked in memory, after flushes, in split children that keep the parent chunk, and after compaction. - [x] `./yb_build.sh --cxx-test ann_methods_vector_lsm-test --gtest_filter VectorLSMTest.Tenants/kYbHnswHnswlibWithPayload`: a tenant search returns only the vectors of the tenant from the insert registry, the mutable chunk, saved chunks and a merged chunk, and a search without a tenant fails. - [x] `./yb_build.sh --cxx-test ann_methods_vector_lsm-test --gtest_filter VectorLSMTest.TenantSearchesDuringInserts/kYbHnswHnswlibWithPayload`: tenant searches concurrent with inserts that move vectors from the insert registry to the mutable chunk. - [x] `./yb_build.sh --cxx-test ann_methods_hnswlib_tenant-test`, one test at a time: searches without a tenant of indexes with tenants fail. - [x] `YB_TEST_YB_CONTROLLER=1 ./yb_build.sh --cxx-test pgwrapper_pg_vector_index_clone-test --gtest_filter 'PgVectorIndexBackupRestoreTest.IndexAfterBackendAndPayloadFlagsChange/*'`, one variant at a time: the restored index with tenants is searched for its tenant. - [x] `./yb_build.sh --cxx-test pgwrapper_pg_vector_index_util-test --gtest_filter PgVectorIndexUtilTest.TenantComponents`, and the routing tests `PgVectorIndexPkRoutingTest.HashKey`, `RangeKey` and `MultiColumnHashKey` of `pgwrapper_pg_vector_index-test`. - [x] `./yb_build.sh --cxx-test hnsw_hnsw-test`

Commit:26c1027
Author:Saurav Tiwary
Committer:GitHub

[BACKPORT 2026.2][#36774] YSQL: Send aborted subtxns of transactions tied at commit time to the walsender (#37446) (#38586) ## Summary With transactional DDL, the walsender reads the catalog with the intents of each record's DocDB transaction, skipping the subtransactions rolled back to a savepoint (#36204). The producer sends that aborted subtransaction set on the BEGIN record. The virtual WAL streams DocDB transactions that commit at the same hybrid time as one pg transaction, and ships only the BEGIN of the transaction with the lowest id. The sets of the other transactions never reached the walsender: debug builds hit an assert, and release builds used the set of the other transaction, so catalog reads could see a DDL rolled back to a savepoint (an added column, a table added to the publication). Changes: - The virtual WAL moves the set of every BEGIN it reads, shipped or merged at a commit time tie, into the new `GetConsistentChangesResponsePB.aborted_subtxns` map, keyed by DocDB transaction id with the commit time. Records no longer carry the set, so the walsender gets every set from one place. - The walsender adds the map of each response to its list and drops the entries at or before a commit time once yb_decode has decoded that COMMIT. It drops by commit time because the COMMIT of a merged block carries the id of only one of its transactions. - The per-record set that #36204 added to `YbcPgRowMessage` is removed. - New test flags `TEST_commit_time_override_ht` and `TEST_commit_time_override_txn_ids` make the given transactions commit at the same hybrid time. ## Upgrade/Rollback safety Adds the map field `aborted_subtxns` (8) to `GetConsistentChangesResponsePB`, and moves the set out of `RowMessage.aborted_subtxns` on the records the virtual WAL returns. The response only goes from a tserver to its own postgres (walsender), which run the same version. The code paths are behind `TEST_ysql_yb_enable_replication_slot_transactional_ddl`. Original commit: db3ef76eaeb4506aa67f6a1dd85bdffc4904fcb9 / #37446 ## Test plan New tests force a commit time tie with the new test flags: - `TestPgReplicationSlotWithTxnDdl#commitTimeTieWithAddColumnRolledBackToSavepoint`: an ADD COLUMN rolled back to a savepoint in the transaction whose BEGIN is merged. Fails on master with the walsender assert. - `TestPgReplicationSlotWithTxnDdl#commitTimeTieWithAlterPublicationRolledBackToSavepoint`: the same with ALTER PUBLICATION ADD TABLE, where the merged transaction's first record triggers a publication refresh. Without its set, the rolled-back table is added to the slot's table list. - `CDCSDKConsumptionConsistentChangesTest.TestVWALShipsAbortedSubtxnsOfTransactionalDdl` checks that the set comes only in the response map. Ran: - [x] `./yb_build.sh debug --java-test org.yb.pgsql.TestPgReplicationSlotWithTxnDdl` - [x] `./yb_build.sh debug --cxx-test integration-tests_cdcsdk_consumption_consistent_changes-test --gtest_filter '*TestCommitTimeTie*:*TestVWALShipsAbortedSubtxnsOfTransactionalDdl*'` 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> ## Merge conflicts ### master --> 2026.2 2026.2 lacks #36725, which added `decode_record_context`, the batch's `response` pointer and `resp->` access, and removed `GetDynamicTypeEntity`. On 2026.2 the batch is allocated in `cached_records_context` and reset as a whole, so the new `aborted_subtxn_sets` array needs no separate freeing. - `src/postgres/src/backend/replication/logical/yb_virtual_wal_client.c:47,210` — [missing-prerequisite] added only `aborted_subtxn_sets_context`, not `decode_record_context` - `src/postgres/src/backend/replication/logical/yb_virtual_wal_client.c:301` — [missing-prerequisite] set only `aborted_subtxn_sets_context` to NULL in destroy; 2026.2 doesn't reset the other context pointers - `src/postgres/src/backend/replication/logical/yb_virtual_wal_client.c:733` — [missing-prerequisite] kept 2026.2's `&GetDynamicTypeEntity` argument, then called `AddAbortedSubtxnSets` - `src/yb/yql/pggate/ybc_pg_typedefs.h:783,801` — [missing-prerequisite] added `YbcPgAbortedSubtxnSet` and the two batch fields, without `response` - `src/yb/yql/pggate/ybc_pggate.cc:3246,3256` — [missing-prerequisite] dropped `.response = nullptr`; `resp->aborted_subtxns()` → `resp.aborted_subtxns()` since `resp` is a value on 2026.2 Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

Commit:2930c9c
Author:Sumukh-Phalgaonkar

CDC: POC to populate a CDC table using the xCluster consumer and poller Not for merge. Behind the new test flag `TEST_cdcsdk_enable_cdc_table`, creating a replication slot also creates `public.pg_yb_cdc_changes` in the slot's database. Pollers built on the xCluster consumer then call CDCSDK `GetChanges` on the slot's stream for every published tablet and the sys catalog tablet, and upsert the changes into that table. The walsender cannot consume slots while the flag is set. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Commit:a9dff7f
Author:Yamen Haddad
Committer:GitHub

[BACKPORT 2026.2][#37241] DocDB: ThinClientService: consistent-prefix reads with a server-computed staleness (#37242) (#38270) Fixes #37241 ### Summary Every read through `ThinClientService.Perform` is STRONG, so the tablet leader serves it; for a caller spread across zones (amp's pageserver reading page rows) that crosses an AZ two thirds of the time. This lets a batch ask for a consistent-prefix read instead, at a read time the **tserver** derives from its own clock minus a caller-supplied staleness, so only the cluster's `max_clock_skew_usec` matters, as for YSQL follower reads. - `thin_client.proto`: `ThinReadTimeOptionsPB` gains `bool read_from_followers = 2` and `optional uint32 follower_read_staleness_ms = 3`, additively. - `thin_client_service.cc`: with `read_from_followers`, every read op of the batch runs at `CONSISTENT_PREFIX`. With a staleness and no pinned `read_time` the session reads at `now - staleness`, through `SetFollowerReadTime`, which moves from `pg_client_session.cc` to `pg_client_session_util.{h,cc}` so both follower-read paths share it. A pinned `read_time` (a paging continuation) keeps its snapshot and only takes the consistency level. `InvalidArgument` for: a staleness on a write batch, a staleness without `read_from_followers` (the leader would serve a historical snapshot nobody asked for), a staleness of 0 (it would read at `now`, which no follower can serve) or below `2 * max_clock_skew_usec` (1000 ms by default; a follower usually cannot serve that either, and YSQL refuses the same range in `is_staleness_acceptable`), and `read_from_followers` with neither a `read_time` nor a staleness. `read_from_followers` on a write batch is ignored. - `thin_client_service.cc`: when the batch's read point holds a time after the flush (pinned, staleness-derived, or stamped by the batcher for a multi-tablet batch), it goes into every op paging state that has none. DocDB binds `read_time` only when it stops inside a tablet (`PgsqlReadOperation::SetPagingState`); the paging state that moves a scan to the next tablet (`Tablet::CreatePagingStateForRead`) has none. Without this, a follower scan whose first page ended at a tablet boundary reached the shim with no snapshot, and its continuation was rejected. DocDB reads `paging_state.read_time` only together with `next_row_key`, so the stamp is a report to the client and changes nothing in the tablet. An unpinned single-tablet leader read leaves the read point empty and is unchanged, so such a scan still moves to a newer snapshot at each tablet boundary; that predates this PR, is tracked as #38112, and the `ybthin_read_async` comment now says so. - `yb_thin_client.{h,cc}`: `ybthin_read_async` takes a `const ybthin_read_options*` in place of its bare `read_time_ht`; NULL or a zeroed struct is the leader read it was before. This changes the function's signature (see the notes below). The shim refuses a fresh follower batch with a zero staleness as `YBTHIN_INVALID` before sending it. The paging envelope (version 4) carries the scan's `read_from_followers` next to its read time, so every page of a follower scan asks for a follower at the first page's snapshot without the caller re-passing anything. A scan keeps the replica choice of its first page: follower options on the continuation of a scan that started at the leader are ignored, and a continuation whose paging state holds no read time is a leader read. ### What the follower handoff costs A follower behind the read time hands the read to another replica through existing code: `ReadQuery::WaitForSafeTime` returns `IllegalState` under `ysql_follower_reads_avoid_waiting_for_safe_time`, and `TabletInvoker` retries the next replica. The caller still sees one Perform, but the handoff is not free: that `IllegalState` carries no `TabletServerError` code, so `TabletInvoker::FailToNewReplica` reaches `MarkReplicaFailed` and the local replica stays failed in the tserver's shared meta cache until its next refresh (`lookup_cache_refresh_secs`, 60 s by default). Every follower read of that tablet through that tserver goes to a remote replica for up to that long. This is pre-existing YSQL follower-read behaviour; this PR changes neither `read_query.cc` nor `tablet_rpc.cc`. A follow-up could return a `STALE_FOLLOWER`-coded status so the invoker retries without marking the replica failed, but that changes YSQL behaviour and needs its own discussion with the DocDB owners. ### Notes for the owners - An unpaged follower batch reports `used_read_time_ht = 0`, because the service only echoes the read time inside a paging state (pre-existing). A follow-up could echo it in an additive response field. - C ABI: `ybthin_read_async`'s signature changes in place instead of gaining a second entry point, as agreed with the owners. The library has one caller, which pins a shim release and ships the library in the same image as the binaries built against it, so the two never differ at run time. The header and README say this policy holds while the library has a single caller. `yb_thin_client.h` and the README said the ABI was additive-only; they now say it may change between shim releases, and that only the wire to the tserver is kept backward-compatible. A caller that declares the functions by hand has to update its declaration with the header, because a stale declaration of the old signature still links. ## Upgrade/Rollback safety No AutoFlag or gflag. Nothing is persisted. The wire change is two appended optional fields on `ThinReadTimeOptionsPB`, plus an existing field (`PgsqlPagingStatePB.read_time`) that is now set in one more case. A thin-client paging state is held in client memory, and its continuation is meant for the tserver process that issued it, where its session lives. Usually a restarted tserver answers `SESSION_LOST`: the shim reports that page as `YBTHIN_NETWORK`, and a retry of the continuation as `YBTHIN_READ_RESTART`. That is not guaranteed, though: session ids are a per-process counter, so a restarted tserver can hand the same id to a new session, and a continuation can then be served by a different process than its first page. - **Can an old tserver read what a new thin client sends?** Yes. It ignores both fields and serves the batch STRONG at the leader: unpinned for a fresh batch, at the pinned `read_time` for a continuation. A leader read at that time is never staler than the follower read that was requested, so the shim does not depend on the receiver honouring the fields. REVIEW.md asks for a gate only in that case. The header documents this fallback. - **Can a new tserver read what an old thin client sends?** Yes, and it behaves as before: an old shim sets neither field, so none of the new checks can fire. The one difference an old shim sees is the stamped `paging_state.read_time` at a tablet boundary of a batch whose read point was fixed (pinned, or a multi-tablet batch the batcher stamped). Old shims already read that field mid-tablet and pin the next page to it, so their scan now keeps its snapshot across that boundary instead of moving forward. - **Mixed versions inside the cluster.** The tserver-to-tserver hop is an ordinary `Read` RPC with the existing `consistency_level = CONSISTENT_PREFIX` and `read_time` fields, which every supported tserver already serves, as it does for YSQL follower reads. No new field crosses between tservers. - **Rollback after thin clients have started sending the new fields.** Nothing to undo, because nothing was written. A rolled-back tserver ignores the fields as above. Take a scan whose first page was a follower read on a new tserver: rolling that tserver back restarts it, so the continuation usually fails as above and the caller restarts the scan, which the old binary serves at the leader. If the continuation instead reaches the rolled-back process under a reissued session id, it carries the scan's pinned `read_time` (a new tserver always reports one for a follower page), which old tservers honour, plus `read_from_followers`, which they ignore: a leader read at the scan's snapshot. - **Upgrade with a scan in flight.** The reverse case: a scan whose first page an old tserver served at the leader, unpinned, ending at a tablet boundary, so its paging state holds no read time. Its continuation can reach an upgraded process under a reissued session id. The new tserver refuses `read_from_followers` without a read time, so the shim never sends the flag on a continuation whose paging state holds no read time, whatever the scan asked for: such a page could not be a follower read anyway. The request is then the same as before this PR, whichever process receives it (`FollowerContinuationWithoutAReadTimeReadsAtTheLeader`). - The shim's paging envelope bumps to version 4. It is process-local (it embeds the shim's session index and generation), so it crosses no process boundary. - The C ABI change is not additive: the fourth parameter of `ybthin_read_async` changes from `uint64_t read_time_ht` to `const ybthin_read_options*`. It has no upgrade or rollback consequence for a running cluster, because the library is not part of the tserver: its caller takes the new signature when it moves its pinned shim release, in the same build that compiles its binaries. ### Test Plan `yb_thin_client-itest`, `PgThinClientFollowerReadTest.*` (RF3 mini cluster). The follower assertions use the tablet's `consistent_prefix_read_requests` counter per replica, which `ReadQuery` bumps for every CONSISTENT_PREFIX read that reaches a replica (before the safe-time decision), so it is process-scoped and transport-independent; no `TEST_assert_reads_served_by_follower`. No fixed sleeps: the tests `WaitFor` the follower's `SafeTime(RequireLease::kFalse)` to pass the Perform tserver's clock minus the staleness, and the rows to be older than that. - `StaleReadIsServedByAFollower`: a Perform sent to a follower's tserver with the flag and a staleness returns the rows; that follower's counter rises by one, the leader's and the other follower's by zero. - `LaggingFollowerHandsTheReadOver`: the follower the Perform lands on has `UpdateConsensus` paused for longer than the staleness; the read still returns the rows well under the RPC deadline; the paused follower's counter rises by exactly one (the refused attempt) and exactly one other replica's rises by one. - `PagedFollowerScanHoldsItsSnapshot`: rows written between page one and page two are not seen; every page reports the same `used_read_time_ht`; continuations pass no options; the follower's counter rises by one per page, the others' not at all. - `PagedFollowerScanCrossesTablets`: a range-sharded table split at 10/100/150, with no leader on the Perform's tserver. Page one (limit 32) ends at the 10-row first tablet, so its paging state comes from `CreatePagingStateForRead`. The scan completes in 8 pages, every page reports page one's nonzero `used_read_time_ht`, rows written into the last tablet after page one are not seen, and the reader's replicas take exactly one consistent-prefix read per page, the other tservers none. Before the fix, page one reported 0 and the continuation failed with `YBTHIN_INVALID`. - `ReadWithoutOptionsStaysOnTheLeader`: `ybthin_read_async` with no options sees rows written a moment ago, and the leader's `Read` RPC count rises while the Perform's own tserver's does not. - `FollowerContinuationWithoutAReadTimeReadsAtTheLeader`: a leader scan of a table split at 10 ends its first page at the tablet boundary with no read time; the test sets the follower flag in the shim's envelope, as a first page served by an old tserver would leave it. The continuation succeeds and no replica takes a consistent-prefix read. Without the fix the server refuses it with `InvalidArgument`. - `ContinuationKeepsTheScansReplicaChoice`: a scan whose first page had no options is continued with follower options; every page succeeds and no replica's counter moves. - `StalenessOutsideAFollowerReadIsRejected`, `TooSmallStalenessIsRejectedByTheService`: batches built on the proxy directly get `InvalidArgument` for a staleness on a write batch, a staleness on a read batch without `read_from_followers`, a zero staleness, and a staleness of 1 ms and of one below `2 * max_clock_skew_usec`. - `ZeroStalenessIsInvalidAtTheShim`: `YBTHIN_INVALID` for a fresh follower batch with staleness 0; the same options pinned to an earlier batch's `used_read_time_ht` succeed. All 38 `yb_thin_client-itest` tests pass, including the four `PgThinClientExternal*` tests, which run tservers as separate processes. --------- Original commit: 2e06e07760fc0a3ac02ec64f4581970b2be4d7d8 / #37242 Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>

Commit:0f57f7a
Author:Yamen Haddad
Committer:GitHub

[BACKPORT 2026.1][#37241] DocDB: ThinClientService: consistent-prefix reads with a server-computed staleness (#37242) (#38271) Fixes #37241 ### Summary Every read through `ThinClientService.Perform` is STRONG, so the tablet leader serves it; for a caller spread across zones (amp's pageserver reading page rows) that crosses an AZ two thirds of the time. This lets a batch ask for a consistent-prefix read instead, at a read time the **tserver** derives from its own clock minus a caller-supplied staleness, so only the cluster's `max_clock_skew_usec` matters, as for YSQL follower reads. - `thin_client.proto`: `ThinReadTimeOptionsPB` gains `bool read_from_followers = 2` and `optional uint32 follower_read_staleness_ms = 3`, additively. - `thin_client_service.cc`: with `read_from_followers`, every read op of the batch runs at `CONSISTENT_PREFIX`. With a staleness and no pinned `read_time` the session reads at `now - staleness`, through `SetFollowerReadTime`, which moves from `pg_client_session.cc` to `pg_client_session_util.{h,cc}` so both follower-read paths share it. A pinned `read_time` (a paging continuation) keeps its snapshot and only takes the consistency level. `InvalidArgument` for: a staleness on a write batch, a staleness without `read_from_followers` (the leader would serve a historical snapshot nobody asked for), a staleness of 0 (it would read at `now`, which no follower can serve) or below `2 * max_clock_skew_usec` (1000 ms by default; a follower usually cannot serve that either, and YSQL refuses the same range in `is_staleness_acceptable`), and `read_from_followers` with neither a `read_time` nor a staleness. `read_from_followers` on a write batch is ignored. - `thin_client_service.cc`: when the batch's read point holds a time after the flush (pinned, staleness-derived, or stamped by the batcher for a multi-tablet batch), it goes into every op paging state that has none. DocDB binds `read_time` only when it stops inside a tablet (`PgsqlReadOperation::SetPagingState`); the paging state that moves a scan to the next tablet (`Tablet::CreatePagingStateForRead`) has none. Without this, a follower scan whose first page ended at a tablet boundary reached the shim with no snapshot, and its continuation was rejected. DocDB reads `paging_state.read_time` only together with `next_row_key`, so the stamp is a report to the client and changes nothing in the tablet. An unpinned single-tablet leader read leaves the read point empty and is unchanged, so such a scan still moves to a newer snapshot at each tablet boundary; that predates this PR, is tracked as #38112, and the `ybthin_read_async` comment now says so. - `yb_thin_client.{h,cc}`: `ybthin_read_async` takes a `const ybthin_read_options*` in place of its bare `read_time_ht`; NULL or a zeroed struct is the leader read it was before. This changes the function's signature (see the notes below). The shim refuses a fresh follower batch with a zero staleness as `YBTHIN_INVALID` before sending it. The paging envelope (version 4) carries the scan's `read_from_followers` next to its read time, so every page of a follower scan asks for a follower at the first page's snapshot without the caller re-passing anything. A scan keeps the replica choice of its first page: follower options on the continuation of a scan that started at the leader are ignored, and a continuation whose paging state holds no read time is a leader read. ### What the follower handoff costs A follower behind the read time hands the read to another replica through existing code: `ReadQuery::WaitForSafeTime` returns `IllegalState` under `ysql_follower_reads_avoid_waiting_for_safe_time`, and `TabletInvoker` retries the next replica. The caller still sees one Perform, but the handoff is not free: that `IllegalState` carries no `TabletServerError` code, so `TabletInvoker::FailToNewReplica` reaches `MarkReplicaFailed` and the local replica stays failed in the tserver's shared meta cache until its next refresh (`lookup_cache_refresh_secs`, 60 s by default). Every follower read of that tablet through that tserver goes to a remote replica for up to that long. This is pre-existing YSQL follower-read behaviour; this PR changes neither `read_query.cc` nor `tablet_rpc.cc`. A follow-up could return a `STALE_FOLLOWER`-coded status so the invoker retries without marking the replica failed, but that changes YSQL behaviour and needs its own discussion with the DocDB owners. ### Notes for the owners - An unpaged follower batch reports `used_read_time_ht = 0`, because the service only echoes the read time inside a paging state (pre-existing). A follow-up could echo it in an additive response field. - C ABI: `ybthin_read_async`'s signature changes in place instead of gaining a second entry point, as agreed with the owners. The library has one caller, which pins a shim release and ships the library in the same image as the binaries built against it, so the two never differ at run time. The header and README say this policy holds while the library has a single caller. `yb_thin_client.h` and the README said the ABI was additive-only; they now say it may change between shim releases, and that only the wire to the tserver is kept backward-compatible. A caller that declares the functions by hand has to update its declaration with the header, because a stale declaration of the old signature still links. ## Upgrade/Rollback safety No AutoFlag or gflag. Nothing is persisted. The wire change is two appended optional fields on `ThinReadTimeOptionsPB`, plus an existing field (`PgsqlPagingStatePB.read_time`) that is now set in one more case. A thin-client paging state is held in client memory, and its continuation is meant for the tserver process that issued it, where its session lives. Usually a restarted tserver answers `SESSION_LOST`: the shim reports that page as `YBTHIN_NETWORK`, and a retry of the continuation as `YBTHIN_READ_RESTART`. That is not guaranteed, though: session ids are a per-process counter, so a restarted tserver can hand the same id to a new session, and a continuation can then be served by a different process than its first page. - **Can an old tserver read what a new thin client sends?** Yes. It ignores both fields and serves the batch STRONG at the leader: unpinned for a fresh batch, at the pinned `read_time` for a continuation. A leader read at that time is never staler than the follower read that was requested, so the shim does not depend on the receiver honouring the fields. REVIEW.md asks for a gate only in that case. The header documents this fallback. - **Can a new tserver read what an old thin client sends?** Yes, and it behaves as before: an old shim sets neither field, so none of the new checks can fire. The one difference an old shim sees is the stamped `paging_state.read_time` at a tablet boundary of a batch whose read point was fixed (pinned, or a multi-tablet batch the batcher stamped). Old shims already read that field mid-tablet and pin the next page to it, so their scan now keeps its snapshot across that boundary instead of moving forward. - **Mixed versions inside the cluster.** The tserver-to-tserver hop is an ordinary `Read` RPC with the existing `consistency_level = CONSISTENT_PREFIX` and `read_time` fields, which every supported tserver already serves, as it does for YSQL follower reads. No new field crosses between tservers. - **Rollback after thin clients have started sending the new fields.** Nothing to undo, because nothing was written. A rolled-back tserver ignores the fields as above. Take a scan whose first page was a follower read on a new tserver: rolling that tserver back restarts it, so the continuation usually fails as above and the caller restarts the scan, which the old binary serves at the leader. If the continuation instead reaches the rolled-back process under a reissued session id, it carries the scan's pinned `read_time` (a new tserver always reports one for a follower page), which old tservers honour, plus `read_from_followers`, which they ignore: a leader read at the scan's snapshot. - **Upgrade with a scan in flight.** The reverse case: a scan whose first page an old tserver served at the leader, unpinned, ending at a tablet boundary, so its paging state holds no read time. Its continuation can reach an upgraded process under a reissued session id. The new tserver refuses `read_from_followers` without a read time, so the shim never sends the flag on a continuation whose paging state holds no read time, whatever the scan asked for: such a page could not be a follower read anyway. The request is then the same as before this PR, whichever process receives it (`FollowerContinuationWithoutAReadTimeReadsAtTheLeader`). - The shim's paging envelope bumps to version 4. It is process-local (it embeds the shim's session index and generation), so it crosses no process boundary. - The C ABI change is not additive: the fourth parameter of `ybthin_read_async` changes from `uint64_t read_time_ht` to `const ybthin_read_options*`. It has no upgrade or rollback consequence for a running cluster, because the library is not part of the tserver: its caller takes the new signature when it moves its pinned shim release, in the same build that compiles its binaries. ### Test Plan `yb_thin_client-itest`, `PgThinClientFollowerReadTest.*` (RF3 mini cluster). The follower assertions use the tablet's `consistent_prefix_read_requests` counter per replica, which `ReadQuery` bumps for every CONSISTENT_PREFIX read that reaches a replica (before the safe-time decision), so it is process-scoped and transport-independent; no `TEST_assert_reads_served_by_follower`. No fixed sleeps: the tests `WaitFor` the follower's `SafeTime(RequireLease::kFalse)` to pass the Perform tserver's clock minus the staleness, and the rows to be older than that. - `StaleReadIsServedByAFollower`: a Perform sent to a follower's tserver with the flag and a staleness returns the rows; that follower's counter rises by one, the leader's and the other follower's by zero. - `LaggingFollowerHandsTheReadOver`: the follower the Perform lands on has `UpdateConsensus` paused for longer than the staleness; the read still returns the rows well under the RPC deadline; the paused follower's counter rises by exactly one (the refused attempt) and exactly one other replica's rises by one. - `PagedFollowerScanHoldsItsSnapshot`: rows written between page one and page two are not seen; every page reports the same `used_read_time_ht`; continuations pass no options; the follower's counter rises by one per page, the others' not at all. - `PagedFollowerScanCrossesTablets`: a range-sharded table split at 10/100/150, with no leader on the Perform's tserver. Page one (limit 32) ends at the 10-row first tablet, so its paging state comes from `CreatePagingStateForRead`. The scan completes in 8 pages, every page reports page one's nonzero `used_read_time_ht`, rows written into the last tablet after page one are not seen, and the reader's replicas take exactly one consistent-prefix read per page, the other tservers none. Before the fix, page one reported 0 and the continuation failed with `YBTHIN_INVALID`. - `ReadWithoutOptionsStaysOnTheLeader`: `ybthin_read_async` with no options sees rows written a moment ago, and the leader's `Read` RPC count rises while the Perform's own tserver's does not. - `FollowerContinuationWithoutAReadTimeReadsAtTheLeader`: a leader scan of a table split at 10 ends its first page at the tablet boundary with no read time; the test sets the follower flag in the shim's envelope, as a first page served by an old tserver would leave it. The continuation succeeds and no replica takes a consistent-prefix read. Without the fix the server refuses it with `InvalidArgument`. - `ContinuationKeepsTheScansReplicaChoice`: a scan whose first page had no options is continued with follower options; every page succeeds and no replica's counter moves. - `StalenessOutsideAFollowerReadIsRejected`, `TooSmallStalenessIsRejectedByTheService`: batches built on the proxy directly get `InvalidArgument` for a staleness on a write batch, a staleness on a read batch without `read_from_followers`, a zero staleness, and a staleness of 1 ms and of one below `2 * max_clock_skew_usec`. - `ZeroStalenessIsInvalidAtTheShim`: `YBTHIN_INVALID` for a fresh follower batch with staleness 0; the same options pinned to an earlier batch's `used_read_time_ht` succeed. All 38 `yb_thin_client-itest` tests pass, including the four `PgThinClientExternal*` tests, which run tservers as separate processes. --------- Original commit: 2e06e07760fc0a3ac02ec64f4581970b2be4d7d8 / #37242 Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>

Commit:b7ba9ca
Author:Bvsk Patnaik
Committer:GitHub

[BACKPORT 2026.2][#37328] YSQL: Detect deadlocks of fast path writes without global object locks (#37359) (#38633) ## Summary ### Background The tserver tags a fast path write (a write outside a distributed transaction) with its object lock owner, so that the deadlock detector sees a cycle that runs through object locks and row locks (#24630). Two kinds of transaction take local object locks for their DMLs but no global ones, so that DDLs with global object locks wait for them: - one that latched `ysql_yb_enable_global_object_locking_infra` off, during its promotion (#37358, #36572); - any transaction on a node with `disable_global_object_locks` on (#37334, #37355). `using_table_locks_globally` stays false for both, so tablets keep checking their catalog version. ### Problem The tserver tags fast path writes only when `using_table_locks_globally` is set, so a fast path write from a transaction without global object locks carries no lock owner. If that writer waits on a row lock while a DDL waits on the writer's local object lock, the deadlock detector does not see the cycle. Both wait until a statement times out. Example, during the `ysql_yb_enable_global_object_locking_infra` promotion: 1. ts-2, after the promotion: `BEGIN; UPDATE test SET v = v + 1 WHERE k = 1;` holds the row lock on k = 1. 2. ts-1, transaction latched off: `UPDATE test SET v = v + 1 WHERE k = 1;` takes a local lock on `test`, then waits on the row lock. 3. ts-2: `ALTER TABLE test ADD COLUMN v1 INT;` waits on ts-1's local lock. ### Root cause `PrepareOperations` gates the tag on `using_table_locks_globally`. Before #37358, only a transaction with global object locks took object locks at all, so "takes object locks" and "tablets skip the catalog version check" were the same fact. #37358 separates them: a transaction latched off takes local locks, but its tablets must still check its catalog version, and #37334 adds a node-wide way to run that way. The gate kept the second meaning. ### Impact A cycle between a fast path write from a transaction without global object locks and a DDL with global object locks: while `ysql_yb_enable_global_object_locking_infra` is being promoted, and for as long as any node runs with `disable_global_object_locks` on. The two sessions wait until a statement times out, instead of the detector aborting one of them. ### Fix Add `PgPerformOptionsPB.using_table_locks_locally`. pggate sets it from `UsingObjectLocksLocally()`, the predicate that decides whether the transaction sends its locks to the tserver. `PrepareOperations` tags a fast path write when it is set, instead of when `using_table_locks_globally` is set. `DoAcquireObjectLock` checks the field. Before #37358 it checked `is_using_table_locks`; #37358 removed that check, since a transaction without global locks sends local ones. An object lock request from a transaction that does not take object locks means a pggate sender skipped `UsingObjectLocksLocally()`. In release builds `RSTATUS_DCHECK` returns `IllegalState`. ### Why the fix works `using_table_locks_locally` and the locks a transaction sends come from the same predicate, so a fast path write carries an owner exactly when its transaction can hold an object lock. The tablet then waits on the row lock as that owner (`ShouldWaitAsDifferentTxn` in `conflict_resolution.cc`), and the detector joins that wait with the DDL's wait on the same owner's object lock. A transaction with global object locks also uses object locks locally, so its writes stay tagged as before. Writes in a distributed transaction are not affected: the gate skips them, because the transaction itself owns the locks. ## Upgrade/Rollback safety The field is new (tag 42) and defaults to false. pggate sends it only to the tserver of its own node, which runs the same binary, because the tserver starts Postgres. No mixed-version pair sees it. Original commit: 7cda819157070e04adcc6b14b7d7ae1d246b894f / #37359 ## Test plan `PgObjectLocksTestLegacyTxnVsNewDdl.LegacyFastPathWriteDeadlockIsDetected` (both parameter cases): a fast path UPDATE from a transaction latched off waits on a row lock held by a transaction latched on, which then runs ALTER TABLE. Exactly one of the two writers commits. With the gate reverted to `using_table_locks_globally`, both cases fail: the ALTER hits its statement timeout. `PgObjectLocksTestGlobalLocksRollingRestart.FastPathWriteDeadlockIsDetected`: the same cycle with the fast path writer on a node restarted with `disable_global_object_locks` on (#37355), which keeps its local locks. With the gate reverted, it fails the same way. The object lock request check: every object lock request goes through it, so every test above exercises it; `PgObjectLocksTestRF1.TestSanity` and `PgObjectLocksTestRF1.TestDeadlockFastPath` also pass (release, with the three tests above, on the current code). With the setter forced to false, `PgObjectLocksTestRF1.TestSanity` fails at this check, which returns "Object lock request from a transaction that does not use object locks locally". ``` ./yb_build.sh release --cxx-test pgwrapper_pg_object_locks-test \ --gtest_filter '*FastPathWriteDeadlockIsDetected*:PgObjectLocksTestRF1.TestSanity:PgObjectLocksTestRF1.TestDeadlockFastPath' ``` Part of stack #37903, bottom to top: #37358, #37334, #37359. Fixes #37328 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> ## Merge conflicts ### master --> 2026.2 No conflicts. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

Commit:f44acfa
Author:Bvsk Patnaik
Committer:GitHub

[BACKPORT 2026.2][#36572] YSQL: Keep new indexes and foreign keys intact for writers that predate object locking (#37358) (#38631) ## Summary ### Background While the `ysql_yb_enable_global_object_locking_infra` AutoFlag is being promoted, some transactions latched it on and some off: each transaction latches the flag when it starts. DDLs wait for concurrent DMLs only through object locks since 7dd192f0d83 (#31758), which replaced CREATE INDEX's `YbWaitForBackendsCatalogVersion` barriers with `WaitForLockers(ShareLock)`. - **object locks** (`UsingObjectLocksLocally()`): the transaction sends PG's locks to the tserver. Off in initdb, binary upgrade, parallel workers, autonomous DDLs, transactions that read the catalog as of `yb_read_time`, and with `enable_object_locking_for_table_locks` off. - **global object locks** (`UsingObjectLocksGloballyForCurrentTxn()`): object locks, and the transaction latched the infra flag on. Its DDLs take global locks and run in the transaction block. This is safe because the flag is promoted only once every tserver runs the new binary. - **autonomous DDL**: a DDL of a transaction that latched the flag off runs in its own DDL transaction. | | DML object locks | DDL object locks | mode of catalog operations | catalog cache refresh | |---|---|---|---|---| | latched off, before | none | none | legacy mode of catalog operations | between transactions | | latched off, after | local (RowExclusive or weaker) | none (autonomous DDL) | PG mode of catalog operations | on a new lock (AcceptInvalidationMessages) | | latched on | local | global | PG mode of catalog operations | on a new lock (AcceptInvalidationMessages) | ### Problem A transaction that started before the promotion runs next to DDLs that started after it. 1. **A DDL does not wait for an older writer.** CREATE INDEX misses the writer's row: ```sql -- ts-1, started before the promotion BEGIN; UPDATE test SET v = 42 WHERE k = 1; -- ts-2, after the promotion CREATE INDEX test_v_idx ON test(v); -- does not wait; backfill reads v = 0 -- ts-1 COMMIT; -- index keeps (0, k = 1) SELECT yb_index_check('test_v_idx'::regclass); -- inconsistent index row ``` ADD FOREIGN KEY does not wait for an older DELETE in the same way. 2. **An older writer uses a catalog cache from before the DDL.** It read `test`, CREATE INDEX ran elsewhere, and its INSERT is planned without the index. Today the tablet rejects it with 40001 (schema version mismatch); after #33037 it would commit with no index entry, or skip a new foreign key check. ### Root cause 1. `PgSession::AcquireObjectLock` returned before any RPC when the transaction latched the infra flag off, so `WaitForLockers` and the DDL's global locks saw no holder. Since 7dd192f0d83. 2. `AcceptInvalidationMessages` refreshed only under `YBCIsObjectLockingEnabled()`, which reads the latched infra flag, so such a transaction never refreshed mid-transaction, unlike PG. ### Fix Two predicates, and every caller asks one of them: - `UsingObjectLocksLocally()` gates the locks a transaction sends, the refresh on a new lock and the mode of catalog operations, legacy or PG (`YBCIsLegacyModeForCatalogOps()`). - `UsingObjectLocksGloballyForCurrentTxn()` = that && the latched infra flag. It gates global and session locks, `WaitForLockers`, DDL in a transaction block, `using_table_locks_globally` (the skip of the tablets' catalog version check and of the DDL abort), and the DDL-side callers in PG. A transaction that latched the flag off sends locks of mode RowExclusive or weaker and no session locks; PG still records every lock locally, as upstream does. It releases them at commit, abort and ROLLBACK TO SAVEPOINT, and keeps both old fences. ### Why the fix works Only the non-obvious parts: - **The refresh and the local locks ship together.** After a refresh the transaction's requests carry the new catalog version, so the tablets' catalog version check no longer stops them. What stops a reader from reading a table rewritten after its snapshot is then its AccessShare lock, which the rewrite waits for, exactly as for a transaction with global object locks. ### Known limitations 1. A transaction that latched the infra flag off loses the transparent single-statement retry on "The catalog snapshot used for this transaction has been invalidated" (postgres.c, keyed on the legacy mode of catalog operations). 2. An autonomous DDL sends no locks, but PG still records the ones it takes. A later statement of the same transaction that asks for the same lock (a SELECT after CREATE VIEW on the same table) finds it held, so PG does not ask YB and the lock is never sent; a DDL with global object locks does not wait for that reader. `enable_object_locking_for_table_locks` keeps its meaning: off means no object locks, local or global. So a running cluster moves between global object locking and no object locks in two rolling restarts, with #37334's `disable_global_object_locks` flag as the middle step: - To turn object locks off: first set `disable_global_object_locks=true`, then set `enable_object_locking_for_table_locks=false`. - To turn them on: first set `enable_object_locking_for_table_locks=true` with `disable_global_object_locks=true`, then set `disable_global_object_locks=false`. One rolling restart is not safe in either direction (#37355): while it runs, a node with `enable_object_locking_for_table_locks` off takes no local object locks, so a DDL with global object locks on another node does not wait for that node's writers. #37958 tracks the docs for this procedure. `ysql_yb_enable_object_locking_infra` (GUC `yb_enable_object_locking_infra`) is renamed `ysql_yb_enable_global_object_locking_infra`: with local locks now independent of it, it gates global object locks and what needs them (transactional DDL, concurrent DDL, the tablets' catalog version skip). It is only on master and the unreleased 2026.2, so it gets no deprecated alias. The request field `is_using_table_locks` is renamed `using_table_locks_globally` for the same reason: a DML-only transaction with it set holds only local locks. ## Upgrade/Rollback safety No new wire field: the renamed request fields keep their field numbers. The AutoFlag `ysql_yb_enable_object_locking_infra` is renamed with no alias. The old name is only on master and the 2026.2 branch. This PR will be backported to 2026.2, and no release has been cut from that branch so far, so no released cluster has the old name. Original commit: f23c0ff6d806ca95f999a845bcfaa6e48ee62d98 / #37358 ## Test plan New tests in `src/yb/yql/pgwrapper/pg_object_locks-test.cc`. Unless noted, each runs with the older transaction's node promoted and not promoted before the DDL (`PgObjectLocksTestLegacyTxnVsNewDdl`). - `CreateIndexWaitsForLegacyWriter`: two older writers, one REPEATABLE READ and one READ COMMITTED, each have an UPDATE open; CREATE INDEX on another node must wait for both, and the index must have the updated values. Before this PR: CREATE INDEX does not wait and the index is corrupt (`inconsistent index row`). - `AddForeignKeyWaitsForLegacyDelete`: ADD FOREIGN KEY must wait for an older DELETE of a parent row. Before this PR: it does not wait. - `LegacyWriteMaintainsNewIndex`: an RR and an RC older transaction read the table, CREATE INDEX runs elsewhere, then each inserts a row; the index must have both. Before this PR: the INSERT fails with 40001 `schema version mismatch`. - `LegacyDeleteChecksNewForeignKey`: after ADD FOREIGN KEY elsewhere, an older transaction's DELETE of a referenced parent row must fail with a foreign key violation. Before this PR: the DELETE is planned without the new foreign key, because the transaction never refreshes. - `LegacyReadOnlyTxnReleasesLocksAtCommit`, `LegacyReadOnlyTxnReleasesLocksAtRollbackToSavepoint`, `LegacyWriterReleasesLocksAtRollback`: a DDL must not wait for an older transaction after its COMMIT, ROLLBACK TO SAVEPOINT or ROLLBACK. Before this PR: such a transaction took no locks, so there was nothing to release. With this PR's locks but without the matching release change, the DDL hits its statement timeout. - `TableRewriteWaitsForLegacyReader`: ALTER COLUMN TYPE (a table rewrite) must wait for an older reader's AccessShare lock, and the reader must still read its 10 rows. Before this PR: the ALTER does not wait, and the reader keeps its old catalog. - `NewDdlDoesNotWaitForLockFirstTakenInDdlMode`: pins known limitation 2. The older transaction runs CREATE VIEW on the table, then SELECTs it; the SELECT's AccessShare is not sent, so ALTER TABLE on another node does not wait for it, and the reader still commits. Before this PR: no locks were sent at all, so the ALTER did not wait either. - `LegacyCreateIndexSucceeds`: a CREATE INDEX in a transaction that latched the flag off must succeed. It takes a session lock through `LockRelationIdForSession`, which FATALs unless PG holds the transaction lock in its own lock table, so the locks pggate does not send must still be recorded by PG. Before this PR: passes; no locks were sent. - `LegacyDdlLeavesNoStaleCatalogEntry` (not parameterized): an autonomous DDL must not refresh its cache and keep stale entries after it ends. Before this PR: such a DDL never refreshed. Making the new refresh apply to autonomous DDLs fails it with `column "w" does not exist`. - `LegacyRewriteNeverEmptiesLegacyReader` (not parameterized): an autonomous table rewrite takes no locks, so nothing makes it wait for an older reader; the reader must get an error or the rows it saw before, never the empty new table. Before this PR: the reader never refreshed. - `LegacyCreateIndexAcrossPromotion` (not parameterized): an autonomous CREATE INDEX blocks between phases (`ysql_yb_test_block_index_phase`), the infra flag is promoted, and the CREATE INDEX must finish without a FATAL. Before this PR: no object locks were sent, before or after the promotion. - `ObjectLockingPromotionUpgradeTest.CreateIndexWaitsForLegacyWriterAcrossPromotion` (`upgrade-tests`): upgrade from 2025.1.1.0, open a writer before the AutoFlags are promoted, then run CREATE INDEX. It must wait for the writer, the index must have the writer's row, and the count must go through the new index (EXPLAIN). Before the promotion, `pg_locks` on the writer's node must show its granted RowExclusiveLock. Release builds only: debug builds have object locking off by default. Before this PR: the writer holds no lock, so CREATE INDEX does not wait. - `AutonomousDdlKeepsLegacyTxnLocks`: an older reader holds AccessShare on a table, then runs an autonomous DDL in the same transaction; a table rewrite with global object locks must still wait for the reader. Without the tserver's skip of the release for autonomous DDL sessions, the rewrite does not wait. - `LegacyCursorStoppedByCatalogVersionCheck` (not parameterized): an older transaction reads from a cursor opened before a REVOKE; the read takes no new lock, so it reaches the tablet with the old catalog version and must fail the tablets' catalog version check. Sending `using_table_locks_globally` for a transaction with only local locks makes the read succeed. - `CursorWithGlobalLocksReadsToEndAfterRevoke` (not parameterized): the same steps for a transaction with global object locks. The tablets skip the check and PG checks privileges only when the cursor is opened, so the cursor reads to the end, as in PG; the transaction's next statement fails with `permission denied`. Waits in `CreateIndexWaitsForLegacyWriter`, `AddForeignKeyWaitsForLegacyDelete`, `TableRewriteWaitsForLegacyReader`, `LegacyDdlLeavesNoStaleCatalogEntry` and the upgrade test wait on the condition (`pg_locks`, ASH, `pg_stat_activity`), not on a fixed sleep. Changed tests: - `TestCatalogOpModeStableAcrossPromotion` (#29766): a transaction with object locks now reads the catalog with a catalog snapshot read time from its first statement, so the test expects that read path both before and after the infra flag is promoted under the open transaction. Keying catalog reads on global object locks again makes it fail at the pre-promotion read. - `LegacyTxnKeepsCatalogVersionCheck` (#32224) is removed. The older reader now refreshes on its next lock, so PG's privilege check stops it (`permission denied`) before the read reaches the tablet, and the test no longer shows the tablets' catalog version check. `LegacyCursorStoppedByCatalogVersionCheck` shows it instead. - `PgLocksV76UpgradeTest.TestPgLocksViewAdvisoryLocksSupport`: after finalize, the open transaction's `pg_locks` read refreshes on its new lock and sees the new row type when table locks are on; with table locks off it takes no locks and keeps the stale row type. The test reads `enable_object_locking_for_table_locks` from the tserver to pick the expected outcome: release builds default it to on and debug builds to off, so each build type covers one outcome. ``` ./yb_build.sh release --cxx-test pgwrapper_pg_object_locks-test \ --gtest_filter '*LegacyTxnVsNewDdl*:PgObjectLocksTestMixModeDuringPromotion.*:*AbortTxnsInMixedMode*' ./yb_build.sh release --cxx-test integration-tests_pg_locks_v76_upgrade-test ``` Results, release, each test in its own process, on the head merged with master: - `pgwrapper_pg_object_locks-test`: 108 passed. - `pgwrapper_pg_read_time-test` (with #36925's `yb_read_time` tests): 167 passed. - `integration-tests_object_locking_promotion_upgrade-test`: pass. - `integration-tests_pg_locks_v76_upgrade-test`: table locks off passes (`Returned row contains 24 attributes, but query expects 21`); table locks on failed once at the first pg_locks read with `Couldn't fetch old transactions`, the #37761 flake that #37814 fixes; it passed on earlier runs. - Mutation checks: each new or changed test above fails with the behavior it pins reverted. Part of stack #37903, bottom to top: #37358, #37334, #37359. Jenkins: no brass Fixes #36572 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> ## Merge conflicts ### master --> 2026.2 No conflicts. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

Commit:387e8b1
Author:Kai Franz
Committer:Kai Franz

DocDB: Record the YSQL command id in intents and filter own intents by it Store the PG command id of the writing statement in each strong write intent, and let a read that carries the reading statement's command id decide visibility of its own transaction's intents by it, the way PG compares a tuple's cmin with the snapshot's curcid. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Commit:9c1e51f
Author:IshanChhangani
Committer:GitHub

[BACKPORT 2025.2][#16670] DocDB: dist-trace: generate request trace attributes for every proto, tag CDC stream ids (#35037) (#38422) Backport stack for 2025.2, part 4 of 4, in master order: #34146 → #34156 → #34194 → #35037. Depends on https://github.com/yugabyte/yugabyte-db/pull/38421; review only the top commit. ## Summary The pb proxy stub drains `(yb.rpc.trace)` request fields through a `TracingAttributes(const PB&)` free function that was emitted into `.messages.cc`, so a proto only got request attributes on its RPC spans if it opted into lightweight messages. #34146 worked around that by moving `cdc_service`, `tserver_admin`, `master_admin` and `master_client` to `MESSAGES_PROTO_FILES`, which generates unused lightweight classes for every message in those files. A new trace generator in `protoc-gen-yrpc` emits `.trace.{h,cc}` for every proto file (unconditionally, so the CMake output list stays static) with the `TracingAttributes` / `AppendTracingAttributes` functions for its tagged messages. The header includes the imports' `.trace.h`, so a service whose request types live in another file (`tserver_service.proto` uses `tserver.proto`'s messages) still compiles. The proxy includes it when any request field is tagged and no longer needs `.messages.h` for the pb path; the lightweight generator keeps only the LW form. The four protos go back to `PROTO_FILES`. Attributes are still produced only in the proxy stub at call time when a trace is active, never during serialization. CDC request messages now tag `stream_id`, `db_stream_id` and `stream_ids` (as requested on #34146), so GetChanges, GetCheckpoint, SetCDCCheckpoint and the other stream-scoped RPCs carry the stream id on their spans. ## Upgrade/Rollback safety The `.proto` change only adds `(yb.rpc.trace)` field options on existing CDC request fields. Field numbers, types and wire encoding are unchanged, so mixed-version clusters and rollback are unaffected. The generated `.trace.*` files are build-time only. Original commit: f1e124bbdbcd32b7ce51fc3f568759fabe9f649a / #35037 ## Test plan - [x] `./yb_build.sh release --cxx-test lwproto-test` (plain-PB `TracingAttributes` now comes from `rtest.trace.h`) - [x] `./yb_build.sh release --cxx-test dist_trace-test --gtest_filter 'DistTraceTest.TestBackfillBackendJoinsQueryTrace:DistTraceRpcTest.TestRpcCrossing*'` — `TestBackfillBackendJoinsQueryTrace` now also asserts `req.tablet_id` on the `BackfillIndex` span, a PB-only RPC ## Merge conflicts 2025.2 has one proxy stub per method instead of master's pb_/lw_ pair (#29295), and no `map_entry` `ExtractKey`/`ExtractValue` helpers. Trace attributes match master. - `src/yb/gen_yrpc/proxy_generator.cc` — took master's `ProxyNeedsTrace` and `.trace.h` include. In 2025.2's single stub the pb_ drain (`kPbTraceAttrs`) no longer depends on `need_messages_`, as on master; a lightweight method keeps `kTraceAttrs`. - `src/yb/gen_yrpc/messages_generator.cc` — took master's removal of the pb_ `TracingAttributes` declaration; did not add master's `map_entry` `ExtractKey`/`ExtractValue` block, which 2025.2 never had. - `src/yb/tserver/CMakeLists.txt` — also dropped the `MESSAGES TRUE` on `tserver_service_proto` that the #34146 backport (#38401) added for the pb_ drain. `.trace.h` now serves that drain, so after this PR the tserver, master and CDC CMake files match plain 2025.2. Co-authored-by: ellabaron-code <ella.baron@shopify.com>

Commit:67f613c
Author:Kai Franz
Committer:Kai Franz

[#32169] YSQL: Online schema change part 8 - ALTER TABLE ... CONCURRENTLY Adds ALTER TABLE ... CONCURRENTLY, which runs a rewriting ALTER TABLE as an online schema change job instead of copying the table under AccessExclusiveLock. The statement clones the table into an empty shadow, runs the ALTER on the shadow so PostgreSQL works out the new schema and the per-column rewrite, turns that rewrite into the job's transform, and lets the job copy the table while DML continues. The cutover re-runs the ALTER on the source and swaps its storage with the shadow's, so the table keeps its OID and everything that refers to it. The lifecycle gets C entry points by job id (YbOscBegin, YbOscFinish, YbOscAbort) that the SQL functions and the statement share. Finish gains the swap cutover, and abort drops the shadow and transform of such a job. The job's own DDL is marked for the master, which leaves it out of DDL-transaction verification. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Commit:f870aca
Author:Kai Franz
Committer:Kai Franz

[#32169,#37751] YSQL: Online schema change part 7 - partitioned tables Extends online schema change to declaratively partitioned sources and targets, at any depth, so a job can also re-partition a table. A job is still one catalog row, one transform between the roots' row types, and one swap that renames only the roots. Begin sends the master the leaves of both hierarchies: the copy runs per source leaf, and delete-marker retention is kept on every target leaf. The capture triggers go on every table of the source hierarchy, since PostgreSQL does not clone statement triggers to partitions. The lifecycle functions lock every table, root first, and partition changes under an active job are refused, since a new partition would get neither a trigger nor a copy. Adds `yb_online_schema_change_clone_partitions` (OID 8140, migration V117) and the view's source_leaf_count / target_leaf_count. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Commit:c30a7f9
Author:Piyush Jain

Address review: reword the birth_time_pending_txn_id comment, test SERIALIZABLE too Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

Commit:c2ce635
Author:Ella Baron
Committer:GitHub

tserver: send a tablet's storage metadata and full compaction status to the master only when changed Both per-tablet sections were resent in every metrics heartbeat. The tserver now keeps what the master confirmed and sends an entry only when it differs; the master sets tablet_metadata_processed in the response once it applied every entry, so an entry that found no replica is resent. One flag, tserver_heartbeat_metrics_send_tablet_metadata_on_change, gates both.

Commit:e53304d
Author:Ella Baron
Committer:GitHub

master: hand tservers the tserver list as TSHeartbeat deltas Every TSHeartbeat response carried the full live tserver list; at 2000 tservers that was 451 MB/s out of the master leader and most of the TSHeartbeat handler time. The master now versions its list (leader term in the high 32 bits, a per-leader change counter in the low 32), each tserver echoes the version it holds, and the response carries only the tservers stamped after it plus tombstones for removals. A term change, an unknown version, one older than the tombstone horizon (master_tserver_list_history_sec, 600) or one ahead of the master's counter gets a full list.

Commit:3a1ab79
Author:Piyush Jain
Committer:Piyush Jain

[#37091] YSQL: Let a transaction read an index it created An index created in a DDL transaction is BIRTH_TIME_PENDING_DDL_COMMIT until that transaction commits, and the read guard answers reads of it with `LeaderNotReadyToServe`, which is retried until the RPC deadline. A statement that reads the index in the transaction that created it (index scan, foreign key check against a new UNIQUE constraint, `ON CONFLICT` on a new unique index, `yb_index_check()`, an extension script) therefore hung: the index only becomes final once that same transaction commits. The creating transaction wrote the index, so the index is complete for it. Let its reads through: - The master stores the creating transaction's id in the new `IndexInfoPB.birth_time_pending_txn_id` together with the pending state, and clears both when the index is finalized. The tablet does the same in `FinalizeIndexBirthTime`. - The read guard does not reject a read of a pending index whose transaction id matches. - pggate no longer sends the request to an index with `skip_intents`, so that it carries the transaction's metadata. With the new-relation fastpath, reads of a table created in the transaction, and until now of its indexes, are sent without it. The index read still happens at in_txn_limit. Restore the three regress reads that 4e76682e27 commented out. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

Commit:6143334
Author:Piyush Jain
Committer:Piyush Jain

[#30797] YSQL: Set index birth_time for indexes built nonconcurrently (#37103) A CREATE INDEX run inside a transaction block is built nonconcurrently and skips the backfill that sets the index birth_time, so birth_time stays 0. The docdb read guard then does not throw a kSnapshotTooOld error for a snapshot read whose read time precedes the index commit, and the index rows (written at the commit) are not visible to that read, so it silently returns an incomplete result. This is the data loss seen in the Jepsen si.append-table test. On a successful nonconcurrent index create, set the index birth_time to the master clock time (>= the DDL commit time, see below) and propagate it to the tablets via a new MarkNonConcurrentIndexDone tserver admin RPC. To reject reads until birth_time is final, introduce a `birth_time_state` enum in the index metadata (IndexInfoPB), set by the master at index creation: - BIRTH_TIME_FINAL (default): birth_time, if set, is final. - BIRTH_TIME_PENDING_BACKFILL: online (backfilled) index, until the backfill completes. - BIRTH_TIME_PENDING_DDL_COMMIT: nonconcurrent index created in a DDL transaction, until the transaction commits. The backfill path was relying on the `retain_delete_markers` flag as a proxy for this; post this change it can rely on `birth_time_state` too (both are checked). The read guard returns two different errors: - While the state is pending on the tablet (birth_time 0): `LeaderNotReadyToServe`. The tserver's client retries it until the RPC deadline, so the read waits until the tablet is finalized. - Once birth_time is set, for a read whose read time is before it: `SnapshotTooOld` (SQLSTATE 72000). The query layer does not retry it, so the application has to retry the transaction. Details: - Online indexes: the state is cleared together with setting birth_time when the backfill completes (master AllowCompactionsToGCDeleteMarkers, tablet BackfillDone). They never take the nonconcurrent path. - Nonconcurrent indexes: the DDL transaction verifier (HandleSuccessfulYsqlDdlTxn) sets birth_time, clears the state and posts AsyncMarkNonConcurrentIndexDone (new monitored task type) to each tablet. The tablet replicates a ChangeMetadata op with the new mark_non_concurrent_index_done field, which sets birth_time and clears the state. This is separate from BackfillDone / mark_backfill_done so that the two paths are not conflated, and each path only clears its own pending state. A finalize op for an index already removed from the tablet (e.g. a DROP INDEX on a colocated table racing with the RPC) is skipped instead of failing the replicated op; this also applies to BackfillDone. - birth_time >= commit time: the transaction coordinator picks the commit time, so the master's clock is moved past it before birth_time is picked. ReportYsqlDdlTxnStatus now carries the reporting tserver's hybrid time (which the commit response moved past the commit time), and when the master polls the transaction instead, its clock is updated from the GetTransactionStatus response. - Recovery of a lost finalize RPC: the master does not resend it (e.g. after a master failover or when a task gives up). The tablet validator also checks indexes whose local state is not final: a tablet is queued when it opens, and at the next periodic run (`tablet_validator_retain_delete_markers_validation_period_sec`, 60s by default) the validator asks the master (GetBackfillStatus now returns the master's birth_time_state) and finalizes the tablet metadata once the master reports the index final. Reads through the index retry until then. The validator asks only once per tablet open, so if that check ran while the index was still pending on the master, a later lost RPC is not repaired until the tablet is reopened, and not even then within an hour, since the validator reuses its cached "pending" answer for that long. The same gap already exists for BackfillDone; both are tracked in - xCluster targets: the birth time is not propagated yet (#33155). However, the target re-creates the index via DDL replication, so it will set the pending state and clear it on commit to allow reads (without stamping a local birth_time, so it stays 0 as before). - Known limitation: statements in the transaction that created a nonconcurrent index and read that index (index scans, `INSERT ... ON CONFLICT` on a unique index, foreign key checks against a UNIQUE constraint, `yb_index_check()`) are retried until statement_timeout, since the index only becomes final once that same transaction commits. This includes `CREATE TABLE ... UNIQUE` followed by an upsert in the same transaction. Tracked in #37091, to be fixed before a release containing this change. The three regress reads that do this (yb.orig.ddl_txn_{triggers,event_triggers,indices}) are commented out until then. Fixes #30797 - New AutoFlag enable_ysql_index_birth_time_state (kLocalPersisted). The master sets a non-final birth_time_state only when it is on; while it is off, behavior is unchanged (every index is final). It is an AutoFlag so that no index is left pending forever: an old master that becomes leader before the CREATE INDEX transaction is verified would not finalize the index, and after a rollback, indexes that were still pending would stay pending with nothing to finalize them on re-upgrade. - The new fields (IndexInfoPB.birth_time_state, IndexStatusPB.birth_time_state, ChangeMetadataRequestPB.mark_non_concurrent_index_done) and the new RPC are only used for indexes with a non-final state, which exist only once the AutoFlag is promoted, i.e. after all tservers run the new version. Older tservers never receive them. - A missing IndexInfoPB.birth_time_state means final, so indexes created before the upgrade keep the current behavior. An older binary that reads persisted metadata containing the field ignores it, and only loses the new read guard. A GetBackfillStatus response without IndexStatusPB.birth_time_state (from an older master) is treated as not final, so the tablet validator does not finalize anything from it. - New optional ReportYsqlDdlTxnStatusRequestPB.propagated_hybrid_time: an older master ignores it and an older tserver does not send it, which is the behavior before this change in both cases. - [x] Ran (release): ``` ./yb_build.sh release --cxx-test pgwrapper_pg_concurrent_ddls-test --gtest_filter 'PgConcurrentDDLsTest.SnapshotRead*:PgConcurrentDDLsTest.NonConcurrentIndex*:PgConcurrentDDLsTest.PendingIndexRejectsStaleRead:PgConcurrentDDLsTest.OnlineIndex*' ./yb_build.sh release --cxx-test pgwrapper_pg_index_backfill-test --gtest_filter 'PgIndexBackfillTest.RetainDeleteMarkers*:PgIndexBackfillColocated.ColocatedRetainDeleteMarkers*' ./yb_build.sh release --cxx-test master_multi_step_monitored_task-test ./yb_build.sh release --java-test 'org.yb.pgsql.TestDdlTransactionBlocks' ``` New tests in pg_concurrent_ddls-test: SnapshotReadFreshNonConcurrentIndexMatrix, PendingIndexRejectsStaleRead, OnlineIndexRejectsStaleReadBeforeBackfillDone, OnlineIndexBirthTimeFinalOnMaster, NonConcurrentIndexWithDropColumnInSameTxn, NonConcurrentIndexFinalizeAfterDropIndex.

Commit:a63729a
Author:Eric Sheng
Committer:Yamen Haddad

[BACKPORT 2026.1][#26950] docdb: Add support for proto3 optional fields Summary: Original commit: 0426fb90083 / D43514 Protobuf 3.12 added support for optional fields when using proto3. However, our code generators for lightweight protobufs do not support it properly, since proto3 optional fields are treated like: ``` oneof _foo { int32_t foo = 1; [proto3_optional=true] } ``` instead of like proto2 optional fields. Additionally, even the protoc-gen-insertions generator which just adds an include at the top does not work since it does not signal that it supports proto3 optional. The changes needed to support proto3 optional fields, which this change implements, are: - Add `GetSupportedFeatures()` on `CodeGenerator`s returning `FEATURE_PROTO3_OPTIONAL` to indicate that our generators support proto3 optionals. - Change `containing_oneof()` to `real_containing_oneof()` to skip over the synthetic `oneof`s generated by proto3 `optional`. Our lightweight protobuf generator already ignores the proto2/proto3 distinction and generates all normal fields like proto2 `optional` by default, so by skipping the synthetic `oneof`s, proto3 `optional` fields are now correctly treated like proto2 `optional` fields for generation. **Upgrade/Rollback Safety** N/A, only modifies pg_client.proto used for tserver/pg communication. Test Plan: Jenkins. Also checked that the generated code for `colocation_id` in `pg_client.messages.*` matches other fields. Reviewers: esheng Reviewed By: esheng Subscribers: yql, ybase Differential Revision: https://phorge.dev.yugabyte.com/D60168

Commit:9b5fa80
Author:druzac
Committer:GitHub

[BACKPORT 2025.2][#34504] DocDB: Write the reverse mapping entries of rows a colocated table had before it was added (#35489) (#38454) ## Summary In xCluster automatic DDL mode, the rows of a new colocated table reach the target's colocated tablet before the DDL queue creates the table there. With no schema packing for the colocation id, `VectorIndexesUpdater` writes none of the vector reverse mapping entries the table owns. Search and vector index compaction treat a vector without an entry as deleted, so a ybhnsw index built on the target finds none of those rows, and searches widen until the statement timeout. The reverse mapping is owned by the table, so the table repairs it, not the vector index. When the change metadata operation adds a new colocated table that owns the reverse mapping, has a vector column (in its schema or an old packing), and whose colocation id already has entries on the tablet: - The table is added with a mark (`TableInfoPB.vector_reverse_mapping_repair_pending`), set together with the table so a superblock flush cannot persist one without the other, and repaired in the background on the vector index backfill pool. The apply itself only does one seek, so the colocated tablet is not blocked for the length of the scan. - The repair scans the raw entries of the colocation, every version after the colocation's table tombstone, and feeds each through `VectorIndexesUpdater` at its own write time, as the apply path would have, so later tombstones of replaced or deleted vectors still win. - It then flushes the regular DB and only then clears the mark. A restart before that starts the repair over, rewriting the same entries. - Vector index backfills of a marked table wait for the repair: vectors backfilled without their entries would be dropped by vector index compaction. Backfills in flight are now tracked per index, so the retry that launches the held back backfills does not start a second backfill of an index already backfilling. This replaces the first version of this PR, which repaired the mapping from the vector index backfill, at the backfill time; the index should not own the table's mapping, and entries at the backfill time could hide later tombstones (#35495). Other alternatives considered: registering the upcoming table's packing on the target tablet before its rows arrive, generating the entries in the xCluster consumer, and repairing inside the change metadata apply (blocks the whole colocated tablet for the scan). Indexes already built on a target before this change are not repaired; they need to be rebuilt. Known gaps: - A snapshot or PITR restore of a snapshot taken during a repair, after the repair finished, brings back the regular DB without the entries while the mark stays cleared. - Rows whose schema version has no packing on the tablet when the repair runs are skipped without a trace, as on the apply path. - The trigger is "entries under the colocation id when the table is added", so bootstrap replay of the ADD after a crash can repair a table that did not need it, at the cost of a scan. Master could flag automatic mode xCluster creates in the AddTableToTablet request instead. - Truncate and storage replacement during a repair are untested. Fixes #34504. ## Upgrade/Rollback safety Adds `TableInfoPB.vector_reverse_mapping_repair_pending` (tablet superblock). No WAL or RPC format change. - Mixed versions: replicas on an old binary do not repair, as today; replicas on the new binary do. Replica contents can differ in reverse mapping entries until all replicas are upgraded and the table's next repair, which only happens when a table is added. - Rollback with a repair pending: the old binary ignores the field and drops it on its next superblock write, and backfills the table's vector indexes without the missing entries, which is today's behavior. Original commit: ca66d14519429821655a7642f7ab949c4dde33d3 / #35489 ## Test plan - [x] `xcluster_ddl_replication-test --gtest_filter=XClusterDDLReplicationColocatedVectorTest.*` (new): rows before the target's CREATE TABLE; with an ALTER before it; with the vector column added by an ALTER before it; rows updated and deleted before it; two tables; 20,000 rows; the index created while the repair is held; the target's tservers restarted while the repair is held; no repair for a table created before its rows - [x] `xcluster_ddl_replication-test` colocated and vector index tests: `*CreateColocated*`, `*ColocatedHistoricalSchemasWithCompactions`, `*AlterExistingColocatedTable`, `*ColocatedFailoverWithPendingCreate`, `*MatViewWithColocation`, `*ColocatedBatchRetryPreservesMetadata`, `*VectorIndex*` 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> ## Merge conflicts ### 2026.1 --> 2025.2 - `src/yb/tablet/tablet.cc:35` — [adjacent-lines] took `<algorithm>`, `<deque>` and `<tuple>`; 2025.2's file has no `<tuple>` include there. - `src/yb/tablet/tablet_metadata.cc:988` — [adjacent-lines] added `HasTable` and kept 2025.2's `GetTableInfoUnlocked(const TableId&)` signature (2026.1 takes `TableIdView`). ### master --> 2026.1 - `src/yb/tablet/tablet.cc:35` — [adjacent-lines] added `<algorithm>` and `<deque>` next to 2026.1's includes. - `src/yb/tablet/tablet.cc:84` — [adjacent-lines] added the `yb/dockv/key_bytes.h` and `yb/dockv/value.h` includes. - `src/yb/tablet/tablet.cc:3278` — [missing-prerequisite] dropped `!table_properties.writes_vector_reverse_mapping()` from `NeedsVectorReverseMappingRepair`. That property comes from #33912 (vector payload mode), which 2026.1 and 2025.2 lack; there, every table that owns the reverse mapping writes it. - `src/yb/tablet/tablet.cc:3375` — [missing-prerequisite] the repair scan checks `abort_pending_op_status_holder_.GetStatus()` in place of master's `abort_pending_op_source_.AbortStatus()` from #32907. It is the same signal: set while pending operations are drained for a shutdown, truncate or restore, and reset after. - `src/yb/tablet/tablet_vector_indexes.cc:792`, `:813` — [missing-prerequisite] `MoveStatus(*read_op)` in place of `read_op->CreateStatus()` (#32907); both return ShutdownInProgress when stopped and TryAgain otherwise. - `src/yb/tablet/tablet_metadata.cc:2637` — [missing-prerequisite] added `SetVectorReverseMappingRepairPending` and kept 2026.1's `OnBackfillDone(table_id)`, which has no `birth_time` parameter. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

Commit:ebc9681
Author:druzac
Committer:GitHub

[BACKPORT 2026.1][#34504] DocDB: Write the reverse mapping entries of rows a colocated table had before it was added (#35489) (#38450) ## Summary In xCluster automatic DDL mode, the rows of a new colocated table reach the target's colocated tablet before the DDL queue creates the table there. With no schema packing for the colocation id, `VectorIndexesUpdater` writes none of the vector reverse mapping entries the table owns. Search and vector index compaction treat a vector without an entry as deleted, so a ybhnsw index built on the target finds none of those rows, and searches widen until the statement timeout. The reverse mapping is owned by the table, so the table repairs it, not the vector index. When the change metadata operation adds a new colocated table that owns the reverse mapping, has a vector column (in its schema or an old packing), and whose colocation id already has entries on the tablet: - The table is added with a mark (`TableInfoPB.vector_reverse_mapping_repair_pending`), set together with the table so a superblock flush cannot persist one without the other, and repaired in the background on the vector index backfill pool. The apply itself only does one seek, so the colocated tablet is not blocked for the length of the scan. - The repair scans the raw entries of the colocation, every version after the colocation's table tombstone, and feeds each through `VectorIndexesUpdater` at its own write time, as the apply path would have, so later tombstones of replaced or deleted vectors still win. - It then flushes the regular DB and only then clears the mark. A restart before that starts the repair over, rewriting the same entries. - Vector index backfills of a marked table wait for the repair: vectors backfilled without their entries would be dropped by vector index compaction. Backfills in flight are now tracked per index, so the retry that launches the held back backfills does not start a second backfill of an index already backfilling. This replaces the first version of this PR, which repaired the mapping from the vector index backfill, at the backfill time; the index should not own the table's mapping, and entries at the backfill time could hide later tombstones (#35495). Other alternatives considered: registering the upcoming table's packing on the target tablet before its rows arrive, generating the entries in the xCluster consumer, and repairing inside the change metadata apply (blocks the whole colocated tablet for the scan). Indexes already built on a target before this change are not repaired; they need to be rebuilt. Known gaps: - A snapshot or PITR restore of a snapshot taken during a repair, after the repair finished, brings back the regular DB without the entries while the mark stays cleared. - Rows whose schema version has no packing on the tablet when the repair runs are skipped without a trace, as on the apply path. - The trigger is "entries under the colocation id when the table is added", so bootstrap replay of the ADD after a crash can repair a table that did not need it, at the cost of a scan. Master could flag automatic mode xCluster creates in the AddTableToTablet request instead. - Truncate and storage replacement during a repair are untested. Fixes #34504. ## Upgrade/Rollback safety Adds `TableInfoPB.vector_reverse_mapping_repair_pending` (tablet superblock). No WAL or RPC format change. - Mixed versions: replicas on an old binary do not repair, as today; replicas on the new binary do. Replica contents can differ in reverse mapping entries until all replicas are upgraded and the table's next repair, which only happens when a table is added. - Rollback with a repair pending: the old binary ignores the field and drops it on its next superblock write, and backfills the table's vector indexes without the missing entries, which is today's behavior. Original commit: ca66d14519429821655a7642f7ab949c4dde33d3 / #35489 ## Test plan - [x] `xcluster_ddl_replication-test --gtest_filter=XClusterDDLReplicationColocatedVectorTest.*` (new): rows before the target's CREATE TABLE; with an ALTER before it; with the vector column added by an ALTER before it; rows updated and deleted before it; two tables; 20,000 rows; the index created while the repair is held; the target's tservers restarted while the repair is held; no repair for a table created before its rows - [x] `xcluster_ddl_replication-test` colocated and vector index tests: `*CreateColocated*`, `*ColocatedHistoricalSchemasWithCompactions`, `*AlterExistingColocatedTable`, `*ColocatedFailoverWithPendingCreate`, `*MatViewWithColocation`, `*ColocatedBatchRetryPreservesMetadata`, `*VectorIndex*` 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> ## Merge conflicts ### master --> 2026.1 - `src/yb/tablet/tablet.cc:35` — [adjacent-lines] added `<algorithm>` and `<deque>` next to 2026.1's includes. - `src/yb/tablet/tablet.cc:84` — [adjacent-lines] added the `yb/dockv/key_bytes.h` and `yb/dockv/value.h` includes. - `src/yb/tablet/tablet.cc:3278` — [missing-prerequisite] dropped `!table_properties.writes_vector_reverse_mapping()` from `NeedsVectorReverseMappingRepair`. That property comes from #33912 (vector payload mode), which 2026.1 lacks; there, every table that owns the reverse mapping writes it. - `src/yb/tablet/tablet.cc:3375` — [missing-prerequisite] the repair scan checks `abort_pending_op_status_holder_.GetStatus()` in place of master's `abort_pending_op_source_.AbortStatus()` from #32907. It is the same signal: set while pending operations are drained for a shutdown, truncate or restore, and reset after. - `src/yb/tablet/tablet_vector_indexes.cc:792`, `:813` — [missing-prerequisite] `MoveStatus(*read_op)` in place of `read_op->CreateStatus()` (#32907); both return ShutdownInProgress when stopped and TryAgain otherwise. - `src/yb/tablet/tablet_metadata.cc:2637` — [missing-prerequisite] added `SetVectorReverseMappingRepairPending` and kept 2026.1's `OnBackfillDone(table_id)`, which has no `birth_time` parameter. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

Commit:7cda819
Author:Bvsk Patnaik
Committer:GitHub

[#37328] YSQL: Detect deadlocks of fast path writes without global object locks (#37359) ## Summary ### Background The tserver tags a fast path write (a write outside a distributed transaction) with its object lock owner, so that the deadlock detector sees a cycle that runs through object locks and row locks (#24630). Two kinds of transaction take local object locks for their DMLs but no global ones, so that DDLs with global object locks wait for them: - one that latched `ysql_yb_enable_global_object_locking_infra` off, during its promotion (#37358, #36572); - any transaction on a node with `disable_global_object_locks` on (#37334, #37355). `using_table_locks_globally` stays false for both, so tablets keep checking their catalog version. ### Problem The tserver tags fast path writes only when `using_table_locks_globally` is set, so a fast path write from a transaction without global object locks carries no lock owner. If that writer waits on a row lock while a DDL waits on the writer's local object lock, the deadlock detector does not see the cycle. Both wait until a statement times out. Example, during the `ysql_yb_enable_global_object_locking_infra` promotion: 1. ts-2, after the promotion: `BEGIN; UPDATE test SET v = v + 1 WHERE k = 1;` holds the row lock on k = 1. 2. ts-1, transaction latched off: `UPDATE test SET v = v + 1 WHERE k = 1;` takes a local lock on `test`, then waits on the row lock. 3. ts-2: `ALTER TABLE test ADD COLUMN v1 INT;` waits on ts-1's local lock. ### Root cause `PrepareOperations` gates the tag on `using_table_locks_globally`. Before #37358, only a transaction with global object locks took object locks at all, so "takes object locks" and "tablets skip the catalog version check" were the same fact. #37358 separates them: a transaction latched off takes local locks, but its tablets must still check its catalog version, and #37334 adds a node-wide way to run that way. The gate kept the second meaning. ### Impact A cycle between a fast path write from a transaction without global object locks and a DDL with global object locks: while `ysql_yb_enable_global_object_locking_infra` is being promoted, and for as long as any node runs with `disable_global_object_locks` on. The two sessions wait until a statement times out, instead of the detector aborting one of them. ### Fix Add `PgPerformOptionsPB.using_table_locks_locally`. pggate sets it from `UsingObjectLocksLocally()`, the predicate that decides whether the transaction sends its locks to the tserver. `PrepareOperations` tags a fast path write when it is set, instead of when `using_table_locks_globally` is set. `DoAcquireObjectLock` checks the field. Before #37358 it checked `is_using_table_locks`; #37358 removed that check, since a transaction without global locks sends local ones. An object lock request from a transaction that does not take object locks means a pggate sender skipped `UsingObjectLocksLocally()`. In release builds `RSTATUS_DCHECK` returns `IllegalState`. ### Why the fix works `using_table_locks_locally` and the locks a transaction sends come from the same predicate, so a fast path write carries an owner exactly when its transaction can hold an object lock. The tablet then waits on the row lock as that owner (`ShouldWaitAsDifferentTxn` in `conflict_resolution.cc`), and the detector joins that wait with the DDL's wait on the same owner's object lock. A transaction with global object locks also uses object locks locally, so its writes stay tagged as before. Writes in a distributed transaction are not affected: the gate skips them, because the transaction itself owns the locks. ## Upgrade/Rollback safety The field is new (tag 42) and defaults to false. pggate sends it only to the tserver of its own node, which runs the same binary, because the tserver starts Postgres. No mixed-version pair sees it. ## Test plan `PgObjectLocksTestLegacyTxnVsNewDdl.LegacyFastPathWriteDeadlockIsDetected` (both parameter cases): a fast path UPDATE from a transaction latched off waits on a row lock held by a transaction latched on, which then runs ALTER TABLE. Exactly one of the two writers commits. With the gate reverted to `using_table_locks_globally`, both cases fail: the ALTER hits its statement timeout. `PgObjectLocksTestGlobalLocksRollingRestart.FastPathWriteDeadlockIsDetected`: the same cycle with the fast path writer on a node restarted with `disable_global_object_locks` on (#37355), which keeps its local locks. With the gate reverted, it fails the same way. The object lock request check: every object lock request goes through it, so every test above exercises it; `PgObjectLocksTestRF1.TestSanity` and `PgObjectLocksTestRF1.TestDeadlockFastPath` also pass (release, with the three tests above, on the current code). With the setter forced to false, `PgObjectLocksTestRF1.TestSanity` fails at this check, which returns "Object lock request from a transaction that does not use object locks locally". ``` ./yb_build.sh release --cxx-test pgwrapper_pg_object_locks-test \ --gtest_filter '*FastPathWriteDeadlockIsDetected*:PgObjectLocksTestRF1.TestSanity:PgObjectLocksTestRF1.TestDeadlockFastPath' ``` Part of stack #37903, bottom to top: #37358, #37334, #37359. Fixes #37328 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

Commit:6cc927a
Author:Sergei Politov
Committer:GitHub

[BACKPORT 2026.2][#36288] DocDB: Keep the vectors of rows whose DELETE or UPDATE is rolled back to a savepoint (#38535) ## Summary A DELETE or UPDATE of a row with a vector puts the ids of the vectors it removes into a delete list record of its batch, and the transaction apply tombstones them in the vector reverse mapping. The record had no subtransaction id, so it was applied even when its subtransaction was rolled back, and the index lost the vector of a live row for good. Besides `ROLLBACK TO SAVEPOINT`, READ COMMITTED statement retries, plpgsql `EXCEPTION` blocks and client autosave roll back subtransactions too. The record now starts with a new value type for vector index records, `kVectorIndexRecord`, followed by the record type `VectorIndexRecordType::kDeleteList` and the subtransaction id of its batch, and the apply skips the records of rolled back subtransactions. A later record format takes a new record type rather than a new value type. The readers that only skip these records (transaction loading, lock status and CDC) recognize both the old and the new record through `dockv::IsVectorIndexRecord`. Also moves the subtransaction id encoding of intent values to `EncodeSubTransactionId`, shared by the intents and the external intents writers, which writes the same bytes. ## Upgrade/Rollback safety An older binary does not know the new record: its apply looks the record up as an intent key, does not find it and skips it, so the vectors of the rows deleted by the transaction stay in the index, and its transaction loader logs a decode failure (both `DFATAL`). So the leader writes the new record only after the new AutoFlag `vector_index_delete_list_with_subtransaction` (`kLocalPersisted`) is promoted, and replicates its choice in the write batch (`delete_vector_ids_with_subtransaction`, a new field of `KeyValueWriteBatchPB`), so all replicas and WAL replay write the same record. Promoting the AutoFlag is the point of no return: after it, an older binary leaves the vectors of the rows deleted by transactions that hold the new record. Until the promotion, rolled back subtransactions still lose vectors. `yb-ts-cli verify_vector_indexes` detects the vectors lost before it, and rebuilding the index or rewriting the affected rows restores them. Original commit: 4c1bd14ecceede59962480a173cd36116a8cb6c7 / #37985 ## Test plan - [x] `./yb_build.sh debug --cxx-test pgwrapper_pg_vector_index-test --gtest_filter PgVectorIndexUtilTest.RolledBackSubtransactionKeepsVectors`: without the fix rows 1, 3, 20 and 1000 are missing from the index search results. - [x] `LoadTransactionWithDeleteList`, `DeleteListsWithBothEncodings`, `SameTxnReplacedVector` and `SameTxnDeletedVector` of `PgVectorIndexUtilTest` pass. - [ ] Jenkins.

Commit:db3ef76
Author:Saurav Tiwary
Committer:GitHub

[#36774] YSQL: Send aborted subtxns of transactions tied at commit time to the walsender (#37446) ## Summary With transactional DDL, the walsender reads the catalog with the intents of each record's DocDB transaction, skipping the subtransactions rolled back to a savepoint (#36204). The producer sends that aborted subtransaction set on the BEGIN record. The virtual WAL streams DocDB transactions that commit at the same hybrid time as one pg transaction, and ships only the BEGIN of the transaction with the lowest id. The sets of the other transactions never reached the walsender: debug builds hit an assert, and release builds used the set of the other transaction, so catalog reads could see a DDL rolled back to a savepoint (an added column, a table added to the publication). Changes: - The virtual WAL moves the set of every BEGIN it reads, shipped or merged at a commit time tie, into the new `GetConsistentChangesResponsePB.aborted_subtxns` map, keyed by DocDB transaction id with the commit time. Records no longer carry the set, so the walsender gets every set from one place. - The walsender adds the map of each response to its list and drops the entries at or before a commit time once yb_decode has decoded that COMMIT. It drops by commit time because the COMMIT of a merged block carries the id of only one of its transactions. - The per-record set that #36204 added to `YbcPgRowMessage` is removed. - New test flags `TEST_commit_time_override_ht` and `TEST_commit_time_override_txn_ids` make the given transactions commit at the same hybrid time. ## Upgrade/Rollback safety Adds the map field `aborted_subtxns` (8) to `GetConsistentChangesResponsePB`, and moves the set out of `RowMessage.aborted_subtxns` on the records the virtual WAL returns. The response only goes from a tserver to its own postgres (walsender), which run the same version. The code paths are behind `TEST_ysql_yb_enable_replication_slot_transactional_ddl`. ## Test plan New tests force a commit time tie with the new test flags: - `TestPgReplicationSlotWithTxnDdl#commitTimeTieWithAddColumnRolledBackToSavepoint`: an ADD COLUMN rolled back to a savepoint in the transaction whose BEGIN is merged. Fails on master with the walsender assert. - `TestPgReplicationSlotWithTxnDdl#commitTimeTieWithAlterPublicationRolledBackToSavepoint`: the same with ALTER PUBLICATION ADD TABLE, where the merged transaction's first record triggers a publication refresh. Without its set, the rolled-back table is added to the slot's table list. - `CDCSDKConsumptionConsistentChangesTest.TestVWALShipsAbortedSubtxnsOfTransactionalDdl` checks that the set comes only in the response map. Ran: - [x] `./yb_build.sh debug --java-test org.yb.pgsql.TestPgReplicationSlotWithTxnDdl` - [x] `./yb_build.sh debug --cxx-test integration-tests_cdcsdk_consumption_consistent_changes-test --gtest_filter '*TestCommitTimeTie*:*TestVWALShipsAbortedSubtxnsOfTransactionalDdl*'` 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

Commit:5d46195
Author:Bvsk Patnaik
Committer:Bvsk Patnaik

[#37328] YSQL: Tag legacy-mode fast path writes with their object lock owner Replays #37359's commits (074ed6508a4..eb4ffe3042a) as one commit on #37358's renamed request field (using_table_locks_globally) and infra flag (ysql_yb_enable_global_object_locking_infra): the fast path owner tag, PgPerformOptionsPB field 42 is_using_local_table_locks and its setter, the check that object lock requests come from a lock-taking transaction, and FastPathWriteDeadlockIsDetected. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Commit:70c55c1
Author:Bvsk Patnaik
Committer:Bvsk Patnaik

[#37328] YSQL: Say what the object lock request check catches The check in DoAcquireObjectLock fires when a lock request comes from a transaction that does not use object locks locally, which means a pggate sender skipped UsingObjectLocksLocally(). Its message, copied from master, said "Table Locking feature not enabled.", which points at the gflags; in release builds that text reaches the client. Say what the check found instead. State the field's contract in its proto comment: pggate sets it from the same predicate that decides whether the transaction sends its locks. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Commit:95fe5b3
Author:Bvsk Patnaik
Committer:Bvsk Patnaik

[#37328] YSQL: Shorten the using_table_locks_locally comment and scale the test timeouts - pg_client.proto: the field comment points to PgSession::AcquireObjectLock for why a transaction without global object locks takes local locks, instead of repeating it with one of its two issues. - IsTxnUsingLocalTableLocks becomes IsTxnUsingTableLocksLocally, to match the field. - CheckFastPathWriteDeadlockIsDetected scales its two statement timeouts by kTimeMultiplier. The ALTER's timeout decides the test, so a slow build must not read as a missed cycle. - The rolling restart test moves next to the other tests of its fixture. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

Commit:c815f11
Author:Bvsk Patnaik
Committer:Bvsk Patnaik

[#37328] YSQL: Rename is_using_local_table_locks to using_table_locks_locally Match its neighbor using_table_locks_globally: the pair now reads as the same question at two scopes, set from UsingObjectLocksLocally() and UsingObjectLocksGloballyForCurrentTxn(). The tag stays 42, so the wire format does not change, and the field is new in this PR. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Commit:57019a2
Author:Deepti Kumar
Committer:GitHub

[BACKPORT 2026.1][PLAT-22592,PLAT-22994] YBA: Keep telemetry exporter credentials out of collector config and logs (#37023) (#38557) ## Summary Telemetry exporter credentials leaked in two places: - PLAT-22592: the collector config held them in cleartext. On K8s this config reaches ConfigMaps, Helm values and support bundles. - PLAT-22994: on VM, `NodeManager` put the AWS keys on the `ybcloud` command line, which YBA logs. Fix: - The collector config refers to each credential as `${env:YB_OTEL_<provider-uuid>_<field>}`. - K8s: the values go into the existing `<release>-otel-secrets` Secret through `otelCollector.secretEnv`. No chart change. - VM: a new proto map, `otelColSecretEnv`, sends the values to node-agent. Node-agent writes them to `secrets.env` (mode 0600), and the unit loads that file. The AWS credentials block is written the same way, not with the keys on a command line. - `NodeManager` no longer puts credentials on the command line. The logged command also masks them. - YBA removes line breaks from single-line credentials. Node-agent rejects them as a second check. The first commit message says the files are written with `O_NOFOLLOW`. A later commit writes them as the task user instead. Not in this change: - After a YBA rollback, `secrets.env` stays on VM nodes. - On K8s, two AWS or GCP providers on one universe still share fixed env names. ## Upgrade/Rollback safety - `yb.proto` adds field 15 to `InstallOtelCollectorInput`. It is additive. - If node-agent is not upgraded yet, the task fails before it uploads any collector file. Retry after the upgrade. - Existing VM nodes keep the old config until the next `ManageOtelCollector` run. - Rollback: the old node-agent writes the old inline config again, so the collector keeps working. Original commit: 1cd6e138d97b312a86c38723b528d3c8263c8728 / #37023 ## Test plan - [x] `OtelCollectorConfigGeneratorTest`: VM and K8s configs hold no credential, and every `${env:X}` resolves from the env that is sent, also on older charts and with a paused audit exporter. Line breaks are removed. - [x] `NodeAgentRpcPayloadTest`, `ShellProcessContextTest`, `LokiConfigTest`, `RedactingServiceTest`, `TelemetryProviderServiceTest`, `TelemetryProviderControllerTest`, `ShellProcessHandlerTest`, `SupportBundleUtilTest`, `NodeManagerTest`. - [x] node-agent `app/task` tests: file quoting and modes, line breaks rejected, managed block removal keeps files at 0600. - [x] Live GCP VM universe, user and root node-agent, with Datadog, Loki, Dynatrace and CloudWatch export: export works, and no credential is in `config.yml`, the unit, or the YBA and node-agent logs. - [ ] Live K8s universe.

Commit:56567db
Author:Kai Franz
Committer:Kai Franz

[#32169] DocDB: Online schema change part 4 - abort and status RPCs, retention release Adds the master RPCs that the SQL interface (part 5) uses to stop a copy (`AbortTableBackfill`) and to read its state and progress (`GetTableBackfillStatus`), with their client, tserver and pggate plumbing. The master now releases the target's delete-marker retention before it records a copy as done, and records the copy's read time as the birth time of the target's indexes, as index backfill does. Before, retention lasted about five minutes past the copy and index reads on the target were retried until then. `StartTableBackfill` names the target by DocDB table id instead of pg_class OID, so a table rewritten by TRUNCATE or ALTER COLUMN TYPE is found. Fixes two bugs in part 1's controller that the first real copies in debug builds hit: a chunk task built for a tablet whose chunk was still in flight was destroyed unrun (fatal in debug builds), and a chunk retry waited for the Raft lock on a reactor thread. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Commit:616f531
Author:Deepti Kumar
Committer:GitHub

[BACKPORT 2026.2][PLAT-22592,PLAT-22994] YBA: Keep telemetry exporter credentials out of collector config and logs (#37023) (#38558) ## Summary Telemetry exporter credentials leaked in two places: - PLAT-22592: the collector config held them in cleartext. On K8s this config reaches ConfigMaps, Helm values and support bundles. - PLAT-22994: on VM, `NodeManager` put the AWS keys on the `ybcloud` command line, which YBA logs. Fix: - The collector config refers to each credential as `${env:YB_OTEL_<provider-uuid>_<field>}`. - K8s: the values go into the existing `<release>-otel-secrets` Secret through `otelCollector.secretEnv`. No chart change. - VM: a new proto map, `otelColSecretEnv`, sends the values to node-agent. Node-agent writes them to `secrets.env` (mode 0600), and the unit loads that file. The AWS credentials block is written the same way, not with the keys on a command line. - `NodeManager` no longer puts credentials on the command line. The logged command also masks them. - YBA removes line breaks from single-line credentials. Node-agent rejects them as a second check. The first commit message says the files are written with `O_NOFOLLOW`. A later commit writes them as the task user instead. Not in this change: - After a YBA rollback, `secrets.env` stays on VM nodes. - On K8s, two AWS or GCP providers on one universe still share fixed env names. ## Upgrade/Rollback safety - `yb.proto` adds field 15 to `InstallOtelCollectorInput`. It is additive. - If node-agent is not upgraded yet, the task fails before it uploads any collector file. Retry after the upgrade. - Existing VM nodes keep the old config until the next `ManageOtelCollector` run. - Rollback: the old node-agent writes the old inline config again, so the collector keeps working. Original commit: 1cd6e138d97b312a86c38723b528d3c8263c8728 / #37023 ## Test plan - [x] `OtelCollectorConfigGeneratorTest`: VM and K8s configs hold no credential, and every `${env:X}` resolves from the env that is sent, also on older charts and with a paused audit exporter. Line breaks are removed. - [x] `NodeAgentRpcPayloadTest`, `ShellProcessContextTest`, `LokiConfigTest`, `RedactingServiceTest`, `TelemetryProviderServiceTest`, `TelemetryProviderControllerTest`, `ShellProcessHandlerTest`, `SupportBundleUtilTest`, `NodeManagerTest`. - [x] node-agent `app/task` tests: file quoting and modes, line breaks rejected, managed block removal keeps files at 0600. - [x] Live GCP VM universe, user and root node-agent, with Datadog, Loki, Dynatrace and CloudWatch export: export works, and no credential is in `config.yml`, the unit, or the YBA and node-agent logs. - [ ] Live K8s universe.

Commit:e452446
Author:IshanChhangani
Committer:GitHub

[BACKPORT 2025.2][#16670] DocDB: dist-trace: (yb.rpc.trace) field option and tags for tablet, table and object-lock request fields (#38401) Backport stack for 2025.2, part 1 of 4, in master order: #34146 → #34156 → #34194 → #35037. ## Summary Adds a `(yb.rpc.trace)` field option to `lightweight_message.proto` and tags the first production fields with it. Tagging a request field with `[(yb.rpc.trace) = {enabled: true}]` makes gen_yrpc emit `TracingAttributes()` for the message, and the generated proxy stub drains it onto the outbound RPC's client span, gated on an active trace context so untraced requests build nothing. Nested and repeated messages are followed, keys are `req.<field>[.<sub>]`, enums by name, `bytes` hex-encoded unless `bytes_as_string`. A repeated field yields one key with `*` in place of the index (`req.ops.*.write.table_id`) whose value lists the distinct values sorted, so the attribute count does not grow with the batch. The shared-memory exchange never goes through a proxy stub, so `PgClientData::StartSharedMemorySpan` drains the same attributes by hand before starting the simulated client span. Tagged: tablet and table ids across tserver, master, consensus, CDC and backup RPCs, and the object-lock oids and lock mode on `PgClientService.AcquireObjectLock`. The pb_ stub can only call `TracingAttributes(req)` from a generated `.messages.cc`, so `master_client`, `master_admin`, `tserver_admin` and `cdc_service` move to `MESSAGES_PROTO_FILES`; without that a tag compiles and emits nothing. Perform, tablet Read and tablet Write spans also get `rpc.table_names`, one `name(id)` entry per distinct table the request touches including the index behind `index_request`, so the raw id attributes can be read without a catalog lookup. Design: https://docs.google.com/document/d/1nwb-V1dIa-8mLM5dvZwsDgoQ1ODLUA9TUFFqjXUeplo/edit?tab=t.0#heading=h.k8r70zn36j4j ## Upgrade/Rollback safety Descriptor-only change: a new `FieldOptions` extension and options on existing fields. Options are never serialized, no field numbers or message shapes change. Generating messages files for four more protos adds lightweight C++ types but changes nothing on the wire. Mixed-version clusters and rollback are unaffected. Original commit: ad0e95cfc93cbc82cd00c8ba8968367a46e57e03 / #34146 ## Test plan - [x] `LWProtoTest.TracingAttributes` tags fields in `rtest.proto` covering string, hex bytes, bytes_as_string, uint32, enum, bool, repeated scalars, nested, repeated nested, pointer and self-referential messages, and checks the lightweight and pb_ forms against one expected list. `./yb_build.sh release --cxx-test lwproto-test` passes. - [x] `dist_trace-test` asserts the attributes end to end on both paths: `req.ops.*.read.table_id` and `req.ops.*.write.table_id` on shared-memory Perform spans, `req.lock_oid.*` and `req.lock_type` on the shared-memory object-lock span, `req.ops.*.read.table_id` on the RPC Perform span plus `req.tablet_id` and `rpc.table_names` on the tserver's Read and Write spans. `./yb_build.sh release --cxx-test dist_trace-test` passes. ## Merge conflicts 2025.2 lacks the lightweight-protobuf refactors this commit builds on on master (#29295 `c5ce9f142d6`, #29705 `69e7f2a1ae2`, #29787 `9f0f237e1df`) and #29846's `split_factor`. The changes below adapt #34146 to 2025.2; trace attributes match master. - `src/yb/tserver/CMakeLists.txt` — added `MESSAGES TRUE` to `tserver_service_proto`, as on master (#29705). Without it the `TabletServerService` pb_ stubs have no `TracingAttributes(req)` to call, so `req.tablet_id` and the other tags never reach the span. - `src/yb/gen_yrpc/proxy_generator.cc` — 2025.2 has no `DeclareMethod`/`DefineMethod` and no pb_/lw_ stub split. Inlined the trace drain into the single stub per method: `kTraceAttrs` for a lightweight method, `kPbTraceAttrs` for a pb_ method when the file generates messages. - `src/yb/tserver/pg_client_session.cc` — kept 2025.2's `PrepareOperations(PgPerformRequestPB* req, ...)` (no arena, no `LWPgPerformRequestPB`); added `PgTableCache& table_cache`, the index-table lookup and `set_index_table` as on master. Index ids pushed as `std::string` (no `.ToBuffer()`). - `src/yb/tserver/pg_table_cache.{h,cc}` — `TableIdView` → `const TableId&` (no `TableIdView` on 2025.2). - `src/yb/yql/pggate/pg_client.cc` — kept the `&PgClientServiceProxy::AcquireObjectLockAsync` member pointer; 2025.2's `PrepareAndSend` invokes `proxy->*method`. Hand-written `rpc.object_lock.*` attributes removed as on master; 2025.2 never had `rpc.object_lock.tablespace_oid`. - `src/yb/client/async_rpc.cc` — added only `PublishPendingRpcTableNames`; master's `resp_.Clear()` and `tserver_service.messages.h` include belong to the lightweight refactor. - `src/yb/tserver/tserver_service.proto` — added `import "yb/rpc/lightweight_message.proto"` (master gained it in #29787); without it protoc rejects `(yb.rpc.trace)`. - `src/yb/tserver/pg_client.proto` — kept 2025.2's `uint64` `ObjectLockOid` fields, added the tags (same as the 2026.1 backport). - `src/yb/master/master_admin.proto`, `src/yb/tserver/tserver_service.proto` — tags added; `split_factor` absent on 2025.2. `dist_trace-test.cc` is identical to master. ## 2025.2 test run Release build on 2025.2: `lwproto-test` passes; `dist_trace-test` `DistTraceTest.*` and `DistTraceRpcTest.*` pass 51/52, including `req.tablet_id` on `TabletServerService.Read`. `DistTraceRpcTest.TestExecuteSpanCoversPortalStart` fails 3/3 on plain 2025.2 (`df1cf912ac7`) without this commit too, so it predates this backport. <!-- Reviewable:start --> - - - This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/yugabyte/yugabyte-db/38401) <!-- Reviewable:end --> Co-authored-by: ellabaron-code <ella.baron@shopify.com>

Commit:1cd6e13
Author:Deepti Kumar
Committer:GitHub

[PLAT-22592,PLAT-22994] YBA: Keep telemetry exporter credentials out of collector config and logs (#37023) ## Summary Telemetry exporter credentials leaked in two places: - PLAT-22592: the collector config held them in cleartext. On K8s this config reaches ConfigMaps, Helm values and support bundles. - PLAT-22994: on VM, `NodeManager` put the AWS keys on the `ybcloud` command line, which YBA logs. Fix: - The collector config refers to each credential as `${env:YB_OTEL_<provider-uuid>_<field>}`. - K8s: the values go into the existing `<release>-otel-secrets` Secret through `otelCollector.secretEnv`. No chart change. - VM: a new proto map, `otelColSecretEnv`, sends the values to node-agent. Node-agent writes them to `secrets.env` (mode 0600), and the unit loads that file. The AWS credentials block is written the same way, not with the keys on a command line. - `NodeManager` no longer puts credentials on the command line. The logged command also masks them. - YBA removes line breaks from single-line credentials. Node-agent rejects them as a second check. The first commit message says the files are written with `O_NOFOLLOW`. A later commit writes them as the task user instead. Not in this change: - After a YBA rollback, `secrets.env` stays on VM nodes. - On K8s, two AWS or GCP providers on one universe still share fixed env names. ## Upgrade/Rollback safety - `yb.proto` adds field 15 to `InstallOtelCollectorInput`. It is additive. - If node-agent is not upgraded yet, the task fails before it uploads any collector file. Retry after the upgrade. - Existing VM nodes keep the old config until the next `ManageOtelCollector` run. - Rollback: the old node-agent writes the old inline config again, so the collector keeps working. ## Test plan - [x] `OtelCollectorConfigGeneratorTest`: VM and K8s configs hold no credential, and every `${env:X}` resolves from the env that is sent, also on older charts and with a paused audit exporter. Line breaks are removed. - [x] `NodeAgentRpcPayloadTest`, `ShellProcessContextTest`, `LokiConfigTest`, `RedactingServiceTest`, `TelemetryProviderServiceTest`, `TelemetryProviderControllerTest`, `ShellProcessHandlerTest`, `SupportBundleUtilTest`, `NodeManagerTest`. - [x] node-agent `app/task` tests: file quoting and modes, line breaks rejected, managed block removal keeps files at 0600. - [x] Live GCP VM universe, user and root node-agent, with Datadog, Loki, Dynatrace and CloudWatch export: export works, and no credential is in `config.yml`, the unit, or the YBA and node-agent logs. - [ ] Live K8s universe.

Commit:2123e30
Author:Piyush Jain
Committer:Piyush Jain

[#37091] YSQL: Let a transaction read an index it created An index created in a DDL transaction is BIRTH_TIME_PENDING_DDL_COMMIT until that transaction commits, and the read guard answers reads of it with `LeaderNotReadyToServe`, which is retried until the RPC deadline. A statement that reads the index in the transaction that created it (index scan, foreign key check against a new UNIQUE constraint, `ON CONFLICT` on a new unique index, `yb_index_check()`, an extension script) therefore hung: the index only becomes final once that same transaction commits. The creating transaction wrote the index, so the index is complete for it. Let its reads through: - The master stores the creating transaction's id in the new `IndexInfoPB.birth_time_pending_txn_id` together with the pending state, and clears both when the index is finalized. The tablet does the same in `FinalizeIndexBirthTime`. - The read guard does not reject a read of a pending index whose transaction id matches. - pggate no longer sends the request to an index with `skip_intents`, so that it carries the transaction's metadata. With the new-relation fastpath, reads of a table created in the transaction, and until now of its indexes, are sent without it. The index read still happens at in_txn_limit. Restore the three regress reads that 4e76682e27 commented out. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

Commit:bd386c4
Author:Piyush Jain
Committer:Piyush Jain

[#30797] YSQL: Set index birth_time for indexes built nonconcurrently (#37103) A CREATE INDEX run inside a transaction block is built nonconcurrently and skips the backfill that sets the index birth_time, so birth_time stays 0. The docdb read guard then does not throw a kSnapshotTooOld error for a snapshot read whose read time precedes the index commit, and the index rows (written at the commit) are not visible to that read, so it silently returns an incomplete result. This is the data loss seen in the Jepsen si.append-table test. On a successful nonconcurrent index create, set the index birth_time to the master clock time (>= the DDL commit time, see below) and propagate it to the tablets via a new MarkNonConcurrentIndexDone tserver admin RPC. To reject reads until birth_time is final, introduce a `birth_time_state` enum in the index metadata (IndexInfoPB), set by the master at index creation: - BIRTH_TIME_FINAL (default): birth_time, if set, is final. - BIRTH_TIME_PENDING_BACKFILL: online (backfilled) index, until the backfill completes. - BIRTH_TIME_PENDING_DDL_COMMIT: nonconcurrent index created in a DDL transaction, until the transaction commits. The backfill path was relying on the `retain_delete_markers` flag as a proxy for this; post this change it can rely on `birth_time_state` too (both are checked). The read guard returns two different errors: - While the state is pending on the tablet (birth_time 0): `LeaderNotReadyToServe`. The tserver's client retries it until the RPC deadline, so the read waits until the tablet is finalized. - Once birth_time is set, for a read whose read time is before it: `SnapshotTooOld` (SQLSTATE 72000). The query layer does not retry it, so the application has to retry the transaction. Details: - Online indexes: the state is cleared together with setting birth_time when the backfill completes (master AllowCompactionsToGCDeleteMarkers, tablet BackfillDone). They never take the nonconcurrent path. - Nonconcurrent indexes: the DDL transaction verifier (HandleSuccessfulYsqlDdlTxn) sets birth_time, clears the state and posts AsyncMarkNonConcurrentIndexDone (new monitored task type) to each tablet. The tablet replicates a ChangeMetadata op with the new mark_non_concurrent_index_done field, which sets birth_time and clears the state. This is separate from BackfillDone / mark_backfill_done so that the two paths are not conflated, and each path only clears its own pending state. A finalize op for an index already removed from the tablet (e.g. a DROP INDEX on a colocated table racing with the RPC) is skipped instead of failing the replicated op; this also applies to BackfillDone. - birth_time >= commit time: the transaction coordinator picks the commit time, so the master's clock is moved past it before birth_time is picked. ReportYsqlDdlTxnStatus now carries the reporting tserver's hybrid time (which the commit response moved past the commit time), and when the master polls the transaction instead, its clock is updated from the GetTransactionStatus response. - Recovery of a lost finalize RPC: the master does not resend it (e.g. after a master failover or when a task gives up). The tablet validator also checks indexes whose local state is not final: a tablet is queued when it opens, and at the next periodic run (`tablet_validator_retain_delete_markers_validation_period_sec`, 60s by default) the validator asks the master (GetBackfillStatus now returns the master's birth_time_state) and finalizes the tablet metadata once the master reports the index final. Reads through the index retry until then. The validator asks only once per tablet open, so if that check ran while the index was still pending on the master, a later lost RPC is not repaired until the tablet is reopened, and not even then within an hour, since the validator reuses its cached "pending" answer for that long. The same gap already exists for BackfillDone; both are tracked in - xCluster targets: the birth time is not propagated yet (#33155). However, the target re-creates the index via DDL replication, so it will set the pending state and clear it on commit to allow reads (without stamping a local birth_time, so it stays 0 as before). - Known limitation: statements in the transaction that created a nonconcurrent index and read that index (index scans, `INSERT ... ON CONFLICT` on a unique index, foreign key checks against a UNIQUE constraint, `yb_index_check()`) are retried until statement_timeout, since the index only becomes final once that same transaction commits. This includes `CREATE TABLE ... UNIQUE` followed by an upsert in the same transaction. Tracked in #37091, to be fixed before a release containing this change. The three regress reads that do this (yb.orig.ddl_txn_{triggers,event_triggers,indices}) are commented out until then. Fixes #30797 - New AutoFlag enable_ysql_index_birth_time_state (kLocalPersisted). The master sets a non-final birth_time_state only when it is on; while it is off, behavior is unchanged (every index is final). It is an AutoFlag so that no index is left pending forever: an old master that becomes leader before the CREATE INDEX transaction is verified would not finalize the index, and after a rollback, indexes that were still pending would stay pending with nothing to finalize them on re-upgrade. - The new fields (IndexInfoPB.birth_time_state, IndexStatusPB.birth_time_state, ChangeMetadataRequestPB.mark_non_concurrent_index_done) and the new RPC are only used for indexes with a non-final state, which exist only once the AutoFlag is promoted, i.e. after all tservers run the new version. Older tservers never receive them. - A missing IndexInfoPB.birth_time_state means final, so indexes created before the upgrade keep the current behavior. An older binary that reads persisted metadata containing the field ignores it, and only loses the new read guard. A GetBackfillStatus response without IndexStatusPB.birth_time_state (from an older master) is treated as not final, so the tablet validator does not finalize anything from it. - New optional ReportYsqlDdlTxnStatusRequestPB.propagated_hybrid_time: an older master ignores it and an older tserver does not send it, which is the behavior before this change in both cases. - [x] Ran (release): ``` ./yb_build.sh release --cxx-test pgwrapper_pg_concurrent_ddls-test --gtest_filter 'PgConcurrentDDLsTest.SnapshotRead*:PgConcurrentDDLsTest.NonConcurrentIndex*:PgConcurrentDDLsTest.PendingIndexRejectsStaleRead:PgConcurrentDDLsTest.OnlineIndex*' ./yb_build.sh release --cxx-test pgwrapper_pg_index_backfill-test --gtest_filter 'PgIndexBackfillTest.RetainDeleteMarkers*:PgIndexBackfillColocated.ColocatedRetainDeleteMarkers*' ./yb_build.sh release --cxx-test master_multi_step_monitored_task-test ./yb_build.sh release --java-test 'org.yb.pgsql.TestDdlTransactionBlocks' ``` New tests in pg_concurrent_ddls-test: SnapshotReadFreshNonConcurrentIndexMatrix, PendingIndexRejectsStaleRead, OnlineIndexRejectsStaleReadBeforeBackfillDone, OnlineIndexBirthTimeFinalOnMaster, NonConcurrentIndexWithDropColumnInSameTxn, NonConcurrentIndexFinalizeAfterDropIndex.

Commit:15b5c07
Author:John Meehan
Committer:John Meehan

[#33444] DocDB: Persist the released marked-write watermark and expose an index-backfill downgrade-safety probe The tserver half of the downgrade fence for deferred unique-index verification. A pre-feature binary replays marked backfill writes with positional write IDs -- byte-divergent replicas -- so a cluster must not be allowed to downgrade while any tablet still holds an active ordering generation or marked WAL entries that have not been flushed below the regular-DB frontier. This change makes that per-tablet condition provable and convergent and exposes it over an admin RPC; the master-side veto that consumes it (refusing to demote the capability AutoFlag) lands with the flag itself and the activation work, so the flag cannot ship ahead of the behavior it protects. Two release changes make the condition convergent: - Release persists released_marked_write_watermark in the generation superblock record: the highest marked-write Raft index the tablet has applied (tracked in memory, monotonic, re-seeded by bootstrap replay, which re-applies exactly the marked writes still above the flushed frontier). Anchoring to the last marked WRITE rather than the release op matters: metadata ops never advance the RocksDB frontier, so a release-op watermark would never converge on an idle tablet. - Release schedules an async regular-DB flush (FlushReason::kIndexBackfillGenerationRelease), pushing the marked writes below the frontier within seconds instead of waiting for an organic flush. A marked write that was appended after the release op but applies after it (the leader admits marked writes until the release applies) sits above the persisted watermark; applying it under the released generation raises the watermark and schedules the same flush, so the fence keeps waiting for exactly that write. The released record now survives in the superblock (previously only active generations were persisted) until the next activation. The tablet validator's straggler heal records the watermark and schedules the flush too, deferring its heal to a later cycle if the tablet is not running (the watermark lives on the Tablet). Because that heal re-checks long-active generations periodically (tablet_validator_index_backfill_generation_recheck_sec), a generation stranded by a lost job converges to released on the running binary; a drain never depends on a restart. CheckIndexBackfillDowngradeSafety (tserver admin RPC) reports, per server, the number of tablets with an active ordering generation and the number of released tablets whose watermark is still above the flushed frontier. It fails closed while a tablet cannot report a frontier (bootstrapping, shutting down) and skips tombstoned/deleted peers (WAL removed) and completed split parents (flushed before the split checkpoint), so husks on the master's cleanup schedule cannot keep a drain from converging. A released tablet it finds still unflushed gets the flush scheduled again, so an idle tablet whose release-time flush was lost converges on the next probe rather than never. Test Plan: ./yb_build.sh release --cxx-test pg_index_backfill-test --gtest_filter 'PgIndexBackfillVerifier.DowngradeSafetyProbeReportsHeldGenerations/0' (release blocked by the fixture: the probe counts the held generations) ./yb_build.sh release --cxx-test pg_index_backfill-test --gtest_filter 'PgIndexBackfillVerifierReleased.DowngradeSafetyProbeClearsAfterRelease/0' (release + release flush: both counts converge to zero within seconds) ./yb_build.sh release --cxx-test pg_index_backfill-test --gtest_filter 'PgIndexBackfillVerifierReleasedNoFlush.DowngradeSafetyProbeReportsUnflushedMarkedWal/0' (release flush disabled via TEST_skip_index_backfill_generation_release_flush: no generation is active, yet the unflushed marked WAL is still reported) ./yb_build.sh release --cxx-test tablet_peer-test (watermark persistence, replay re-seeding, release record survival) ./yb_build.sh release --cxx-test fixed_hybrid_time_write_id-itest

Commit:bc38777
Author:John Meehan
Committer:John Meehan

[#33444] DocDB: Add a keyed, per-job violation fingerprint to unique-index verification Verification failure reports have been value-free by design: encoding classes and counts, never the duplicate key. That satisfies the privacy contract but leaves support blind to correlation -- "is this the same duplicate as the last retry?" This adds the design document's keyed, scope-limited diagnostic fingerprint: a non-reversible token identifying the violating index key, safe for low-cardinality values because confirming a candidate requires the key, and the key's scope limits how long confirmation is possible at all. Mechanics: - Per-job secret (32 bytes, OpenSSL RAND_bytes; a CSPRNG failure fails the job launch instead of the master) generated in BackfillTable::Launch for SKIP_ALL jobs (other modes never verify and carry no key), persisted in BackfillJobPB (field 9, latched-per-job like the mode and the gate), and deleted with the job: fingerprints correlate across the job's retries and master failover, and are unconfirmable once the job is gone. - The verifier exposes the first violating group's raw encoded DocKey in-process only (excluded from ToString; never serialized). The tserver handler renders the token -- "<jobtag8hex>-<digest16hex>", both truncated HMAC-SHA256 under the job key (yb/util/keyed_fingerprint.{h,cc}) -- into a structured response field and a "fingerprint=" suffix on the still-value-free reason. The job tag is derived from the key alone, so tokens from different jobs are visibly incomparable. - The reason rides the existing paths into the persisted verification state, the master outcome log, and (in gating mode) backfill_error_message -> the CREATE INDEX error. The structured token is also persisted (UniqueIndexVerificationStatePB field 5) for support tooling. - The key never leaves the master/tserver trust domain. One helper, ScrubBackfillJobSecrets (catalog_manager_util), is applied at every SysTablesEntryPB egress: GetBackfillJobs and the admin sys-catalog dump (DumpSysCatalogEntries, which otherwise renders the table entry verbatim; both pinned by test), CatalogManager::DumpState, the catalog loader's and the create/alter/delete paths' VLOGs of whole table entries, and the snapshot export repack (a backup taken during a backfill would otherwise carry the key off the cluster). Both request-log sites (master send VLOG, tserver receive DVLOG) share RedactedDebugString -- which also elides start_key, a raw index DocKey. Test Plan: ./yb_build.sh release --cxx-test keyed_fingerprint-test --gtest_filter 'KeyedFingerprintTest.TokenFormatAndDeterminism' ./yb_build.sh release --cxx-test keyed_fingerprint-test --gtest_filter 'KeyedFingerprintTest.TagIdentifiesKeyScope' (token format, determinism under one key, shared scope tag per key, cross-key incomparability) ./yb_build.sh release --cxx-test unique_index_verifier-test --gtest_filter 'UniqueIndexVerifierTest.ViolationExposesGroupPrefixInMemoryOnly' (the privacy pin: the raw group prefix is exposed in memory for the caller and excluded from ToString) ./yb_build.sh release --cxx-test pg_index_backfill-test --gtest_filter 'PgIndexBackfillFailClosedVerification.ViolationFailsCreateIndex/0' (extended: the CREATE INDEX error now carries "fingerprint=<token>"; verified rendering end-to-end: fingerprint=2d929cea-dbf259e3ff0d7297) ./yb_build.sh release --cxx-test pg_index_backfill-test --gtest_filter 'PgIndexBackfillMultiMaster.UniqueCheckModePersistedAcrossMasterFailover/0' (the key is absent from GetBackfillJobs and from the sys-catalog dump of the table entry while the job is live) Full unique_index_verifier-test 20/20. Regressions: shadow violation (waiter matches the enriched line), shadow clean multi-tablet, failover resume -- all green.

Commit:2491fe0
Author:John Meehan
Committer:John Meehan

[#33444] DocDB: Add verification scan accounting and master outcome counters Verification/job observability for deferred unique-index verification, plus the accounting fix deferred from the verifier-core review: versions_scanned was inconsistent on the bounded-memory fallback path (the forward pass stopped counting at buffer overflow, then the reverse walk re-counted its re-visits -- over- and under-counting the same group). The forward pass now owns the accounting -- every physical version in a scanned group is counted exactly once regardless of replay path -- and the new fallback_groups counter records the reverse walk's extra work instead (a fallback group is read roughly twice). Plumbing and aggregation: - VerifyUniqueIndexTabletResponsePB gains fallback_groups (field 8); the handler fills it from the verifier result. - The shadow coordinator aggregates the job's scan totals from every response and logs them with the recorded outcome: "... VERIFY_CLEAN [observational] (dockey_groups=300, versions=300, fallback_groups=0)". - Five cluster-level master counters (backfill_aborted pattern): unique_index_verification_outcome_{clean,violation,inconclusive} (one per job: counted on the transition into the terminal state, which is final -- a coordinator failure after a recorded verdict neither overwrites it nor counts again) and unique_index_verification_{versions_scanned,fallback_groups} (aggregated from responses; late responses dropped by the single-winner guard still count -- the scan happened). A persistently high fallback_groups rate is the signal to raise unique_index_verify_max_buffered_versions_per_group; the counter description says so. Test Plan: ./yb_build.sh release --cxx-test unique_index_verifier-test --gtest_filter 'UniqueIndexVerifierTest.FallbackAccountingMatchesBuffered' (the accounting pin: identical data verified through the buffered and fallback paths reports identical versions_scanned and dockey_groups_scanned; the fallback run reports fallback_groups = 1) ./yb_build.sh release --cxx-test pg_index_backfill-test --gtest_filter 'PgIndexBackfillShadowVerification.CleanOutcomeRecordedMultiTablet/0' (extended: asserts the master outcome counter == 1 -- exact on a fresh cluster, catches double-counting -- and versions_scanned > 0 via the cluster metric entity) Full unique_index_verifier-test 19/19. Regressions: ViolationRecordedButDoesNotBlockPublication (waiter matches the enriched log line), ViolationFailsCreateIndex (gating path through Record), failover resume, paginated clean -- all green.

Commit:5e87216
Author:John Meehan
Committer:John Meehan

[#33444] DocDB: Gate unique-index publication on the deferred verification outcome The last functional piece of deferred uniqueness verification: behind ysql_index_backfill_fail_closed_verification, the verification phase's outcome decides publication instead of merely being recorded. The publication order becomes the design document's: backfill completes -> stay at DO_BACKFILL -> verify all index tablets -> persist the outcome -> RWD -> publish birth_time -> release generation + retention -> indisvalid The gate defaults to ON. Observational (record-and-publish) mode is the explicit opt-out, never the default polarity of a safety property; the flag is inert while no job selects SKIP_ALL, and the mode selector remains hardcoded to CHECK_ALL, so nothing changes for any cluster until activation. Tests drive the gate through the master-side TEST override. In gating mode, backfill-success marking for the job's unique index is deferred from chunk completion to verification completion: a clean index is then marked successful (including the xCluster backfill-completed notification, which must not fire for an index that may still fail) and proceeds through the terminal funnel to READ_WRITE_AND_DELETE and birth_time publication. A VIOLATION or unresolvable INCONCLUSIVE outcome fails the index through the existing backfill failure path -- the persisted backfill status carries a value-free reason, CREATE INDEX returns it, the index is never READ_WRITE_AND_DELETE and never indisvalid, and the existing invalid-index cleanup applies. Since bae5985de0c/#33837 that status reaches the backend with its error codes intact, so a VIOLATION is tagged PgsqlError(YB_PG_UNIQUE_VIOLATION): CREATE UNIQUE INDEX fails with SQLSTATE 23505 exactly as a CHECK_ALL duplicate (or vanilla PostgreSQL) does. INCONCLUSIVE and coordinator failures carry no code (XX000): they are infrastructure failures, not statements about the data. A coordinator failure records INCONCLUSIVE against the job's index (resolvable from the job even when the phase failed before selecting its target) and fails it rather than publishing; an index dropped before the phase could verify it takes the same failure path. A non-unique index has nothing to verify and is marked successful at chunk completion as before. The marking is filtered through indexes_to_build() (the index while still IN_PROGRESS), and a resume that finds a persisted terminal outcome re-applies the marking and finishes without re-scanning: the job stays persisted through the release wait, so a failover after publication would otherwise re-run the phase and mark the index successful a second time -- including a second xCluster backfill-completed checkpoint. YSQL backfills exactly one index per job (MultiStageAlterTable never batches or defers YSQL indexes), so the gate reasons about "the job's index" rather than a per-index set. The gating decision is latched per job and persisted in BackfillJobPB, mirroring the uniqueness-check mode: failover, retries, and runtime flag flips cannot reinterpret an active job. Reading the flag live at each decision point would misbehave in both directions -- gating->observational after the deferred marking leaves the index IN_PROGRESS at the funnel; observational->gating mid-phase would fail an index whose success (and xCluster notification) already fired. The funnel warns if it ever sees an IN_PROGRESS index (reachable when the terminal-state marking itself failed, e.g. on a lost leader epoch); the pre-existing mapping publishes such an index as failed. The gate implies the verification phase runs even when ysql_index_backfill_shadow_verification is off. Observational mode is untouched -- outcome logs now carry a [gating] / [observational] tag. Test Plan: ./yb_build.sh release --cxx-test pg_index_backfill-test --gtest_filter 'PgIndexBackfillFailClosedVerification.CleanBuildPublishes/0' ./yb_build.sh release --cxx-test pg_index_backfill-test --gtest_filter 'PgIndexBackfillFailClosedVerification.ViolationFailsCreateIndex/0' ./yb_build.sh release --cxx-test pg_index_backfill-test --gtest_filter 'PgIndexBackfillFailClosedInconclusive.InconclusiveFailsCreateIndex/0' ./yb_build.sh release --cxx-test pg_index_backfill-test --gtest_filter 'PgIndexBackfillFailClosedPhaseFailure.CoordinatorFailureFailsCreateIndex/0' ./yb_build.sh release --cxx-test pg_index_backfill-test --gtest_filter 'PgIndexBackfillShadowVerificationFailover.DuplicateWrittenBeforeFailoverIsStillReported/0' ./yb_build.sh release --cxx-test pg_index_backfill-test --gtest_filter 'PgIndexBackfillShadowVerification.ViolationRecordedButDoesNotBlockPublication/0' The violation test is the feature's end state: a real pre-existing duplicate survives the SKIP_ALL build and fails CREATE INDEX with a verification error carrying SQLSTATE 23505; pg_index shows indisvalid = false, the base table stays usable, and the invalid index is droppable. The inconclusive test pins fail-closed = "not proven clean is not clean": an unresolvable verification outcome fails the build even though the data is clean. The phase-failure test injects a coordinator failure and asserts the uniform INCONCLUSIVE (coordinator error) failure with no 23505. The failover test pins that re-activation (which moves the generation's base_op_index above every marked write of the first leader) does not hide a duplicate written before the failover. The observational test now opts out of the default-on gate explicitly and still records-and-publishes. Also run: the SKIP_ALL / verifier / shadow / multi-master fixtures with the gate at its new default. Three pre-existing tests change with the default: the verifier fixture (which drives the RPC itself over builds that contain duplicates) and the mode plumbing test UniqueCheckModePlumbedFromMasterToWritePath (which deliberately publishes an index over a duplicate to prove the write path skipped the checks) opt out explicitly, and UniqueCheckModePersistedAcrossMasterFailover now asserts the verification VIOLATION -- a resumed job that wrongly re-selected CHECK_ALL fails earlier with the foreground "duplicate key" error instead, so the assertion still distinguishes the two.

Commit:7caf555
Author:John Meehan
Committer:John Meehan

[#33444] DocDB: Add the shadow verification coordinator with durable job state The first production caller of the verification RPC: after a SKIP_ALL unique-index backfill completes -- while the ordering generations are still active, before the terminal funnel releases them -- the master runs the deferred uniqueness verification scan observationally. The outcome is persisted and logged but never gates index publication; the fail-closed verify-before-publication gate is a separate, later capability. Disabled by default (ysql_index_backfill_shadow_verification). The phase is single-index by construction: SKIP_ALL is YSQL-only and MultiStageAlterTable backfills exactly one YSQL index per job (no batching, no deferral), so the job's unique index is the verification target. The coordinator holds one state, one target tablet set and one outcome, and a job that violates the invariant fails closed (DFATAL in debug builds, INCONCLUSIVE in release). Durable state (BackfillJobPB.unique_index_verification, one message): verification state, verify_upper_ht, the target tablet set, tablets confirmed clean, and a value-free reason. verify_upper_ht is chosen ONCE from cluster hybrid time (never wall clock) and persisted before any scan -- retries and failover resume verify the same window. Clean tablets are durable progress at tablet granularity; finer intra-tablet resume keys are deliberately not persisted, since re-scanning one tablet after failover is idempotent. All persistence goes through the indexed table's write lock and a sys-catalog upsert under the job's leader epoch. Tablet-set stability. Splits of the index are allowed during the build (children inherit the ordering generation), so verification establishes the stable set itself: persisting IN_PROGRESS arms a durable split fence (the tablet-split manager refuses to split an index whose job's verification state is not VERIFY_NONE -- it reads sys-catalog state, so it holds across failover until the funnel clears the job after release), then the coordinator disables automatic splitting of the index, drains any split in flight (the same wait the backfill applies to the indexed table), resolves the tablets and persists them as the target set. A resume that finds a different set reports VERIFY_INCONCLUSIVE instead of scanning a moving set. Coordinator mechanics (VerifyUniqueIndexForTablet, the GetSafeTimeForTablet task shape): - Bounded per-tablet concurrency (index_backfill_shadow_verification_max_concurrent_tablets). - Pagination: index_backfill_verify_dockey_groups_per_rpc caps each RPC's group budget; a tablet resumes from the returned resume key without touching the join counts. - Short-circuit under a single-winner protocol: the terminal check and the launch-next / index-done decision are one critical section, so a CLEAN callback racing a short-circuiting VIOLATION can neither overwrite the recorded outcome nor double-finish the phase; the slow clean-tablet persist happens after the join decision. Late responses cannot change a recorded outcome. The join guards on job failure, not done() -- the whole phase runs in the job's kSuccess state. Aborted tasks join too: the task is registered on the index table, which DROP INDEX tears down while the phase waits on it, and a join that never arrives would strand the job; an abort is an inconclusive tablet (a leadership-loss abort then fails the epoch-checked sys-catalog writes, and the new leader resumes the phase). - Fail-open on coordinator errors: the phase is on the CREATE INDEX critical path, so every coordinator failure (task launch, persistence, pagination resume) degrades to VERIFY_INCONCLUSIVE and continues into publication -- no failure may strand the job short of the terminal funnel, and a latch keeps the funnel single-entry when a failure after the recorded outcome degrades through the same path. The tserver returns typed error codes for deterministic failures (generation mismatch, non-unique-index tablet, cutoff violations) so the coordinator does not burn its retry budget on requests that cannot succeed. - An index with no tablets records VERIFY_INCONCLUSIVE: an empty tablet set is not evidence of uniqueness. - Master failover mid-phase resumes with the persisted window. Generic lost-backfill resume (load-time enqueue, resuming a job whose table has left ALTERING) is upstream since #33207/c830a71489 -- this part originally carried equivalent machinery and now relies on upstream's, adding only the verification-phase awareness: a resumed job whose indexes are all SUCCESS skips straight to the completion phase (and from there into verification) instead of relaunching empty backfill chunks. - No generation_base_op_index in the requests, on purpose: the base is per-tablet (the activation op's own Raft index) and a re-activated generation's higher base is still the generation this job's marked writes live under; active + index-table match is the semantic guard. Each verification RPC gets its own deadline (index_backfill_verify_rpc_timeout_ms, default 60s), mirroring the backfill chunks' ysql_index_backfill_rpc_timeout_ms instead of inheriting the generic 30s master_ts_rpc_timeout_ms: the deadline sizes one page of the scan (the tserver stops a grace margin early and returns a resume key), and pages should be sizable independently of unrelated master RPCs. Test Plan: ./yb_build.sh release --cxx-test pg_index_backfill-test --gtest_filter 'PgIndexBackfillShadowVerificationSlow.SplitFencedDuringVerification/0' (split refused by the durable fence while the scan runs; accepted again after release) ./yb_build.sh release --cxx-test pg_index_backfill-test --gtest_filter 'PgIndexBackfillShadowVerification.CleanOutcomeRecordedMultiTablet/0' ./yb_build.sh release --cxx-test pg_index_backfill-test --gtest_filter 'PgIndexBackfillShadowVerification.ViolationRecordedButDoesNotBlockPublication/0' ./yb_build.sh release --cxx-test pg_index_backfill-test --gtest_filter 'PgIndexBackfillShadowVerificationPaginated.CleanAcrossManyRpcs/0' ./yb_build.sh release --cxx-test pg_index_backfill-test --gtest_filter 'PgIndexBackfillVerifier.GenerationMismatchFailsCleanly/0' ./yb_build.sh release --cxx-test pg_index_backfill-test --gtest_filter 'PgIndexBackfillShadowVerificationFailover.ResumesWithPersistedWindowAcrossFailover/0' The violation test is the shadow-semantics pin: a real duplicate is recorded (NOT CLEAN: VERIFY_VIOLATION in the master log) while CREATE INDEX still succeeds and the index serves reads. The multi-tablet test exercises the fan-out/join; the paginated fixture forces one DocKey group per RPC through the resume path. GenerationMismatchFailsCleanly covers the 4b review's L1 (table-id and base mismatches fail without scanning). The failover test holds the verification RPCs open with a retryable rejection, steps the master leader down mid-phase, and asserts the new leader resumes with the persisted window before completing the build. ./yb_build.sh release --cxx-test pg_index_backfill-test --gtest_filter 'PgIndexBackfillShadowVerificationDeadlinePaginated.CleanAcrossDeadlinePages/0' The deadline-paginated test is the regression for the measurement-run defect: a slowed scan against a 3s verify budget and 1s grace margin must complete CLEAN through several deadline-bounded pages; before the margin (4b) and this budget existed, the page response always arrived after the coordinator's deadline and the tablet failed INCONCLUSIVE at the retry ceiling. Regressions: shadow-flag-off skip_all e2e, CHECK_ALL mode test, 4b verifier e2e -- all green. The failover-resume e2e now passes through upstream

Commit:bc72b30
Author:John Meehan
Committer:John Meehan

[#33444] DocDB: Expose the unique-index verification scan as a tserver admin RPC Wires the verifier core to the world: VerifyUniqueIndexTablet on the tablet server admin service, read-only and paginated, with the preconditions the scan's correctness rests on enforced at the entry point. No production caller yet -- the verification coordinator drives it in a later part; end-to-end tests drive it directly. The RPC (leader-only, BackfillIndex-style pagination): - Validates first, before anything expensive: the active ordering generation against the request's expectation (index table, and base_op_index when set -- the coordinator always sets it; omitting it relaxes to "any active generation for this table" for test tooling), the window, and that the tablet is not colocated (one generation record and one key space per tablet; SKIP_ALL never selects colocated indexes). A released or replaced generation fails the RPC without scanning: the window no longer describes the tablet's marked writes. - Discharges the applied-intents precondition in two steps: waits for the tablet's safe time to pass verify_upper_ht (leader lease held; no future write can land at or below the bound), then TransactionParticipant::ResolveIntents(verify_upper_ht) -- safe time alone does not imply applied, since transactional apply is asynchronous, and a committed-but-unapplied foreground write is invisible to a regular-DB-only scan (the record missed could be exactly the foreground half of a duplicate). ResolveIntents now also waits for the batched applies of large transactions (more than txn_max_apply_batch_records intents) to finish: a local commit time alone leaves their remaining records in the intents DB. - Holds a reader registration (ScopedReadOperation) at backfill_read_ht.Decremented() for the whole scan. Registration fails with SnapshotTooOld unless the committed history cutoff is strictly below the window -- strict because the marked duplicates the verifier exists to find share one hybrid time, and a cutoff equal to the window's lower bound lets compaction drop the lower-write-ID versions at that coordinate -- and while held it pins every future cutoff below the window. The retain_delete_markers bypass in RegisterReaderTimestamp is disabled under an active ordering generation (its justification assumes the forward/backward uniqueness checks regenerate conflicts, which SKIP_ALL removes), so the registration is a real check. The *applied* cutoff -- the regular DB's flushed frontier, the cutoff past compactions actually used -- is checked before and after the scan as well, so a compaction that was already running with a later cutoff fails the call instead of silently removing window history. Deterministic failures (generation mismatch, non-unique-index tablet, colocated tablet, invalid or inverted window) return typed error codes, so callers do not burn a retry budget on requests that cannot succeed. A page that completes no DocKey group before the deadline (the preamble consumed the budget) is answered ServiceUnavailable rather than as a clean, empty page the caller would count. The scan holds the RocksDB shutdown guard for its duration (the not-blocking variant: tablet shutdown aborts the scan, never the reverse), and an unset backfill_read_ht is rejected at the boundary -- it decodes to the invalid sentinel, which compares greater than any real hybrid time and would make the history-cutoff checks pass vacuously. Tablet::VerifyUniqueIndex resolves the scan options from the tablet: the index table's ybidxbasectid column (the table held for the scan), the tablet metadata as the schema-packing provider, the regular DB within the tablet's key bounds, and the new runtime flag unique_index_verify_max_buffered_versions_per_group. TEST_block_index_backfill_ordering_generation_release (master) holds the terminal funnel's release open so tests can verify an index built by the real backfill path while its generation is still active. The e2e suite closes the encoding-drift loop the verifier-core review required: every physical shape asserted by the unit tests is re-asserted against tablets written by the production SKIP_ALL backfill and live pg DML. Deadline grace margin (found by the measurement runs): the scan paginates at the raw client deadline, so a deadline-bounded page's response was serialized exactly when the coordinator's RPC expired -- always discarded, with every retry rescanning the same page from scratch until the retry budget failed the tablet INCONCLUSIVE (deterministically ~1176s under the default 30s timeout and 20-retry budget; any tablet whose scan exceeds one RPC deadline hit it). The handler now stops unique_index_verify_deadline_grace_margin_ms (default 2s, clamped to half the remaining budget) before the client deadline, mirroring the backfill chunk path's backfill_index_timeout_grace_margin_ms. A new TEST_unique_index_verify_delay_per_group_ms flag makes deadline pagination deterministically reproducible with small data. Test Plan: ./yb_build.sh release --cxx-test pg_index_backfill-test --gtest_filter 'PgIndexBackfillVerifier.CleanBuildWithForegroundDml/0' ./yb_build.sh release --cxx-test pg_index_backfill-test --gtest_filter 'PgIndexBackfillVerifier.PkUpdateAcrossBackfillBoundaryIsClean/0' ./yb_build.sh release --cxx-test pg_index_backfill-test --gtest_filter 'PgIndexBackfillVerifier.DetectsPreexistingDuplicates/0' ./yb_build.sh release --cxx-test pg_index_backfill-test --gtest_filter 'PgIndexBackfillVerifierPackedUpdate.PkUpdateAcrossBackfillBoundaryIsClean/0' ./yb_build.sh release --cxx-test pg_index_backfill-test --gtest_filter 'PgIndexBackfillVerifierReleased.VerifyAfterReleaseFailsCleanly/0' The PK-update regression runs under both physical shapes (standalone ybidxbasectid column with default flags; packed V2 + kIsUpdateFlag with pack_full_row_update + mark) and must be Clean in both. Regressions: unique_index_verifier-test (18), OrderingGenerationActivatedAndReleased (release still fires with the TEST flag off), OrderingGenerationChangeMetadataOp tablet_peer test -- all green.

Commit:672c550
Author:John Meehan
Committer:John Meehan

[#33444] DocDB: Orchestrate the index-backfill ordering-generation lifecycle from the master The persisted ordering generation (#33580) had no production lifecycle: nothing activated it before marked backfill writes and nothing released it. This part wires the full lifecycle and flips marked-write gating fail-closed. Activation rides ChangeMetadataOperation (a new `ChangeMetadataRequestPB.index_backfill_ordering_generation` variant, applied via `Tablet::UpdateIndexBackfillOrderingGeneration` exactly like `mark_backfill_done`): Raft-replicated, follower-applied, WAL-replayed, and `base_op_index` is the activation operation's own Raft index, so every replica and every replay derives the same base with no master-side bookkeeping. Re-activation (failover resume) is idempotent -- the base only moves up, and the verifier classifies marked records by write-ID domain, not by base, so candidates written before a failover are never hidden. Master flow, SKIP_ALL jobs on PGSQL indexed tables only (the mode rides YSQL chunk requests; YCQL backfill writes are never marked): - Before the first chunk (`DoBackfill`), the job disables and drains index-table splitting, then fans out one waited `UpdateOrderingGenerationForTablet` task per index tablet (the `GetSafeTimeForTablet` join pattern). All acks -> splitting is re-enabled and the chunks launch; any failure -> the job aborts before a single marked write exists, and CREATE INDEX fails cleanly. The drain closes the activation/split TOCTOU: a split in flight across activation could produce children whose superblock checkpoint predates the record. Splits are otherwise allowed throughout the build -- children continue the parent's Raft sequence and inherit the record, so per-key write-ID uniqueness and marked-write admission survive a split (`SplitAllowedDuringBackfill`). Only the verification scan needs a stable tablet set; that fence lands with it. - Release is owned by the persisted job. The terminal funnel (`UpdateIndexPermissionsForIndexes`, which every success/failure/abort path traverses) updates the index permissions, then fans out waited release tasks in rounds over the live index tablets -- a round ends when every task joined, the next covers tablets that have not acknowledged (split children created meanwhile) -- and clears the job (`FinishIndexPermissionUpdate`) only once a round finds nothing left. A tablet the master no longer lists -- deleted with its index, or a retired split parent -- counts as released (decided from the catalog; a tserver NotFound only means a leader move); any other failure re-launches that tablet's release after `index_backfill_generation_release_retry_delay_ms`, without bound, from the task's own join so each tablet has one retry chain, and the chain stops once this master's leader epoch is stale. A master that dies mid-release therefore leaves a persisted job that the next leader resumes (`LaunchBackfillTablets` fast-forwards a job with nothing left to backfill into the funnel), not tablets holding their retention barrier until something notices (`OrderingGenerationReleaseResumesAcrossMasterFailover`). Both task kinds join on abort too: they are registered on the index table, which DROP INDEX (and the DDL rollback of a failed CREATE INDEX) tears down while the job waits, and a join that never arrives would leave the indexed table backfilling forever. An abort mid-activation defers the release to the last activation callback, so a late activation cannot re-arm a released tablet. CREATE INDEX is not delayed: the indexed table's alter request (which is what makes the new permissions visible to clients) is sent before the release wait, so the DDL never waits on a tablet's release ack; it is sent again after the job clears, since only an alter completing with `is_backfilling_` clear picks up an index queued behind this job. - The tserver metadata validator is the backstop for a release that still never arrives, mirroring the retain_delete_markers machinery: tablets with an active generation join its GetBackfillStatus poll at tablet open, and additionally every validation period once the generation has been active longer than `tablet_validator_index_backfill_generation_recheck_sec` (cached status evicted first, so the master is actually asked). The generation is released locally (no Raft) once the master reports a terminal state (`ValidatorReleasesStrandedGeneration`). `IndexStatusPB` gains `BACKFILL_FAILED` (removal-path index permissions map to it) so a failed SKIP_ALL job cannot leave an orphaned invalid index retention-pinned; the retain_delete_markers heal itself stays success-only. - The record is one per tablet (colocated indexes are excluded from SKIP_ALL by eligibility). The admin handler refuses, before Raft, an activation that would displace another index's active generation, and answers a stale release for another table without an op; apply never rejects (a failed replicated apply is fatal) and ignores a mismatched op with a warning. Gating flip: `TabletPeer::CheckReplicateAllowed` now rejects marked writes with *no* active generation (previously deferred) -- a marked write outside a generation would store versions nothing tracks or releases (e.g. a stale chunk retry after the job's terminal state) -- and `WriteQuery` refuses, pre-Raft, a marked batch targeting a table other than the one whose generation the tablet holds. Tests that drive marked writes directly now activate a generation first. `write_id_floor_version` numbering lands as `kIndexBackfillWriteIdFloorVersion = 1`. The mode selector remains hardcoded to CHECK_ALL (#33484), so this whole flow stays production-unreachable; tests drive it through the master-side TEST override. Test Plan: ./yb_build.sh release --cxx-test tablet_peer-test --gtest_filter 'TabletPeerTest.OrderingGenerationChangeMetadataOpSetsBaseFromOwnRaftIndex' ./yb_build.sh release --cxx-test tablet_peer-test --gtest_filter 'TabletPeerTest.FixedHybridTimeWriteRejectedAtOrBelowGenerationBase' ./yb_build.sh release --cxx-test pg_index_backfill-test --gtest_filter 'PgIndexBackfillSkipAllRaftOrdering.OrderingGenerationActivatedAndReleased/0' ./yb_build.sh release --cxx-test pg_index_backfill-test --gtest_filter 'PgIndexBackfillSkipAllActivationFailure.ActivationFailureFailsCreateIndexCleanly/0' ./yb_build.sh release --cxx-test pg_index_backfill-test --gtest_filter 'PgIndexBackfillSkipAllSlowBuild.SplitAllowedDuringBackfill/0' ./yb_build.sh release --cxx-test pg_index_backfill-test --gtest_filter 'PgIndexBackfillMultiMaster.OrderingGenerationReleaseResumesAcrossMasterFailover/0' ./yb_build.sh release --cxx-test pg_index_backfill-test --gtest_filter 'PgIndexBackfillSkipAllStrandedRelease.ValidatorReleasesStrandedGeneration/0' Retrofitted (marked writes now require an active generation): ./yb_build.sh release --cxx-test tablet_peer-test --gtest_filter 'TabletPeerTest.FixedHybridTimeWrite*' ./yb_build.sh release --cxx-test fixed_hybrid_time_write_id-itest Regressions: skip_all e2e + marker canary through the real activation flow, uniq-idx-1 mode tests, DuplicatesExistBeforeBackfill, 3b-i generation tests, 0a/0b abort tests, and drop-path tests for the permissions mapping change (PgIndexBackfillTest.Drop, PgIndexBackfillFastClientTimeout.DropWhileBackfilling).

Commit:fb4de8b
Author:John Meehan
Committer:John Meehan

[#33444] DocDB: Add persisted index-backfill ordering-generation record and tablet clone fence Deferred uniqueness verification scans the marked versions that unique-index backfill wrote at the fixed backfill hybrid time. That scan is only sound against a stable tablet set and a validated write-ID sequence, so each index tablet needs a durable record of the in-flight backfill's ordering state -- one that survives restart, remote bootstrap, and inheritance by split children. The record (`IndexBackfillOrderingGenerationPB`, superblock field 39, persistence following the `cdc_sdk_safe_time` pattern): `active`, the owning index `table_id`, `base_op_index`, `retention_barrier_ht`, and `write_id_floor_version` (numbering pinned in the proto: 1 = the initial floor, 0 = absent/unknown). The full combined shape is defined here; `retention_barrier_ht` is consumed by the verification read fence and identity/activation by the master orchestration, both in later parts. The durable form of an inactive generation is field absence: `LoadFromSuperBlock` clears the in-memory record when the field is absent (so a remote-bootstrap superblock replacement from a source without a generation cannot resurrect a released one), and the setter normalizes inactive input to the default -- the caller contract is "set an active generation or set {}", keeping in-memory and durable state identical. While a generation is active on a tablet: - Marked fixed-hybrid-time writes must carry Raft indexes strictly above `base_op_index` (checked per round in `TabletPeer::CheckReplicateAllowed`, with the write-ID ceiling; InvalidArgument since every replica shares the generation). The base is the activation operation's own Raft index, so an index at or below it cannot belong to this generation -- it would indicate misrouting or replay of a foreign sequence. (Rejecting marked writes when *no* generation is active arrives with the master activation flow; until then marked writes remain production-unreachable.) - Tablet splitting is deliberately NOT fenced here: split children continue the parent's Raft sequence and inherit the record, so per-key write-ID uniqueness holds across a split (proven by `SplitPreservesPerKeyWriteIdUniqueness`). The master fences splits only around the verification scan, for tablet-set stability (verification part). - Tablet cloning is rejected in the same hook (IllegalState, which the master's clone task retries after the release). Clone targets build fresh metadata (`RaftGroupMetadata::CreateNew`), not a superblock copy, so a clone would carry hard-linked snapshot data containing generation-scoped marked write IDs with no generation record describing them and no backfill job tracking their lifecycle. Test Plan: ./yb_build.sh release --cxx-test tablet_peer-test --gtest_filter 'TabletPeerTest.FixedHybridTimeWriteRejectedAtOrBelowGenerationBase' ./yb_build.sh release --cxx-test tablet_peer-test --gtest_filter 'TabletPeerTest.IndexBackfillOrderingGenerationPersistsAcrossReload' ./yb_build.sh release --cxx-test tablet_peer-test --gtest_filter 'TabletPeerTest.IndexBackfillOrderingGenerationClearedBySuperblockReplacement' ./yb_build.sh release --cxx-test tablet_peer-test --gtest_filter 'TabletPeerTest.CloneRejectedWhileOrderingGenerationActive' ./yb_build.sh release --cxx-test fixed_hybrid_time_write_id-itest --gtest_filter 'FixedHybridTimeWriteIdITest.OrderingGenerationSurvivesRemoteBootstrap' ./yb_build.sh release --cxx-test fixed_hybrid_time_write_id-itest --gtest_filter 'FixedHybridTimeWriteIdITest.SplitPreservesPerKeyWriteIdUniqueness' Regressions: 3a fixed-hybrid-time tablet_peer tests and leader-change itest, 0a/0b abort tests, PgIndexBackfillSkipAllRaftOrdering e2e -- all green.

Commit:92d959d
Author:John Meehan
Committer:John Meehan

[#33444] DocDB: Add marked fixed-hybrid-time write path with Raft-index write IDs SKIP_ALL unique-index backfill writes run no uniqueness checks, so distinct candidates sharing one unique-index key at the fixed backfill hybrid time must survive as distinct physical versions -- with the existing positional per-batch write IDs, both land at (key, backfill_ht, 0) and one silently overwrites the other, destroying the evidence deferred verification exists to inspect. Replicated write batches gain a self-describing marker (KeyValueWriteBatchPB.use_raft_index_for_write_id). For a marked batch, every replica derives the storage write ID from the Raft operation index as kBackfillWriteIdFloor | index -- one derivation site in Tablet::ApplyOperation covers leader apply, follower apply, and WAL-replay bootstrap, so replicas converge byte-identically and replay never consults runtime flags. The floor (an alias of kIntraTxnWriteIdLimit, #33499) keeps the marked domain disjoint from every unmarked writer, so a foreground write at exactly the backfill hybrid time cannot collide with a marked version; kBackfillWriteIdIndexMax preserves the reserved kMaxWriteId sentinel. Guards, all failing cleanly before WAL or storage effects: - WriteQuery::MaybeMarkRaftIndexWriteIdBatch marks batches whose ops carry UNIQUE_INDEX_BACKFILL_SKIP_ALL, and fails closed if a SKIP_ALL op is not an eligible batch (non-transactional, all-backfill PGSQL_INSERT, fixed hybrid time) -- writing SKIP_ALL unmarked would reintroduce the silent overwrite. The cluster-capability check (refuse marked writes until every node can replay them) lands with the activation work and its AutoFlag. - ValidateRaftIndexWriteIdBatch rejects marked batches that are transactional, apply external transactions (those records bypass the write-ID override), or contain duplicate unversioned keys (docdb::HasDuplicateWriteKeys, shared with the apply-time check), before Raft submission. A duplicate surviving to apply means corrupt or incompatible replicated data; the batch writer fail-stops with Corruption rather than silently collapsing a candidate. - TabletPeer::CheckReplicateAllowed (the existing per-round leader hook, next to CheckWriteFence) rejects a marked write whose assigned Raft index is above kBackfillWriteIdIndexMax. The hook sees the op after its id is assigned and rejects only that round -- Raft rolls the index back for reuse and fails the one write -- so a bad marked write never fails the foreground writes the Preparer batched with it. InvalidArgument, not IllegalState: every replica shares the index space, so the client must not retry elsewhere. TEST_backfill_write_id_index_max lowers the ceiling for tests. The master never selects SKIP_ALL yet (#33484's hardcoded CHECK_ALL selector), so the marked path remains production-unreachable; tests drive it through the master-side TEST_ysql_index_backfill_unique_check_mode override, exercising the full job-mode plumbing. Test Plan: ./yb_build.sh release --cxx-test doc_hybrid_time-test --gtest_filter 'DocHybridTimeTest.WriteIdEncodedSizeGrowth' ./yb_build.sh release --cxx-test non_transactional_batch_writer-test --gtest_filter 'NonTransactionalBatchWriterTest.SeparateBatchesAtSameFixedHybridTime' ./yb_build.sh release --cxx-test non_transactional_batch_writer-test --gtest_filter 'NonTransactionalBatchWriterTest.SeparateBatchesAtSameFixedHybridTimeWithWriteIdOverride' ./yb_build.sh release --cxx-test non_transactional_batch_writer-test --gtest_filter 'NonTransactionalBatchWriterTest.MarkedAndUnmarkedWritesWithSameHybridTimeAndWriteIdCollide' ./yb_build.sh release --cxx-test non_transactional_batch_writer-test --gtest_filter 'NonTransactionalBatchWriterTest.FloorSeparatesMarkedAndUnmarkedWriteIdDomains' ./yb_build.sh release --cxx-test tablet_peer-test --gtest_filter 'TabletPeerTest.FixedHybridTimeWritesUseRaftIndexWriteId' ./yb_build.sh release --cxx-test tablet_peer-test --gtest_filter 'TabletPeerTest.FixedHybridTimeWriteIdRequiresUniqueUnversionedKeys' ./yb_build.sh release --cxx-test tablet_peer-test --gtest_filter 'TabletPeerTest.FixedHybridTimeWriteIdOverflowRejectedBeforeRaftAppend' ./yb_build.sh release --cxx-test fixed_hybrid_time_write_id-itest --gtest_filter 'FixedHybridTimeWriteIdITest.LeaderChangePreservesDistinctWriteIds' ./yb_build.sh release --cxx-test pg_index_backfill-test --gtest_filter 'PgIndexBackfillSkipAllRaftOrdering.UniqueIndexBackfillAndForegroundCheck/0' ./yb_build.sh release --cxx-test pg_index_backfill-test --gtest_filter 'PgIndexBackfillRaftOrderingActivation.MarkerReachesRaftIndexValidation/0' Regressions: uniq-idx-1 mode tests, DuplicatesExistBeforeBackfill, 0a/0b tablet_peer tests, write-ID cap e2e -- all green.

Commit:c6fec55
Author:John Meehan
Committer:John Meehan

[#33444] DocDB: Add immutable per-job uniqueness-check mode for unique index backfill First slice of deferred uniqueness verification (#33444): a typed `UniqueIndexBackfillMode` (`CHECK_ALL` = existing behavior, `SKIP_ALL` = skip both duplicate-check reads) that the master selects exactly once per backfill job, persists in `BackfillJobPB`, and carries immutably through the whole backfill write path: BackfillJobPB -> BackfillIndexRequestPB -> PgsqlBackfillSpecPB -> BACKFILL INDEX statement -> pggate -> PgsqlWriteRequestPB -> DocDB The mode travels with the work rather than being read from runtime flags, so master failover, retries, and flag changes cannot reinterpret an active job: the `BackfillTable` constructor reuses the persisted value on resume, and tservers act only on the mode carried by each write request. Not production-selectable yet: the selector returns `CHECK_ALL` unconditionally (SKIP_ALL requires the marked-write ordering and verification machinery from later parts of #33444). The capability AutoFlag that must gate production selection is deliberately not introduced here: an AutoFlag promotes once every node merely defines it, so defining it ahead of the behavior it protects would let a release that contains only this slice promote it vacuously. It lands with the activation work, together with its consumers. Missing or unknown mode values resolve to `CHECK_ALL` at every consumer, so older or unaware components keep the fully checked path. Foreground writes never carry the field; their uniqueness enforcement is untouched (the mode is consulted only in the is_backfill branch of PGSQL_INSERT). `TEST_ysql_index_backfill_unique_check_mode` (test-only, check_all / skip_all) is a master-side override of the selected and persisted mode, so tests exercise the full production plumbing. Tservers never consult a flag on the per-row path: they act only on the mode carried by the request. Test Plan: ./yb_build.sh release --cxx-test pg_index_backfill-test --gtest_filter 'PgIndexBackfillTest.UniqueCheckModePlumbedFromMasterToWritePath/0' ./yb_build.sh release --cxx-test pg_index_backfill-test --gtest_filter 'PgIndexBackfillMultiMaster.UniqueCheckModePersistedAcrossMasterFailover/0' ./yb_build.sh release --cxx-test pg_index_backfill-test --gtest_filter 'PgIndexBackfillBlockDoBackfill.DuplicatesExistBeforeBackfill/0'

Commit:f942712
Author:Ella Baron
Committer:Ella Baron

tserver: send a tablet's storage metadata and full compaction status to the master only when changed Both per-tablet sections were resent in every metrics heartbeat. The tserver now keeps what the master confirmed and sends an entry only when it differs; the master sets tablet_metadata_processed in the response once it applied every entry, so an entry that found no replica is resent. One flag, tserver_heartbeat_metrics_send_tablet_metadata_on_change, gates both.

Commit:82aaab6
Author:Ella Baron
Committer:Ella Baron

master: hand tservers the tserver list as TSHeartbeat deltas Every TSHeartbeat response carried the full live tserver list; at 2000 tservers that was 451 MB/s out of the master leader and most of the TSHeartbeat handler time. The master now versions its list (leader term in the high 32 bits, a per-leader change counter in the low 32), each tserver echoes the version it holds, and the response carries only the tservers stamped after it plus tombstones for removals. A term change, an unknown version, one older than the tombstone horizon (master_tserver_list_history_sec, 600) or one ahead of the master's counter gets a full list.

Commit:97857f3
Author:Nikhil Londhe
Committer:Nikhil Londhe

[pg19] merge: master commit 11848c2452a46160d47da9f8f40d4f757ce398fd into pg19 Summary: Merge of master commit 11848c2452a46160d47da9f8f40d4f757ce398fd into pg19. First parent: 903bcd24802d77d68916ac482d489c2dd6a1973a (pg19) Second parent: 11848c2452a46160d47da9f8f40d4f757ce398fd (master) Merge base: 9b3c2cbee421b89c6bf89611b934f1190330999a The range is 281 master commits; 30 of them touch the 70 conflicted files. Upstream imports: One master commit imports upstream fixes: - master commit 40e6790afe4a3212532ec7e7a3e59db2de6ab8ad imports the PG 15.18 and 15.19 security fixes, 19 REL_15_STABLE back-patches: 16fda4df63226fe0d5239fe13cdb5aec580c133f and d8d46710cdbb0b0cfb9a8d614b7e8937297aff58 (raw_connect() in Cluster.pm and its missing include), 3fb66d3022f7bf89143f1452e030d86bd0e1f58e (ProcessStartupPacket() recursion), 9dcfcb92fff82b398f2ba0c03eb7bea9c197aab2 (timingsafe_bcmp()), c95275f18bee5caac175d314cbd997dd68f160f3 (timingsafe_bcmp() in authentication paths), 126a236ba87c749bfcf0b74ef67f9682cd3d4d19 (snprintf() format codes), c3fff3950f215732365c379bb87bc13d453eb4a5 (pg_strftime()), 08c397b0234b09084195930dc41d125eca5d334d (CREATE privilege on the multirange type schema), 17b6083dbce2ba10ea5b5cf0156ec77cfa681eb5 (plan cache invalidation after role changes), 89283e173e9fe7b4ae48e348012a5fb1b190546f (its PGDLLIMPORT marker), c7c82a88c312dc6ff626c037ac12846f1d0ca9c4 (pg_trgm picksplit), 5737110b60afffb90d1178acfaa0fe6881c2f5a9 (ecpg bytea), e43e74756b4c89b1a145ffda0696acfa9f9b492a (amcheck search_path), 8cdbabea74a42dedaa2731658529336e5ef5a5f8 (psql \if lexer state), 3fdcfa8f7939783adff360f515ef7b37f1659c17 (psql COPY FROM STDIN skip after failure), ed0a2b70709ac215f3c669d607f043e570f0cbf5 (psql COPY FROM STDIN counting), 3b925133b54df3fc89bcec47c525b6a7386dcc88 (ascii() multibyte reads), 472ef7e4a1d755723adf4c0b24851d822c0ef9f7 (PGP encryption error handling) and 953be116c978bf517e7d5e4498d75e229e6c5368 (pgcrypto ignore-cipher-failure option). On pg19 (checked against pg19's upstream base 90630ec42939d074ecc7b6b959b48252eed32646): - 9dcfcb92fff82b398f2ba0c03eb7bea9c197aab2 as upstream PG commit 09be39112654c3f158098fdb5f820143c0330763. - 16fda4df63226fe0d5239fe13cdb5aec580c133f and d8d46710cdbb0b0cfb9a8d614b7e8937297aff58 as upstream PG commit 85ec945b7880931cb506392101cb0b00209b78ba. Not on pg19: the other 16. Their upstream master versions landed after pg19's upstream base: 3fb66d3022f7bf89143f1452e030d86bd0e1f58e as b63f25bddfebc67b1e78f86341a6aecb0e9fe576, c95275f18bee5caac175d314cbd997dd68f160f3 as 5924e256c499c010dd369984498c8d5a4ee17894, 126a236ba87c749bfcf0b74ef67f9682cd3d4d19 as 76ab76f875a85d51da91dca18d64a6b717533ad9, c3fff3950f215732365c379bb87bc13d453eb4a5 as ec8ded4b327f9d121811f43bf0177d0f289c3949, 08c397b0234b09084195930dc41d125eca5d334d as 4793fc41f82893b059bd59739bdcbe3d58c4df7e (2026-05-11), and 17b6083dbce2ba10ea5b5cf0156ec77cfa681eb5 as ffca23839ccaaf8aed5e0729cb041b53f981f04c, c7c82a88c312dc6ff626c037ac12846f1d0ca9c4 as 66c2dbb20445195f92225708a930112ee9e96bf4, 5737110b60afffb90d1178acfaa0fe6881c2f5a9 as c7da85fec96666b038fdf83c81dbe1e1573a9e60, e43e74756b4c89b1a145ffda0696acfa9f9b492a as 0fbf785b13e6d8db093adc98fcbf8fd6d1642b80, 8cdbabea74a42dedaa2731658529336e5ef5a5f8 as cf754f7414ba6159733ee9cdffde90a3cee3c32c, 3fdcfa8f7939783adff360f515ef7b37f1659c17 as 3045a25ba813b554e23c723ebe94cf504feb370c, 3b925133b54df3fc89bcec47c525b6a7386dcc88 as a16c31d39c8bc1ed979b1b778238e3e1e72a311f, 472ef7e4a1d755723adf4c0b24851d822c0ef9f7 as d0ecee6de9afec143a4766a72df0d3018a06d662, 953be116c978bf517e7d5e4498d75e229e6c5368 as 49e795414848bbd1f5122f64c0645827d8d90313 (2026-08-10), 89283e173e9fe7b4ae48e348012a5fb1b190546f as e6bccd5abed38f93f972469ccffcc99cfc801e66 and ed0a2b70709ac215f3c669d607f043e570f0cbf5 as 0d7baa209a302bba358bc4f6415a08339dae547f (2026-08-14). They come to pg19 with the next upstream merge. master commit 40e6790afe4a3212532ec7e7a3e59db2de6ab8ad is left out: the 23 conflicted files whose only master change it is (listed below) and 67 auto-merged files take YB pg19's content; pgp-encrypt_1.out, src/test/postmaster/t/004_negotiate.pl and src/test/regress/sql/generated.sql are not added; its YB test changes (yb.orig.rowsecurity, yb.orig.pgp-fips-cipher, yb.port.multirangetypes and the COPY "\." lines in yb.port.copy2, yb.port.generated, yb.port.rowsecurity, yb.port.privileges, yb.port.alter_table, yb.port.copy and yb.orig.feature_copyselect) are left out with the fixes. Files whose only master change in the range is the left-out import take YB pg19's content, apart from the files in the next section; files with other master commits (within the range) keep those commits' changes. Files with a left-out import and other master changes: Each of these files is touched by the left-out import and by another master commit in the range, or edited here for one. It drops the import's change and keeps the other change, so its content is YB pg19's content plus that change. - src/postgres/src/backend/postmaster/postmaster.c: - left out: ProcessStartupPacket()'s SSL/GSS negotiation retry loop, via master commit 40e6790afe4a3212532ec7e7a3e59db2de6ab8ad. - kept: two edits for master commit b5132ae1036a141056ea3972853c18b721a5bdaf's new backend type YB_OSC_BACKFILL_BACKEND (see Other changes). - src/postgres/src/test/regress/sql/yb.orig.feature_copy.sql: - left out: four "\." lines after COPY statements that fail, the YB test change for the psql COPY FROM STDIN fix, via master commit 40e6790afe4a3212532ec7e7a3e59db2de6ab8ad. - kept: master commit 5b97ed225f9bf1b45e336befb7ee802450cd7ddf (upsert-mode guard test H on a partitioned table). Attribution key: - "master commit ..." = YB master-branch commit, i.e. what independently changed on master after the merge base. - "upstream PG commit ..." = upstream PostgreSQL commit whose change is on the pg19 branch via the initial merge. - "YB pg19 commit ..." = YB-authored commit on the pg19 branch. Conflicted files: - src/postgres/src/backend/utils/misc/guc.c: - the ConfigureNames* tables and the GUC functions: - upstream PG commit 63599896545c7869f7dd28cd593e8b548983d613 generates the GUC tables from guc_parameters.dat, and YB pg19 commit 11876241ac840ae4cef033971a1082399ff687bd moved the YB GUCs there, so every hunk pairs master's table edits with unrelated pg19 code. - master commits 679286558f53a6dc5ac327d02e27a1ddbb3dcf1f, 7659e8b4f189e1afe17a3050a72a47f790e5ec3d, 510da0f5d3745457fb0bfadde804ff0199d96c32, 4190b8a74f48522d4b97d6f7416af67cfb0265bb, 92ec0c0517dd276627133ee4e0a8c1d55084c17d, e4089230d61bad87addd5ecd5e8249d8f13326d7, 53cc8bf2f0f8f2beb546e9ec550f86cae8f3f7b8, c192ba7f8532bd0ad04b12a4618b6ec40a77f801, 73d72e0fcee40bd268a29d9f07561222631e9eaa, 5f981e89455d06e3fe630ed1a6c2b7f497106dec, 6dd04a0a5ea9cb1791b5facdcc03001e475f78ce, 2fe29a63f0d339a19822c3b9e52547dbe2c4aafb, 387582a1ca34bc9d576b8df65504638d3f43bf21 and 70e0cc61c69a7fb82a1d3488bea1aa30c12aef93 add or change GUC entries and the GUC code. - Resolved by keeping pg19's side of every hunk and porting master's changes. guc_parameters.dat gets 8 new GUCs (yb_conn_mgr_replication_origin_permanent_sticky, yb_enable_colocated_tables_with_tablespaces, yb_enable_object_locking_infra, yb_tcmalloc_drain_cpu_caches_after_statement, yb_tcmalloc_release_after_statement, yb_tcmalloc_skip_subrelease_long_interval, yb_tcmalloc_skip_subrelease_short_interval and yb_xcluster_invalid_index_ddl_check) and 11 changed ones (enable_object_locking_infra, now a deprecated no-op on yb_deprecated_enable_object_locking_infra, yb_enable_stale_catalog_prefetch, yb_enable_pg_stat_statements_docdb_metrics, yb_startup_free_memory_release_threshold, yb_max_merge_scan_streams, seed, default_table_access_method, default_text_search_config, yb_conn_mgr_client_hostname, yb_ycm_internal_client_addr and yb_ycm_internal_client_port). master commit 7659e8b4f189e1afe17a3050a72a47f790e5ec3d's static yb_deprecated_enable_object_locking_infra goes to guc_tables.c, next to the other YB statics. master commit 679286558f53a6dc5ac327d02e27a1ddbb3dcf1f's assign hook yb_assign_conn_mgr_replication_origin_permanent_sticky() is non-static and declared in guc_hooks.h, as the generated tables need. master's code changes are re-applied by hand: GUC_YB_NO_CONN_MGR_REPORT in yb_should_report_guc() and AtEOXact_GUC(); the yb_read_time reset check of master commit 2fe29a63f0d339a19822c3b9e52547dbe2c4aafb, placed next to the GUC_NO_RESET check as master's TODO(PG16 merge) asks, comparing the GUC name because yb_read_time_string is static in guc_tables.c; the custom-sticky latch removed; the SUSET and GUC_YB_CONN_MGR_STICKY stickiness at the end of set_config_with_handle(); and GUC_YB_CONN_MGR_STICKY in DefineCustomStringVariable(). - src/postgres/src/include/utils/guc.h: - GUC flag bits: - upstream PG commit 7ac918ada0037fc1101456271b3e7d0982d6a375 widened the unit masks, and YB pg19 commit 16fb8e1be8c9fc88bb73172297f4419d09354cd0 put GUC_YB_CUSTOM_STICKY on 0x00010000. - master commits 6dd04a0a5ea9cb1791b5facdcc03001e475f78ce and 5f981e89455d06e3fe630ed1a6c2b7f497106dec rename GUC_YB_CUSTOM_STICKY to GUC_YB_CONN_MGR_STICKY (0x40000000) and add GUC_YB_NO_CONN_MGR_REPORT (0x20000000). - Resolved by renaming pg19's flag to GUC_YB_CONN_MGR_STICKY on 0x00010000 and putting GUC_YB_NO_CONN_MGR_REPORT on the free bit 0x00020000. On pg19, 0x20000000 is GUC_UNIT_S. - src/postgres/src/backend/utils/adt/float.c: - setseed(): - upstream PG commit e6341323a8da64b18e9af3e75a4578230702d61c moved setseed() to pseudorandomfuncs.c. - master commit 5f981e89455d06e3fe630ed1a6c2b7f497106dec makes setseed() mark the Connection Manager connection sticky. - Resolved by taking pg19's and adding master's stickiness block and YB includes to setseed() in pseudorandomfuncs.c. - src/postgres/src/backend/utils/init/miscinit.c: - GetBackendTypeDesc(): - upstream PG commit dbf8cfb4f02eb9ec5525de1761675f9babfd30e3 builds the descriptions from proctypelist.h. - master commit b5132ae1036a141056ea3972853c18b721a5bdaf adds YB_OSC_BACKFILL_BACKEND to the switch. - Resolved by taking pg19's and adding a PG_PROCTYPE entry for YB_OSC_BACKFILL_BACKEND to proctypelist.h, as YB pg19 commit 57dac1c319f6d04ce3ee10c838d01f1824548d96 does for the other YB backends. - src/postgres/src/backend/Makefile: - SUBDIRS: - upstream PG commit 35739b87dcfef9fc0186aca659f262746fecd778 adds archive, and upstream PG commit 9adf32da6b48e423e632ac24d6c634021e1ab154 lists one directory per line. - master commit b5132ae1036a141056ea3972853c18b721a5bdaf adds yb_online_schema_change. - Resolved by adding yb_online_schema_change after ybgate in pg19's list. - src/postgres/src/backend/commands/tablecmds.c: - ATExecAlterColumnType() dependency loop: - upstream PG commit d4e66a39eb96ca514e3f49c85cf0b4b6f138854e moved the loop to RememberAllDependentForRebuilding(), and upstream PG commit 5d06e99a3cfc23bbc217b4d78b8c070ad52f720e added its subtype argument. - master commit 46e094f899d6c7de2aaad0b5920c30b1e67c4076 calls YbCheckAlterTypeOfVectorIndexColumn() in the loop. - Resolved by taking pg19's and adding the call in RememberAllDependentForRebuilding(), only for AT_AlterColumnType, to keep master's "alter type" scope. - ATExecAttachPartitionIdx(): - upstream PG commit 14e87ffa5c543b5f30ead7413084c25f7735039f adds a verifyPartitionIndexNotNull() call at the same spot. - master commit 4190b8a74f48522d4b97d6f7416af67cfb0265bb calls YbCheckAttachedIndexIsValid(partIdx). - Resolved by taking pg19's and adding the call before IndexSetParentIndex() in ATExecAttachPartitionIdx(). - src/postgres/src/backend/nodes/copyfuncs.c, equalfuncs.c, outfuncs.c and readfuncs.c: - hand-written node support: - upstream PG commits 964d01ae90c314eb31132c2e7712d5d9fc237331 and 2be87f092a2ac786264b2020797aafa837de5a8e generate node support from the node headers. - master commit c192ba7f8532bd0ad04b12a4618b6ec40a77f801 adds hand-written copy, equal, out and read code for Scan.ybOptions, YbMergeScanInfo, YbMergeScanStreamColInfo and YbSortInfo. - Resolved by taking pg19's. gen_node_support.pl generates the same code from master's auto-merged plannodes.h fields, so parallel workers receive the hint options. - src/postgres/src/include/nodes/pathnodes.h: - end of IndexOptInfo: - upstream PG commit 1e3b5edb8eb236f16529ca8dae917996ad1f433e named the typedef at its closing brace. - master commit c192ba7f8532bd0ad04b12a4618b6ec40a77f801 adds yb_options. - Resolved by adding master's member before pg19's closing. - src/postgres/src/backend/optimizer/path/costsize.c: - YB includes: - YB pg19 initial merge 4c01ae04c6d8e5487679d2079e44a067a66febcd includes optimizer/planmain.h. - master commit c192ba7f8532bd0ad04b12a4618b6ec40a77f801 includes optimizer/planner.h for ybGetValueForMergeOption(). - Resolved by keeping both. - cost_index(), disable check: - upstream PG commit e22253467942fdb100087787c3e1e3a8620c54b2 marks disabled paths with disabled_nodes instead of adding disable_cost. - upstream PG commit 4020b370f214315b8c10430301898ac21658143f computes disabled_nodes from pgs_mask instead of enable_indexscan. - master commit c192ba7f8532bd0ad04b12a4618b6ec40a77f801 adds disable_cost when ybShouldPenalizeMergeOption(path) is true. - Resolved by adding "&& !ybShouldPenalizeMergeOption(path)" to the condition that leaves disabled_nodes at 0, so a penalized path counts one disabled node, as master adds disable_cost once. - yb_cost_index(), disable check: - YB pg19 commit caa39b2fbcc736a88a7f54ced92ce5f15ebeef16 moved the enable_indexscan check to the pgs_mask disabled_nodes block at the top of yb_cost_index(). - master commit c192ba7f8532bd0ad04b12a4618b6ec40a77f801 adds ybShouldPenalizeMergeOption() to the old disable_cost check. - Resolved by taking pg19's and adding master's condition to that block. create_index_path() sets merge_scan_stream_cols before the call. - src/postgres/src/backend/optimizer/path/allpaths.c: - standard_join_search(), Leading-hint comment: - YB pg19 commit eee1e0a213efa8241049e51bf8c0d9ccf22168ff adds the T_GatherPath/T_GatherMergePath exception. - master commit 6961a55e34bcd99aa149fa4ea731e3dbf23035dd adds the IS_DUMMY_REL exception. - Resolved by naming both exceptions in the comment; the code auto-merged. - src/postgres/src/backend/optimizer/plan/planner.c: - includes before the YB block: - upstream PG commit dbbca2cf299b81299112ca6ada671a36235ec008 dropped utils/syscache.h. - master commit c192ba7f8532bd0ad04b12a4618b6ec40a77f801 drops the blank line after it. - Resolved by taking pg19's. master's new YB includes auto-merged. - src/postgres/src/backend/optimizer/plan/createplan.c: - YB includes: - YB pg19 initial merge 4c01ae04c6d8e5487679d2079e44a067a66febcd includes parser/parse_relation.h. - master commit 5bf305bbbe625ce97783db823779dee0781f86e6 includes parser/parse_coerce.h for IsBinaryCoercible(). - Resolved by keeping both. - src/postgres/third-party-extensions/pg_hint_plan/pg_hint_plan.c: - YB pg19 commit 18fa59c1ce3389e958e0d754d5c8bcd316287809 subtree-merges pg_hint_plan's PG19 version and re-applies the YB changes. That version includes upstream pg_hint_plan commits ec1d8d6b50e581ce1bb14c62ada4a62192447593 (pgindent layout) and c400184904909880c0a4736e0bd053fe8b780829 (build_simple_rel_hook() instead of get_relation_info_hook()), and set_parent_index_infos(), so diff3 pairs most of master's hunks with moved code. - master commit c192ba7f8532bd0ad04b12a4618b6ec40a77f801 adds per-index merge options: a ybOptions field in struct Hint, a yb_options_list parameter on parse_parentheses(), ybApplyIndexOptionList(), ybApplyPerIndexOptions() and ybSetIndexOptions(), their calls in restrict_indexes(), and ybIndexName in the parent index info. - Resolved by taking pg19's file and adding every master hunk by hand. Every parse_parentheses() caller passes the new parameter, including the pg19-only DisableIndexHintParse() ((List **) NULL), and the pg19-only DisableIndexHintCreate() also sets hint->base.ybOptions = NIL. master's parent_index_info->ybIndexName assignment goes into pg19's set_parent_index_infos(), which replaced the setup_hint_enforcement() code master edits. - src/postgres/third-party-extensions/pg_hint_plan/core.c (deleted by pg19, modified by master): - YB pg19 commit 18fa59c1ce3389e958e0d754d5c8bcd316287809 deleted core.c, so pg19's pg_hint_plan has no copy of standard_join_search(). - master commit 6961a55e34bcd99aa149fa4ea731e3dbf23035dd changes the copy. - Resolved by deleting it; master's change is in allpaths.c. - src/postgres/src/test/regress/expected/yb.orig.hints.out: - EXPLAIN output and "Generated hints" lines: - YB pg19 commits 18fa59c1ce3389e958e0d754d5c8bcd316287809, 4caa3979800af7350c6fdc5e00c21dff3f434ba2, ce91882bf699a2a987df250e79e1ab833eda0a04, 57b375fc8cc8a5a8dede185625b78cde1384b777 and 5f74bb76287a6c81fe580d025a1429ba6d8c466d rebless the plans for PG19. - master commit c192ba7f8532bd0ad04b12a4618b6ec40a77f801 adds Set(yb_max_merge_scan_streams 64) before Set(geqo false) in every generated hint string, uses single spaces between sub-block hints, and adds a yb_merge_scan_hint section and a noSeqScan block under yb_enable_cbo. - Resolved by keeping pg19's plans with master's hint-string changes re-applied (the same transform reproduces master's golden from the merge base exactly), and taking master's new sections. See Other changes for their PG19 output. - src/postgres/contrib/pg_stat_statements/Makefile, pg_stat_statements.c and pg_stat_statements.control: - YB pg19 commit 945ea71848edb621ae23a3c7e9dcb5029dcb0567 adds version 1.13-yb-1.0: PG19's pg_stat_statements 1.13 plus YB's columns, 72 output columns. - master commit e4089230d61bad87addd5ecd5e8249d8f13326d7 adds docdb_bloom_filter_useful and docdb_bloom_filter_checked as version 1.10-yb-2.2 (63 columns). On pg19 that version sorts before 1.13-yb-1.0. - Resolved by adding the two columns to 1.13-yb-1.0 itself (74 columns), since no release has shipped 1.13-yb-1.0, and keeping master's 1.10-yb-2.2. pgssVersion gets YB_V2_2_PGSS_V1_10 after YB_V2_1 (before the upstream PGSS_V1_11). The C entry point yb_2_2_pg_stat_statements_1_10, the column-count defines (YB_PG_STAT_STATEMENTS_COLS_V1_13 is now 74), the natts switch, the histogram condition and the output Assert cover the new version. The bloom-filter counters are added on pg19's entry-> alias (upstream PG commit 8928817769de0d81758bc760333d3056c67b63c1 dropped the volatile e pointer). default_version stays '1.13-yb-1.0'. pg_stat_statements--1.13-yb-1.0.sql and --1.10-yb-2.1--1.13-yb-1.0.sql get the two columns, the new --1.10-yb-2.2--1.13-yb-1.0.sql updates master's new version, and DATA lists it with master's 1.10-yb-2.2 scripts. yb.orig.oldextversions's expected output shows the two columns after the update to 1.13-yb-1.0. - src/postgres/src/test/regress/expected/yb.port.rules.out: - pg_stat_statements and pg_stat_statements_with_server_uuid views: - YB pg19 commit 411b9a503c8e8399a38e254820abb7edc8dac9eb reblessed them for PG19's deparse. - master commit e4089230d61bad87addd5ecd5e8249d8f13326d7 adds the two bloom-filter columns. - Resolved by adding master's columns to pg19's deparse. - src/postgres/third-party-extensions/pgaudit/sql/yb.port.pgaudit.sql and expected/yb.port.pgaudit.out: - ALTER EXTENSION pg_stat_statements UPDATE TO: - YB pg19 commit 921156450d4888d00c60822095acc17b76624b8f updates to 1.13-yb-1.0, and pgaudit 18 logs the statement without ';'. - master commit e4089230d61bad87addd5ecd5e8249d8f13326d7 updates to 1.10-yb-2.2. - Resolved by taking pg19's: 1.13-yb-1.0 is still the latest version. - src/postgres/yb-extensions/yb_xcluster_ddl_replication/expected/create_colocated_table.out, create_drop_index.out, create_drop_table.out, materialized_views.out, mixed_temporary_and_persistent.out, nested_ddls.out, table_rewrite.out and truncate.out: - captured DDL rows (yb_data): - YB pg19 commit 3132bcd822a921f9fcc1af88c6b9fb28559e5e53 stops capturing standard_conforming_strings (upstream PG commit 45762084545ec14dbbe66ace1d69d7e89f8978ac). - master commit a28919f11a97f9e2d3759d3832886c6e39f3bb3a adds the new yb_alter_type_reuse_expr_and_partial_indexes variable to every captured row. - Resolved by taking master's rows without standard_conforming_strings and recomputing the header widths. The same script reproduces pg19's golden from the merge base byte for byte. - src/postgres/src/backend/replication/logical/origin.c: - yb_replication_origin_session_setup_shared(): - upstream PG commit 8f1e2dfe033e9a3236265c3b9f61bd226f4a8f54 renamed replorigin_session_origin to replorigin_xact_state.origin. - master commit 679286558f53a6dc5ac327d02e27a1ddbb3dcf1f calls YbConnMgrLatchIfReplicationOriginExcluded() instead of increment_sticky_object_count(). - Resolved by combining both. - src/postgres/src/backend/replication/slot.c: - ReplicationSlotAcquire(), YB slot allocation: - YB pg19 commit 3ab03c3d91f6e4e8dce79b796735fac94854726a comments that the fields must be 0. - master commit 6066d76d67e9d4b900d1f0e7a2144d057d579e5c uses a static yb_acquired_slot with MemSet, fixing a use-after-free on error. - Resolved by keeping pg19's comment over master's code. - ReplicationSlotAcquire(), free-entry search bound and full array: - upstream PG commit e76d8c749c3152657711ed733f0aea61c0e36a91 reserves the max_repack_replication_slots tail for REPACK; YB pg19 commit 41f97dd09a2718f31a210e4424bd434844646d9c adds the full-array error. - master commit 6066d76d67e9d4b900d1f0e7a2144d057d579e5c loops to max_replication_slots (pg_wrapper now sets that GUC from the gflag), releases stale entries, checks for concurrent creation and changes the error hint. - Resolved by taking master's. - ReplicationSlotDrop(), exclusive-lock path: - upstream PG commit f41d8468ddea34170fe19fdc17b5a247e7d3ac78 changed ReplicationSlotAcquire()'s arguments. - master commit 6066d76d67e9d4b900d1f0e7a2144d057d579e5c drops a slot under the lock only, with a PG_TRY release, without taking an array entry. - Resolved by taking master's. - src/postgres/src/backend/replication/logical/yb_virtual_wal_client.c: - YBCDestroyVirtualWal(): - YB pg19 commit 379125327554eb69bed957e668f31c5667fe89a0 leaves the virtual WAL contexts. - master commit a910a67ce40fe8f6b5651816d3c7097a224083d6 frees the aborted-subtransaction set. - Resolved by keeping both, pg19's first, so the set is freed before the contexts are deleted. - java/yb-pgsql/src/test/java/org/yb/pgsql/TestPgReplicationSlotWithTxnDdl.java: - new tests at the end of the class: - YB pg19 commit dceabb8a057ae2549f5283ab9bfa44f6054749b4 adds publication-rename tests. - master commit 863fe3fbf620c08947162b6faba0242d97ef9a84 adds DROP/RENAME COLUMN tests. - Resolved by keeping both, pg19's first. - java/yb-ysql-conn-mgr/src/test/java/org/yb/ysqlconnmgr/BaseYsqlConnMgr.java: - ysql_conn_mgr_superuser_sticky field: - YB pg19 commit b658cc7ebad82d5c1d1cccbc09fe46a106c2a0c7 keeps the field static and resets it in @After. - master commit b4ff016ba2f20cf1a43fab67b33430926c08f2e9 makes the field non-static. - Resolved by taking pg19's: both fix the same cross-method leak. - src/postgres/src/bin/pg_dump/pg_backup_archiver.c: - _printTocEntry(), YB per-statement TOC block: - upstream PG commit 0f129052150487afab9fe64889c5bf7534f7bbc3 adds the --transaction-size semicolon loop, which diff3 paired with the old YB block. - master commit e994cd5dd1048d71e7dea81552bfea00667125fd fixes a leak in the YB block. - Resolved by taking pg19's and porting master's fix into pg19's moved YB block. - src/postgres/src/backend/executor/nodeYbBitmapIndexscan.c: - ExecEndYbBitmapIndexScan(): - YB pg19 commit eab5245fd0933de14ca682bece232adf2227ffc7 removes the dead NOT_USED ExprContext block. - master commit 1e767152ed4dbe295d18413d08fd02efd9a9a268 removes the yb_idx_pushdown pfree; pushdown expressions now live in biss_pushdown_cxt. - Resolved by taking neither. - managed/src/main/resources/version.txt: - version line: - YB pg19 commit 478a4e5377366a4a807d27cef65aba8bbaa5f95d sets 2.31.0.1900-b0, the pg19 Jenkins version. - master commit 93771dac07c24af8374dfb90b22b6ee3f858e5e3 sets 2.33.0.0. - Resolved by taking pg19's. - src/postgres/src/test/regress/expected/yb.orig.alter_table_rewrite.out: - typmod and expression index tests: - YB pg19 merge fa50619c7df560f780ac36af0367db9a223dae33 added PG19's LINE 1 caret (upstream PG commit ba75f717526cbaa9000b546aac456e43d03aaf53) to the INCLUDE (lower(b)) error. - master commit 05f14e65379922fb4b7d488e0fa614acab5f1fd1 moves the block to the new yb.orig.alter_table_typmod_index. - Resolved by taking master's and carrying the caret into yb.orig.alter_table_typmod_index.out (see Other changes). - src/yb/integration-tests/upgrade-tests/builds.xml (both sides rewrote the end of the file): - YB pg19 commit ff8bb22fbb99b665eb2acaa6c80b475d1900feee adds 2026.1.4.0 with debug URLs. - master commit 51dcfedd18604c21c73b61b56d25774a97161863 reads only release builds plus an optional linux_fastdebug_x86 build; master commit 15132ac06656320bce82302bc89c7efbc492597a adds 2026.1.2.0. - Resolved by taking master's file and adding pg19's 2026.1.4.0 entry without its debug URLs. - src/yb/integration-tests/upgrade-tests/upgrade_test_base.h and upgrade_test_base.cc: - YB pg19 commits ff8bb22fbb99b665eb2acaa6c80b475d1900feee and 351f78803310289ef2b79472c2c5168ca6dc9c9a add kBuild_2026_1_4_0, <tuple> and the IsUpgradeSupported() comment. - master commits 15132ac06656320bce82302bc89c7efbc492597a, dd6de24c448d58313d7ae677b59fdd21cc3f6c5a and 07b109cc69e7594b71484a78a36844779d7e38f4 add kBuild_2026_1_2_0 and the extraction-cache includes, and drop WaitForAutoFlagApply(). - Resolved by keeping both build constants and both include sets, dropping WaitForAutoFlagApply() (no caller is left) and keeping pg19's comment. - src/yb/integration-tests/upgrade-tests/ysql_major_extension_upgrade-test.cc: - YB pg19 commits fa88e2751284105520986d03ae51bdc45b4eb624, 3535b518a58dfdfe5fb289584af2b8a1bf2b4f33 and 175a2a2ac74443ad95609a729a943cfaacb31833 adapt each extension test to a PG15 source. - YB pg19 commit 3552a8beebc7ce4f239abeaf2385f292b5bf867c turns the PgHintPlan and Anon YB_TODO_PG19MERGE markers into TODO(#31987) and TODO(#37181). - master commit 8f8a2bcfba749dd6f2937ede9bf4567546bbd83d runs all extensions in one upgrade (ExtensionUpgradeCase) and adds IncompatibleExtensions. - Resolved by taking master's layout with pg19's cases: pg19's SetUpOptions (preload only passwordcheck); HStore and HypoPG as UnchangedCase; FuzzyStrMatch as a NewObjectCase on daitch_mokotoff() plus a soundex() check; PgStatStatements with pg19's stats_since query; PostgresFdw as a NewObjectCase on postgres_fdw_get_connections(true) with CREATE EXTENSION IF NOT EXISTS; Orafce with pg19's two databases, sys_extract_utc() and the orafce_set_umask role check; PlPgsql with pg19's DROP EXTENSION. The PgHintPlan and Anon cases stay, but out of Upgrade, with pg19's TODO(#31987) and TODO(#37181) reasons, because one failing case fails the shared upgrade. master's IncompatibleExtensions is dropped and pg19's PgStatMonitor test kept. - src/yb/integration-tests/upgrade-tests/ysql_major_upgrade_expression_pushdown-test.cc: - YB pg19 commit ff8bb22fbb99b665eb2acaa6c80b475d1900feee tests PG18/PG19 functions against a PG15 source. - master commit 8f8a2bcfba749dd6f2937ede9bf4567546bbd83d restructures the test into PushdownCase objects. - Resolved by taking master's layout with NewPg19FunctionsCase (to_bin, uuid_extract_version, tid_block, erf, gamma), casefold() instead of regexp_like()/regexp_substr(), and pg19's PG15/PG19 labels. - The conflicted files whose only master change is the left-out import (see Upstream imports), resolved by taking pg19's content exactly: src/lint/upstream_repositories.csv (keeps pg19's yb-pg19 fork pin) src/postgres/configure.ac src/postgres/src/include/pg_config.h.in src/postgres/src/include/port.h src/postgres/src/backend/libpq/auth-scram.c src/postgres/src/backend/libpq/auth.c src/postgres/src/backend/libpq/crypt.c src/postgres/src/interfaces/libpq/fe-auth-scram.c src/postgres/src/test/Makefile src/postgres/src/test/postmaster/Makefile (add/add) src/postgres/src/test/perl/PostgreSQL/Test/Cluster.pm src/postgres/src/backend/utils/adt/acl.c src/postgres/src/backend/utils/cache/plancache.c src/postgres/contrib/amcheck/verify_nbtree.c src/postgres/contrib/amcheck/expected/check_btree.out src/postgres/src/bin/psql/common.c src/postgres/src/bin/psql/t/001_basic.pl src/postgres/src/fe_utils/psqlscan.l src/postgres/src/test/regress/sql/copy2.sql src/postgres/src/test/regress/sql/multirangetypes.sql src/postgres/src/test/regress/expected/multirangetypes.out src/postgres/src/tools/msvc/Mkvcbuild.pm (modify/delete; kept deleted, upstream PG commit 1301c80b2167feb658a738fa4ceb1c23d0991e23 removed the MSVC build) src/postgres/src/tools/msvc/Solution.pm (modify/delete; kept deleted) Other changes: - Auto-merged changes reverted to pg19 (left-out import): src/postgres/configure src/postgres/contrib/amcheck/sql/check_btree.sql src/postgres/contrib/pg_trgm/trgm_gist.c src/postgres/contrib/pgcrypto/expected/pgp-decrypt.out src/postgres/contrib/pgcrypto/expected/pgp-decrypt_1.out src/postgres/contrib/pgcrypto/expected/pgp-info.out src/postgres/contrib/pgcrypto/expected/pgp-pubkey-decrypt.out src/postgres/contrib/pgcrypto/expected/pgp-pubkey-decrypt_1.out src/postgres/contrib/pgcrypto/expected/yb.orig.pgp-fips-cipher.out src/postgres/contrib/pgcrypto/expected/yb.orig.pgp-fips-cipher_1.out src/postgres/contrib/pgcrypto/pgp-cfb.c src/postgres/contrib/pgcrypto/pgp-decrypt.c src/postgres/contrib/pgcrypto/pgp-encrypt.c src/postgres/contrib/pgcrypto/pgp-pgsql.c src/postgres/contrib/pgcrypto/pgp-pubkey.c src/postgres/contrib/pgcrypto/pgp.c src/postgres/contrib/pgcrypto/pgp.h src/postgres/contrib/pgcrypto/sql/pgp-decrypt.sql src/postgres/contrib/pgcrypto/sql/pgp-pubkey-decrypt.sql src/postgres/contrib/pgcrypto/sql/yb.orig.pgp-fips-cipher.sql src/postgres/doc/src/sgml/pgcrypto.sgml src/postgres/src/backend/commands/typecmds.c src/postgres/src/backend/utils/adt/oracle_compat.c src/postgres/src/backend/utils/adt/timestamp.c src/postgres/src/bin/psql/command.c src/postgres/src/bin/psql/common.h src/postgres/src/bin/psql/copy.c src/postgres/src/bin/psql/mainloop.c src/postgres/src/bin/psql/psqlscanslash.h src/postgres/src/bin/psql/psqlscanslash.l src/postgres/src/bin/psql/startup.c src/postgres/src/fe_utils/conditional.c src/postgres/src/include/fe_utils/conditional.h src/postgres/src/include/fe_utils/psqlscan.h src/postgres/src/include/fe_utils/psqlscan_int.h src/postgres/src/include/utils/acl.h src/postgres/src/interfaces/ecpg/ecpglib/data.c src/postgres/src/interfaces/ecpg/ecpglib/error.c src/postgres/src/interfaces/ecpg/include/ecpgerrno.h src/postgres/src/interfaces/ecpg/test/expected/sql-bytea.c src/postgres/src/interfaces/ecpg/test/expected/sql-bytea.stderr src/postgres/src/interfaces/ecpg/test/sql/bytea.pgc src/postgres/src/port/timingsafe_bcmp.c src/postgres/src/test/postmaster/.gitignore src/postgres/src/test/postmaster/README src/postgres/src/test/regress/expected/encoding.out src/postgres/src/test/regress/expected/psql.out src/postgres/src/test/regress/expected/yb.orig.rowsecurity.out src/postgres/src/test/regress/expected/yb.port.multirangetypes.out src/postgres/src/test/regress/sql/alter_table.sql src/postgres/src/test/regress/sql/copy.sql src/postgres/src/test/regress/sql/copyselect.sql src/postgres/src/test/regress/sql/encoding.sql src/postgres/src/test/regress/sql/privileges.sql src/postgres/src/test/regress/sql/psql.sql src/postgres/src/test/regress/sql/rowsecurity.sql src/postgres/src/test/regress/sql/yb.orig.feature_copyselect.sql src/postgres/src/test/regress/sql/yb.orig.rowsecurity.sql src/postgres/src/test/regress/sql/yb.port.alter_table.sql src/postgres/src/test/regress/sql/yb.port.copy.sql src/postgres/src/test/regress/sql/yb.port.copy2.sql src/postgres/src/test/regress/sql/yb.port.generated.sql src/postgres/src/test/regress/sql/yb.port.multirangetypes.sql src/postgres/src/test/regress/sql/yb.port.privileges.sql src/postgres/src/test/regress/sql/yb.port.rowsecurity.sql src/postgres/src/timezone/strftime.c src/postgres/src/tools/pgindent/typedefs.list src/postgres/contrib/pgcrypto/expected/pgp-encrypt_1.out (not added) src/postgres/src/test/postmaster/t/004_negotiate.pl (not added) src/postgres/src/test/regress/sql/generated.sql (not added) - src/postgres/src/test/regress/sql/generated_stored.sql: - git's rename detection carried the four COPY "\." lines that master commit 40e6790afe4a3212532ec7e7a3e59db2de6ab8ad adds to generated.sql into pg19's renamed generated_stored.sql. Reverted to YB pg19's content: without upstream 3fdcfa8f7939783adff360f515ef7b37f1659c17, a stray "\." is an error. - src/postgres/src/backend/postmaster/postmaster.c: - BTYPE_MASK_ALL computes (1 << BACKEND_NUM_TYPES) - 1, which overflows int now that master commit b5132ae1036a141056ea3972853c18b721a5bdaf's YB_OSC_BACKFILL_BACKEND makes BACKEND_NUM_TYPES 31. Changed to 1U. - PostmasterStateMachine() asserts that every backend type is either stopped with the regular backends or allowed to keep running. Added master commit b5132ae1036a141056ea3972853c18b721a5bdaf's YB_OSC_BACKFILL_BACKEND to the YB backend types it stops with the regular backends, next to YB_INDEX_BACKFILL_DDL. Without it, assert builds abort the postmaster at every shutdown and crash restart. - src/postgres/src/include/miscadmin.h, src/postgres/src/include/postmaster/proctypelist.h, src/postgres/src/backend/utils/activity/pgstat_backend.c and pgstat_io.c: - master commit b5132ae1036a141056ea3972853c18b721a5bdaf adds backend type YB_OSC_BACKFILL_BACKEND. BACKEND_NUM_TYPES is now YB_OSC_BACKFILL_BACKEND + 1 (31 types); the uint32 BackendTypeMask has room for no more. proctypelist.h gets its PG_PROCTYPE entry, and pgstat_backend.c and pgstat_io.c add it to the YB backend-type cases. - src/postgres/src/backend/utils/misc/yb_internal_conn.c: - master commit b5132ae1036a141056ea3972853c18b721a5bdaf sets .backend_desc in the online schema change backfill entry. YB pg19 commit 57dac1c319f6d04ce3ee10c838d01f1824548d96 removed that field; dropped it. - src/postgres/src/backend/yb_online_schema_change/yb_osc_util.c: - master commit b5132ae1036a141056ea3972853c18b721a5bdaf calls RelationGetPrimaryKeyIndex() with one argument. On pg19 it also takes deferrable_ok; passed false, as every YB pg19 caller does. - master commit b5132ae1036a141056ea3972853c18b721a5bdaf declares NoKeyWitness() with a trailing pg_attribute_noreturn(). Upstream PG commit 3691edfab97187789b8a1cbb9dce4acf0ecd8f5a replaced that macro with a leading pg_noreturn. Changed the declaration to pg_noreturn static void. - src/postgres/src/backend/utils/misc/pg_yb_utils.c: - master commit 65c8784d869f6271ebac99c3484d7886a9fc1550 raises and catches ERRCODE_SNAPSHOT_TOO_OLD. Upstream PG commit f691f5b80a85c66d715b4340ffabb503eb19393e removed that code from errcodes.txt, but DocDB still returns SQLSTATE 72000 (YB_PG_SNAPSHOT_TOO_OLD). Added a local YB_ERRCODE_SNAPSHOT_TOO_OLD for it. - master commit 679286558f53a6dc5ac327d02e27a1ddbb3dcf1f's YbHasReplicationOriginSetup() reads replorigin_session_origin and InvalidRepOriginId. Upstream PG commit 8f1e2dfe033e9a3236265c3b9f61bd226f4a8f54 renamed them; changed to replorigin_xact_state.origin and InvalidReplOriginId. - src/postgres/src/backend/replication/logical/origin.c: - master commit 679286558f53a6dc5ac327d02e27a1ddbb3dcf1f's YbConnMgrLatchIfReplicationOriginExcluded() reads replorigin_session_origin and InvalidRepOriginId; changed as in pg_yb_utils.c. - src/postgres/src/backend/utils/adt/network.c: - master commit 73d72e0fcee40bd268a29d9f07561222631e9eaa's inet_client_addr() calls network_in() with two arguments. Upstream PG commit 17407a8eaa2afa8ac0de4b0a494f33d8eb7a98bd added the escontext argument; passed NULL, as pg19's other call in the function does. - src/postgres/src/backend/utils/adt/pseudorandomfuncs.c: - master commit 5f981e89455d06e3fe630ed1a6c2b7f497106dec's setseed() stickiness block (see float.c above). - src/postgres/contrib/pg_stat_statements/pg_stat_statements.c: - master commit e4089230d61bad87addd5ecd5e8249d8f13326d7's bloom-filter output block tests api_version >= YB_V2_2_PGSS_V1_10. On pg19 the upstream 1.11 to 1.13 versions sort after it and have no bloom-filter columns; changed to == YB_V2_2_PGSS_V1_10 || == YB_PGSS_V1_13. - src/yb/yql/pgwrapper/ysql_migrations/V113__33956__pg_stat_statements_add_bloom_filter_metrics.sql: - new in master commit e4089230d61bad87addd5ecd5e8249d8f13326d7. Updates to 1.13-yb-1.0 instead of 1.10-yb-2.2; pg19 has no 1.13-yb-1.0 -> 1.10-yb-2.2 path. - src/postgres/src/include/access/yb_scan_core.h and src/postgres/src/backend/utils/misc/guc.c: - master commit 11e8e78a79ee0aafd9846eb8a2849cbd4dff284c's header comment lets guc.c include yb_scan_core.h for the GUC variables the header declares. On pg19 those GUCs are in guc_tables.c (upstream PG commit 0a20ff54f5e66158930d5328f89f087d4e9ab400) and guc_parameters.dat (upstream PG commit 63599896545c7869f7dd28cd593e8b548983d613), and guc_tables.c includes the header. Changed the comment to name guc_tables.c. - guc.c's include of yb_scan_core.h, from master commit 7d994f45e0647863da599b90ef91cc585da32e4c, is unused on pg19; removed it. - src/postgres/src/test/regress/expected/yb.orig.alter_table_typmod_index.out: - new in master commit 05f14e65379922fb4b7d488e0fa614acab5f1fd1. Added PG19's LINE 1 caret under the INCLUDE (lower(b)) error (see yb.orig.alter_table_rewrite.out). - src/postgres/src/test/regress/sql/yb.orig.hints.sql and expected/yb.orig.hints.out: - YB pg19's two TODO(#33985) INFO lines carry the generated hint string, so they get master commit c192ba7f8532bd0ad04b12a4618b6ec40a77f801's Set(yb_max_merge_scan_streams 64). - Under a scan that a merge option disables, PG19's EXPLAIN prints "Disabled: true" (upstream PG commit 161320b4b960ee4fe918959be6529ae9b106ea5a). The plans match master's. - For master's three EXCEPT ALL statements, PG19 plans HashSetOp over the two inputs without Append (upstream PG commit 27627929528e24a547d1058a5444b35491057a56), and the generated hints include Leading(dt1dt2), the bug YB pg19 already tracks as TODO(#33985) for the v4 and v5 statements. Reblessed with a TODO(#33985) comment. - src/postgres/third-party-extensions/pg_hint_plan/expected/yb.orig.leading.out: - new in master commit 6961a55e34bcd99aa149fa4ea731e3dbf23035dd. PG19's EXPLAIN adds "Replaces: Join on ja, jb" under the Result (upstream PG commit f2bae51dfd5b2edc460c86071c577a45a1acbfd7). - src/postgres/src/test/regress/sql/yb.orig.ddl_savepoint_parallel.sql, yb.orig.ddl_txn_parallel.sql (new in master commit a94d07b6be3fe042ef415e7afddb6d1a1dcdc864), yb.orig.get_current_transaction_priority.sql, yb.orig.yb_index_check.sql (master commit a38606406b16607a032438e17dea424755d42b10), src/postgres/src/test/isolation/specs/yb.orig.yb_read_time_catalog_version_check.spec, and their expected outputs: - master's new cases SET force_parallel_mode. Upstream PG commit 5352ca22e0012d48055453ca9992a9515d811291 renamed it debug_parallel_query; renamed. The plans then match master's. - master's new EXPLAIN ANALYZE output prints "actual rows=N"; upstream PG commit ddb17e387aa28d61521227377b00f997756b8a27 prints "actual rows=N.00". Reblessed for that format only, here and in yb.orig.distinct_pushdown_pred.out (master commit d1c46e0647c1cd1d8f0610e096f3226ca96de739). - src/postgres/src/test/regress/sql/yb.orig.matview.sql and expected/yb.orig.matview.out, yb.orig.matview_1.out, yb.orig.matview_2.out and yb.orig.matview_3.out: - master commit 631f8ed95543a0c6936aeb6e3bf9427f9815168d's event-trigger function creates an index on unqualified cv_t. Fired inside REFRESH MATERIALIZED VIEW CONCURRENTLY, it runs with the restricted search_path of upstream PG commit 2af07e2f749a9208ca1ed84fa1d8fe0e75833288, so cv_t is not found; qualified it as public.cv_t. - The ANALYZE permission warning takes upstream PG commit b7a5ef17cf75c712b0fe5c5a20133a88da897aab's wording. - The alternative outputs get the same two changes. yb.orig.matview_2.out and yb.orig.matview_3.out, new in master commit 631f8ed95543a0c6936aeb6e3bf9427f9815168d, also take pg19's "permission denied for materialized view mtest_yb", which yb.orig.matview.out and yb.orig.matview_1.out already have. Each alternative output still differs from yb.orig.matview.out only as it does on master. - src/postgres/contrib/pg_stat_statements/sql/yb.port.utility.sql and expected/yb.port.utility.out: - master commit 01462d4e3d10038a2d6f69099f9531e832a54365 makes index backfill run SET yb_origination_time_override, which pg_stat_statements records. Added it to the test's filter of internal backfill statements. - java/yb-ysql-conn-mgr/src/test/java/org/yb/ysqlconnmgr/TestNonTransactionalStickiness.java: - new in master commit e80bc65d7756b0debca167c2d64644a61d7c64c5. It GRANTs EXECUTE on pg_replication_origin_session_setup(text). Upstream PG commit 5b148706c5c8ffffe5662fe569a0f0bcef2351d9 added an optional pid argument; changed the signature to (text, integer). - java/yb-ysql-conn-mgr/src/test/java/org/yb/ysqlconnmgr/TestSessionParameters.java: - master commit 5f981e89455d06e3fe630ed1a6c2b7f497106dec's testSeedMakesConnectionSticky runs RESET seed. Upstream PG commit 385366426511399a91da327c0f04765bbcfd5322 marked seed GUC_NO_RESET; dropped the RESET. DISCARD ALL still runs before the stickiness check. - java/yb-pgsql/src/test/java/org/yb/pgsql/TestPgReplicationSlot.java: - YB pg19 commit 41f97dd09a2718f31a210e4424bd434844646d9c's replicationSlotAcquireFailsWhenLocalArrayIsFull expects pg19's hint. The slot.c resolution takes master commit 6066d76d67e9d4b900d1f0e7a2144d057d579e5c's hint, "Free one or increase gflag "--max_replication_slots" on master and tserver."; updated the expected hint. - java/yb-pgsql/src/test/java/org/yb/pgsql/TestPgPushdownRescanMemory.java: - new in master commit 1e767152ed4dbe295d18413d08fd02efd9a9a268. testIndexScanInNestedLoop expects the inner index scan's pushed-down join filter as "Storage Filter: (o.v = v)". PG19 prints it as "Storage Filter: (v = o.v)", the form testSeqScan already expects; changed the expected line. Expected failures after the merge: - PgLegacyDdlUserContextTest.RetriedStatementRestoresAnalyzeUser, new in master commit 631f8ed95543a0c6936aeb6e3bf9427f9815168d, fails on pg19: "plancache reference ... is not owned by resource owner Portal" when the query layer retries SELECT analyze_once(). YB pg19 commit cdca957c59c03dab1de748ceb1588683927c1a78 deletes and recreates TopTransactionResourceOwner on a query-layer retry, which frees PL/pgSQL's shared simple-expression resource owner, a child of it. PL/pgSQL's pointer to that owner and its per-expression plan refcounts are not reset, so the retried function releases a plan refcount through the freed owner. Needs a pg19 fix to the retry path, tracked in #38288. Test Plan: - ./yb_build.sh release (Linux x86_64, clang21): full build passes; initial sys catalog snapshot created. - These pass (Linux x86_64, clang21, release): - ./yb_build.sh release --java-test org.yb.pgsql.TestPgRegressExtension - ./yb_build.sh release --java-test org.yb.pgsql.TestPgRegressMergeScan - ./yb_build.sh release --java-test org.yb.pgsql.TestPgRegressPgStatStatements - ./yb_build.sh release --java-test org.yb.pgsql.TestPgRegressRules - ./yb_build.sh release --java-test org.yb.pgsql.TestPgRegressYbExtensionsYbXclusterDdlReplication - ./yb_build.sh release --java-test org.yb.pgsql.TestPgRegressAlterTable - ./yb_build.sh release --java-test org.yb.pgsql.TestPgRegressContribYbGlobalViews - ./yb_build.sh release --java-test org.yb.pgsql.TestPgRegressFeature - ./yb_build.sh release --java-test org.yb.pgsql.TestPgRegressIsolationObjectLocking - ./yb_build.sh release --java-test org.yb.pgsql.TestDdlSavepoints - ./yb_build.sh release --java-test org.yb.pgsql.TestDdlTransactionBlocks - ./yb_build.sh release --java-test org.yb.pgsql.TestPgRegressDistinctPushdown - ./yb_build.sh release --java-test org.yb.pgsql.TestPgRegressDml - ./yb_build.sh release --java-test org.yb.pgsql.TestPgRegressPushdown - ./yb_build.sh release --java-test org.yb.pgsql.TestPgRegressColocation - ./yb_build.sh release --java-test org.yb.pgsql.TestPgRegressProc - ./yb_build.sh release --java-test org.yb.pgsql.TestPgRegressInsertOnConflict - ./yb_build.sh release --java-test org.yb.pgsql.TestPgRegressMatview - ./yb_build.sh release --java-test org.yb.pgsql.TestPgRegressMatviewLegacyDdl - ./yb_build.sh release --java-test 'org.yb.pgsql.TestPgPushdownRescanMemory#testIndexScanInNestedLoop' - ./yb_build.sh release --java-test org.yb.pgsql.TestPgRegressTablespaces - ./yb_build.sh release --java-test org.yb.pgsql.TestPgRegressMisc - ./yb_build.sh release --java-test org.yb.pgsql.TestPgRegressTrigger - ./yb_build.sh release --java-test org.yb.pgsql.TestPgRegressYbIndexCheck - ./yb_build.sh release --java-test org.yb.pgsql.TestPgRegressThirdPartyExtensionsPgvector - ./yb_build.sh release --java-test org.yb.pgsql.TestPgRegressThirdPartyExtensionsPgaudit - ./yb_build.sh release --java-test org.yb.pgsql.TestPgRegressTypesUDT - ./yb_build.sh release --java-test org.yb.pgsql.TestPgRegressIsolation - ./yb_build.sh release --java-test org.yb.pgsql.TestPgListenNotify - ./yb_build.sh release --java-test org.yb.pgsql.TestPgStatStatements - ./yb_build.sh release --java-test org.yb.pgsql.TestYbQueryDiagnostics - ./yb_build.sh release --java-test org.yb.pgsql.TestPgReplicationSlotWithTxnDdl - ./yb_build.sh release --java-test org.yb.ysqlconnmgr.TestCustomGucStickiness - ./yb_build.sh release --java-test org.yb.ysqlconnmgr.TestSusetGucStickiness - ./yb_build.sh release --java-test org.yb.ysqlconnmgr.TestNonTransactionalStickiness - ./yb_build.sh release --java-test org.yb.ysqlconnmgr.TestReplicationOriginStickiness - ./yb_build.sh release --java-test org.yb.ysqlconnmgr.TestSessionParameters - ./yb_build.sh release --java-test org.yb.ysqlconnmgr.TestPgStatActivity - ./yb_build.sh release --cxx-test ysql_major_extension_upgrade-test - ./yb_build.sh release --cxx-test ysql_major_upgrade_expression_pushdown-test - ./yb_build.sh release --cxx-test ysql_cdc_upgrade-test (the four CdcUpgradeTest and CdcUpgradeFrom2026_1Test tests skip) - ./yb_build.sh release --cxx-test pg_read_time-test - ./yb_build.sh release --cxx-test pg_backends-test - ./yb_build.sh release --cxx-test online_schema_change-test - ./yb_build.sh release --cxx-test pg_index_backfill-test - ./yb_build.sh release --cxx-test pg_hint_table-test - ./yb_build.sh fastdebug (Linux x86_64, clang21): full build passes. - These pass (Linux x86_64, clang21, fastdebug, read committed off): - ./yb_build.sh fastdebug --cxx-test pgwrapper_pg_wrapper-test --gtest_filter 'PgWrapperOneNodeClusterTest.TestPostgresLockFiles:PgWrapperOneNodeClusterTest.TestPgStatUserTablesPersistence' - ./yb_build.sh fastdebug --java-test org.yb.pgsql.TestPgRegressMatview - ./yb_build.sh fastdebug --java-test org.yb.pgsql.TestPgRegressMatviewLegacyDdl - These fail only in tests that already fail on pg19 before the merge (pg19 commit b7fe591841b6408822899045bf105569f5bbb793; for PgHintPlan, the plan diffs that #38109 lists as still failing on pg19): - ./yb_build.sh release --java-test org.yb.pgsql.TestPgRegressThirdPartyExtensionsPgHintPlan - ./yb_build.sh release --java-test org.yb.pgsql.TestPgReplicationSlot - ./yb_build.sh release --java-test org.yb.pgsql.TestYbQueryDiagnostics#testIntermediateFlushing - Jenkins

Commit:1acda2c
Author:Saurav Tiwary
Committer:GitHub

[BACKPORT 2026.2][#33495] CDC: Txn DDL upgrade path - Convert paused pub-refresh slots to implicit slots with yb-admin (#37571) (#38381) ## Summary Replication slots that detect publication changes through publication refresh (pub-refresh slots, created before 2026.1) have no retention barrier on the sys catalog tablet. With transactional DDL, the virtual WAL decodes schema and publication changes from catalog tables on that tablet, so a pub-refresh slot can't simply be given those tables: the changes committed since its restart time may already be gone. This PR converts such a slot to an implicit slot when it has been prepared for that, and makes it unusable otherwise. It follows #36926, which prepares implicit slots. New yb-admin commands, each taking a slot name and rejecting gRPC streams, LISTEN/NOTIFY slots, slots that are already implicit and slots marked unusable: - `set_replication_slot_catalog_barrier <replication_slot_name>` sets a retention barrier on the sys catalog tablet for the slot's stream and records a time T on the stream: the master's time after the barrier plus the maximum clock skew, which is past every change a consumer stopped beforehand can have received. It is idempotent while the barrier lives. - From VWAL init on, a pub-refresh slot with a recorded T ships no change committed after T, and once everything up to T is shipped, its safe time moves to T. Without this, a change after T that was shipped but not acknowledged before the pause would be shipped again after the conversion, and an implicit slot can give it a different LSN: it streams DDL from the catalog tables and picks up publication changes at their commit time rather than at the next refresh. - `pause_replication_slot <replication_slot_name> [<timeout_secs>]` marks the slot paused once its restart time reaches T, waiting for that up to the timeout (default 30 minutes). The restart time reaches T once the consumer has acknowledged every change up to T, even with no workload, since VWAL moves it to its safe time when everything shipped is acknowledged. VWAL init refuses a paused slot. - `release_replication_slot <replication_slot_name>` clears the barrier and the pause and deletes the slot's sys catalog cdc_state entry, so the slot streams as a pub-refresh slot again once its consumer restarts. Unless another stream still needs them (an implicit slot, or another slot with a barrier), it also releases the sys catalog tablet's retention barriers on every master, since an older master may never move them after a rollback. It fails if any of this fails, and can be rerun. Run it before rolling back, or to resume a paused slot without converting it. `prepare_replication_slots_for_transactional_ddl` now also handles pub-refresh slots. It reads and updates cdc_state outside the stream lock, then rechecks the slot under the write lock that adds the catalog tables, and fails with TryAgain if the slot changed in between: - A paused slot whose barrier is still live is converted: `detect_publication_changes_implicitly` is set, the pause marker is cleared, and the catalog tables are added. The recorded barrier time is kept. The sys catalog cdc_state entry that holds the retention stays, as the implicit slot's own entry. Its active time is reset, so the consumer has `cdc_intent_retention_ms` from the prepare to resume. - Any other pub-refresh slot is marked `cdcsdk_unusable_with_transactional_ddl`, its sys catalog cdc_state entry is deleted, and the command lists it as skipped with the reason. VWAL init refuses it with an error that says to create a new slot. Nothing keeps the conversion window alive. A pub-refresh slot doesn't poll the sys catalog, so the barrier expires `cdc_intent_retention_ms` (default 8 hours) after it is set, and prepare marks a paused slot whose barrier expired unusable. A paused slot's consumer is stopped, so its checkpoints on the user tablets expire after `cdc_intent_retention_ms` too. The flow below raises that flag on every server for the duration of the upgrade. A converted slot whose consumer resumes too late fails VWAL init through the existing expiry check, which covers both its user tablets and the sys catalog tablet. Barrier and pause fail once `TEST_ysql_yb_enable_replication_slot_transactional_ddl` is on. The master serializes the barrier command and release with each other and with adding a new stream to its maps, so release can't drop a sys catalog barrier that one of them just set. ### User flow `prepare_replication_slots_for_transactional_ddl` handles both kinds of slots in one run. **Pub-refresh slot (created before 2026.1):** converted to an implicit slot, and the catalog tables are added. Run these steps during the upgrade, once all masters and tservers run the new release, since a walsender on an older tserver ignores the pause and the barrier time. Don't create replication slots from step 1 until the upgrade is finalized. 1. Raise these flags to at least 24 hours, at runtime and in the gflags config so that the remaining restarts keep them. Raise `cdc_wal_retention_time_secs` before `cdc_intent_retention_ms`, since the validator keeps `cdc_intent_retention_ms` at or below it. - On all masters: `cdc_wal_retention_time_secs`, `cdc_intent_retention_ms`, `log_min_seconds_to_retain`, `cdc_min_replicated_index_considered_stale_secs_master`. - On all tservers: `cdc_wal_retention_time_secs`, `cdc_intent_retention_ms`. 2. Stop the consumer. 3. Run `yb-admin set_replication_slot_catalog_barrier <slot>`. It sets the barrier and prints its time T. Note T down. 4. Run `yb-admin pause_replication_slot <slot> [<timeout_secs>]`. It waits up to the timeout (default 30 minutes) for the slot's restart time to reach T, and then pauses the slot. If it times out, it prints the slot's restart time and T; rerun it until it succeeds. 5. While pause waits, start the consumer. It streams the changes committed up to T, and nothing after T until step 9, so its lag grows until then. Once the consumer has acknowledged every change up to T, VWAL moves the restart time to T within `cdcsdk_update_restart_time_interval_secs` (60 seconds by default), and the pause completes. 6. Stop the consumer. The pause is checked only at VWAL init, so it doesn't stop a consumer that is already streaming. Finalize: 7. Finalize the upgrade. 8. Run `yb-admin prepare_replication_slots_for_transactional_ddl` within `cdc_intent_retention_ms` of step 3. A paused slot with a live barrier is converted to implicit: its pause marker is cleared, the catalog tables are added, and the masters' caches are invalidated. Its sys catalog retention stays in place for the consumer to resume from. A slot that isn't paused, or whose barrier expired, is now unusable: it is marked so and listed as skipped. If the command reports an unreachable master, investigate why it is unreachable. Once that is fixed, rerun the command; it is safe to rerun. 9. Restart the consumer. 10. Once the consumer has caught up, revert all the flags from step 1 on the masters and tservers, `cdc_intent_retention_ms` before `cdc_wal_retention_time_secs`. Lag alone doesn't lose data, since barriers expire on inactivity rather than age, but the converted slot's user tablets were last active before the pause, so lowering `cdc_intent_retention_ms` before the consumer has polled each of them expires them at once. **Implicit slot that lacks the catalog tables** (as in #36926): 1. Before finalizing, raise the flags listed in step 1 of the pub-refresh case, in the same way, so that the slot doesn't expire while its consumer is stopped. 2. Stop the consumer. 3. Finalize the upgrade. 4. Run `yb-admin prepare_replication_slots_for_transactional_ddl`. It adds the catalog tables and invalidates the masters' caches. If it reports an unreachable master, investigate why it is unreachable. Once that is fixed, rerun the command; it is safe to rerun. 5. Start the consumer. 6. Once the consumer has caught up, revert all the flags from step 1, `cdc_intent_retention_ms` before `cdc_wal_retention_time_secs`. ### Follow ups 1. A YBA precheck at finalize that warns about pub-refresh slots that aren't paused, since prepare makes them unusable. ## Upgrade/Rollback safety New fields on `SysCDCStreamEntryPB` (`cdcsdk_implicit_conversion_barrier_ht`, `cdcsdk_paused_for_upgrade`, `cdcsdk_unusable_with_transactional_ddl`) and `CDCStreamInfoPB`, plus three new master RPCs and yb-admin commands. All are additive. - **Rollback before finalize:** 1. Stop the consumer of each slot that has a barrier. A consumer started after the barrier streams nothing after T until it restarts, even once the slot is released. 2. While the masters still run the new binary, run `yb-admin release_replication_slot <slot>` for each of those slots. It removes the barrier time and the pause marker from the stream, deletes the slot's sys catalog cdc_state entry, and releases the sys catalog retention barriers unless another slot still needs them. If it fails, rerun it. 3. Roll back the upgrade. 4. Restart the consumers. 5. Once the consumers have caught up, revert all the flags from step 1, `cdc_intent_retention_ms` before `cdc_wal_retention_time_secs`. - **After finalize:** conversion and unusable marking happen only in the prepare command, which requires the transactional DDL flag, which is set only after finalize. - **New yb-admin, old master:** the RPCs fail with no such method before anything changes. Original commit: 37b8414c133b7cddf8432cede210177ea151bb64 / #37571 ## Test plan ``` ./yb_build.sh debug --cxx-test cdcsdk_consumption_consistent_changes-test --gtest_filter 'CDCSDKConsumptionConsistentChangesTest.TestConvertPausedPubRefreshSlotToImplicit' ./yb_build.sh debug --cxx-test cdcsdk_consumption_consistent_changes-test --gtest_filter 'CDCSDKConsumptionConsistentChangesTest.TestReleaseAndExpirePausedPubRefreshSlot' ./yb_build.sh debug --cxx-test cdcsdk_consumption_consistent_changes-test --gtest_filter 'CDCSDKConsumptionConsistentChangesTest.TestConvertedSlotExpiresIfConsumerResumesLate' ./yb_build.sh debug --cxx-test cdcsdk_consumption_consistent_changes-test --gtest_filter 'CDCSDKConsumptionConsistentChangesTest.TestPrepareSlotForTransactionalDdl' ./yb_build.sh debug --cxx-test cdcsdk_consumption_consistent_changes-test --gtest_filter 'CDCSDKConsumptionConsistentChangesTest.TestPausedPubRefreshSlotRetention' ./yb_build.sh debug --cxx-test cdcsdk_consumption_consistent_changes-test --gtest_filter 'CDCSDKConsumptionConsistentChangesTest.TestStoppedImplicitSlotRetention' ./yb_build.sh debug --cxx-test ysql_cdc_upgrade-test --gtest_filter 'CdcUpgradeFrom2026_1Test.*' ./yb_build.sh debug --cxx-test master_xrepl-test ./yb_build.sh debug --cxx-test cdcsdk_ysql-test --gtest_filter 'CDCSDKYsqlTest.TestValidationAndSyncOfCDCStateEntriesAfterUserTableRemoval*' ``` - [x] `TestConvertPausedPubRefreshSlotToImplicit`: a consumer started after the barrier gets the change committed before it but not one committed after it, and the slot pauses once its restart time reaches T. After the conversion, that held-back change comes first, with an LSN past the last one sent before the pause. `pause_replication_slot` with a 1-second timeout fails on a slot whose consumer hasn't streamed up to the barrier. The converted slot decodes a DROP COLUMN committed while it was paused, which only `pg_attribute` shows. Releasing a slot keeps the sys catalog barriers that the paused slot needs. A slot that wasn't paused is reported as skipped and VWAL init refuses it. - [x] `TestReleaseAndExpirePausedPubRefreshSlot`: release releases the sys catalog WAL, intent and history barriers when no other slot needs them, and a released slot streams as a pub-refresh slot again. Release also deletes a sys catalog cdc_state entry that the stream has no barrier recorded for. A paused slot whose barrier expired is marked unusable instead of being converted, and release then rejects it. - [x] `TestConvertedSlotExpiresIfConsumerResumesLate`: `validate_and_sync_cdc_state_table_entries_on_change_data_stream` on a paused slot keeps its barrier entry, and the slot is still converted. A converted slot whose consumer resumes after `cdc_intent_retention_ms` fails VWAL init. With the user tablet's active time refreshed, it still fails on the sys catalog tablet. - [x] `TestPrepareSlotForTransactionalDdl` still passes. - [x] `TestPausedPubRefreshSlotRetention` and `TestStoppedImplicitSlotRetention`: with time-based WAL and history retention off, a slot stopped longer than the `cdc_intent_retention_ms` it was created with, after that flag is raised, keeps everything it needs on its user tablet and the sys catalog tablet through flushes, full compactions and WAL GC. Each compaction's history cutoff stays before the first change made while the consumer was stopped, both tablets keep intents, and the consumer then gets every change, the before images of `REPLICA IDENTITY FULL` updates, and a DROP COLUMN. - [x] `CdcUpgradeFrom2026_1Test.PausedPubRefreshSlotConvertedToImplicit` (upgrade from 2026.1.2.0): pub-refresh slots are created on the old version. One is paused before finalize, with a walsender consumer started after the barrier. Its restart time then equals the barrier time, and after the prepare it decodes DDL committed while it was paused. The other becomes unusable. `ImplicitSlotDecodesTxnDdlAfterUpgrade` still passes. - [x] `master_xrepl-test` 22/22. - [x] `TestValidationAndSyncOfCDCStateEntriesAfterUserTableRemovalOnNonConsistentSnapshotStream` and `...OnConsistentSnapshotStream` still pass. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> ## Merge conflicts ### master --> 2026.2 - `src/yb/integration-tests/cdcsdk_consumption_consistent_changes-test.cc:23` — [adjacent-lines] added only `sys_catalog_constants.h` and `rocksdb/db.h`; 2026.2 lacks the neighboring `master.h` and `master_tserver.h` includes, which the new code doesn't need Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

Commit:98ad689
Author:Hari Krishna Sunder

Rust master layer 3: leader duties and catalog gaps the web itests found Running master_path_handlers-itest on Rust leaders, and reviewing the series, found gaps outside the pages: - A tablegroup parent, including a colocated database's per-tablespace one, now follows its tablespace's placement, as TableTypeUsesTablespacesForPlacement says; it got the cluster RF. - The leader creates the test echo stateful service's table under TEST_echo_service_enabled. - YSQL history retention pins: heartbeats' per-database pins are aggregated and sent back with cluster_ysql_db_pins_ready, and the oldest is published to HISTORY_RETENTION_PIN and given to the host's sys catalog cutoff. Responses left the field unset before, and since it defaults to true tservers dropped their pins on every heartbeat from a Rust leader. - The host tells a master removed from the quorum to go back to shell mode. The pages gain the balancer tasks' live states, ongoing remote bootstraps, and each table's tserver RPC tasks in flight (TableTasks in master-task-runtime). Fixes from review: HybridTime <invalid> and <max> were swapped on /api/v1/xcluster, the UI unescaped names the JSON writes raw, and byte sizes used binary units with decimal names. web/ui/scale-test checks every page at 5000 tservers and 1M tables. In mixed quorums the itest reads a UI page's HTML from a C++ leader, since a Rust follower serves the web UI. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015zgRgyWVg8tMiDeykWemBo

Commit:e13e810
Author:IshanChhangani
Committer:GitHub

[BACKPORT 2026.1][#16670] DocDB: dist-trace: generate request trace attributes for every proto, tag CDC stream ids (#35037) (#36761) Backport stack for 2026.1, part 4 of 4, in master order: #34146 → #34156 → #34194 → #35037. Depends on https://github.com/yugabyte/yugabyte-db/pull/36760; review only the top commit. ## Summary The pb proxy stub drains `(yb.rpc.trace)` request fields through a `TracingAttributes(const PB&)` free function that was emitted into `.messages.cc`, so a proto only got request attributes on its RPC spans if it opted into lightweight messages. #34146 worked around that by moving `cdc_service`, `tserver_admin`, `master_admin` and `master_client` to `MESSAGES_PROTO_FILES`, which generates unused lightweight classes for every message in those files. A new trace generator in `protoc-gen-yrpc` emits `.trace.{h,cc}` for every proto file (unconditionally, so the CMake output list stays static) with the `TracingAttributes` / `AppendTracingAttributes` functions for its tagged messages. The header includes the imports' `.trace.h`, so a service whose request types live in another file (`tserver_service.proto` uses `tserver.proto`'s messages) still compiles. The proxy includes it when any request field is tagged and no longer needs `.messages.h` for the pb path; the lightweight generator keeps only the LW form. The four protos go back to `PROTO_FILES`. Attributes are still produced only in the proxy stub at call time when a trace is active, never during serialization. CDC request messages now tag `stream_id`, `db_stream_id` and `stream_ids` (as requested on #34146), so GetChanges, GetCheckpoint, SetCDCCheckpoint and the other stream-scoped RPCs carry the stream id on their spans. ## Upgrade/Rollback safety The `.proto` change only adds `(yb.rpc.trace)` field options on existing CDC request fields. Field numbers, types and wire encoding are unchanged, so mixed-version clusters and rollback are unaffected. The generated `.trace.*` files are build-time only. Original commit: f1e124bbdbcd32b7ce51fc3f568759fabe9f649a / #35037 ## Test plan - [x] `./yb_build.sh release --cxx-test lwproto-test` (plain-PB `TracingAttributes` now comes from `rtest.trace.h`) - [x] `./yb_build.sh release --cxx-test dist_trace-test --gtest_filter 'DistTraceTest.TestBackfillBackendJoinsQueryTrace:DistTraceRpcTest.TestRpcCrossing*'` — `TestBackfillBackendJoinsQueryTrace` now also asserts `req.tablet_id` on the `BackfillIndex` span, a PB-only RPC Co-authored-by: ellabaron-code <ella.baron@shopify.com>

Commit:37b8414
Author:Saurav Tiwary
Committer:GitHub

[#33495] CDC: Txn DDL with CDC upgrade path - Convert paused pub-refresh slots to implicit slots with yb-admin (#37571) ## Summary Replication slots that detect publication changes through publication refresh (pub-refresh slots, created before 2026.1) have no retention barrier on the sys catalog tablet. With transactional DDL, the virtual WAL decodes schema and publication changes from catalog tables on that tablet, so a pub-refresh slot can't simply be given those tables: the changes committed since its restart time may already be gone. This PR converts such a slot to an implicit slot when it has been prepared for that, and makes it unusable otherwise. It follows #36926, which prepares implicit slots. New yb-admin commands, each taking a slot name and rejecting gRPC streams, LISTEN/NOTIFY slots, slots that are already implicit and slots marked unusable: - `set_replication_slot_catalog_barrier <replication_slot_name>` sets a retention barrier on the sys catalog tablet for the slot's stream and records a time T on the stream: the master's time after the barrier plus the maximum clock skew, which is past every change a consumer stopped beforehand can have received. It is idempotent while the barrier lives. - From VWAL init on, a pub-refresh slot with a recorded T ships no change committed after T, and once everything up to T is shipped, its safe time moves to T. Without this, a change after T that was shipped but not acknowledged before the pause would be shipped again after the conversion, and an implicit slot can give it a different LSN: it streams DDL from the catalog tables and picks up publication changes at their commit time rather than at the next refresh. - `pause_replication_slot <replication_slot_name> [<timeout_secs>]` marks the slot paused once its restart time reaches T, waiting for that up to the timeout (default 30 minutes). The restart time reaches T once the consumer has acknowledged every change up to T, even with no workload, since VWAL moves it to its safe time when everything shipped is acknowledged. VWAL init refuses a paused slot. - `release_replication_slot <replication_slot_name>` clears the barrier and the pause and deletes the slot's sys catalog cdc_state entry, so the slot streams as a pub-refresh slot again once its consumer restarts. Unless another stream still needs them (an implicit slot, or another slot with a barrier), it also releases the sys catalog tablet's retention barriers on every master, since an older master may never move them after a rollback. It fails if any of this fails, and can be rerun. Run it before rolling back, or to resume a paused slot without converting it. `prepare_replication_slots_for_transactional_ddl` now also handles pub-refresh slots. It reads and updates cdc_state outside the stream lock, then rechecks the slot under the write lock that adds the catalog tables, and fails with TryAgain if the slot changed in between: - A paused slot whose barrier is still live is converted: `detect_publication_changes_implicitly` is set, the pause marker is cleared, and the catalog tables are added. The recorded barrier time is kept. The sys catalog cdc_state entry that holds the retention stays, as the implicit slot's own entry. Its active time is reset, so the consumer has `cdc_intent_retention_ms` from the prepare to resume. - Any other pub-refresh slot is marked `cdcsdk_unusable_with_transactional_ddl`, its sys catalog cdc_state entry is deleted, and the command lists it as skipped with the reason. VWAL init refuses it with an error that says to create a new slot. Nothing keeps the conversion window alive. A pub-refresh slot doesn't poll the sys catalog, so the barrier expires `cdc_intent_retention_ms` (default 8 hours) after it is set, and prepare marks a paused slot whose barrier expired unusable. A paused slot's consumer is stopped, so its checkpoints on the user tablets expire after `cdc_intent_retention_ms` too. The flow below raises that flag on every server for the duration of the upgrade. A converted slot whose consumer resumes too late fails VWAL init through the existing expiry check, which covers both its user tablets and the sys catalog tablet. Barrier and pause fail once `TEST_ysql_yb_enable_replication_slot_transactional_ddl` is on. The master serializes the barrier command and release with each other and with adding a new stream to its maps, so release can't drop a sys catalog barrier that one of them just set. ### User flow `prepare_replication_slots_for_transactional_ddl` handles both kinds of slots in one run. **Pub-refresh slot (created before 2026.1):** converted to an implicit slot, and the catalog tables are added. Run these steps during the upgrade, once all masters and tservers run the new release, since a walsender on an older tserver ignores the pause and the barrier time. Don't create replication slots from step 1 until the upgrade is finalized. 1. Raise these flags to at least 24 hours, at runtime and in the gflags config so that the remaining restarts keep them. Raise `cdc_wal_retention_time_secs` before `cdc_intent_retention_ms`, since the validator keeps `cdc_intent_retention_ms` at or below it. - On all masters: `cdc_wal_retention_time_secs`, `cdc_intent_retention_ms`, `log_min_seconds_to_retain`, `cdc_min_replicated_index_considered_stale_secs_master`. - On all tservers: `cdc_wal_retention_time_secs`, `cdc_intent_retention_ms`. 2. Stop the consumer. 3. Run `yb-admin set_replication_slot_catalog_barrier <slot>`. It sets the barrier and prints its time T. Note T down. 4. Run `yb-admin pause_replication_slot <slot> [<timeout_secs>]`. It waits up to the timeout (default 30 minutes) for the slot's restart time to reach T, and then pauses the slot. If it times out, it prints the slot's restart time and T; rerun it until it succeeds. 5. While pause waits, start the consumer. It streams the changes committed up to T, and nothing after T until step 9, so its lag grows until then. Once the consumer has acknowledged every change up to T, VWAL moves the restart time to T within `cdcsdk_update_restart_time_interval_secs` (60 seconds by default), and the pause completes. 6. Stop the consumer. The pause is checked only at VWAL init, so it doesn't stop a consumer that is already streaming. Finalize: 7. Finalize the upgrade. 8. Run `yb-admin prepare_replication_slots_for_transactional_ddl` within `cdc_intent_retention_ms` of step 3. A paused slot with a live barrier is converted to implicit: its pause marker is cleared, the catalog tables are added, and the masters' caches are invalidated. Its sys catalog retention stays in place for the consumer to resume from. A slot that isn't paused, or whose barrier expired, is now unusable: it is marked so and listed as skipped. If the command reports an unreachable master, investigate why it is unreachable. Once that is fixed, rerun the command; it is safe to rerun. 9. Restart the consumer. 10. Once the consumer has caught up, revert all the flags from step 1 on the masters and tservers, `cdc_intent_retention_ms` before `cdc_wal_retention_time_secs`. Lag alone doesn't lose data, since barriers expire on inactivity rather than age, but the converted slot's user tablets were last active before the pause, so lowering `cdc_intent_retention_ms` before the consumer has polled each of them expires them at once. **Implicit slot that lacks the catalog tables** (as in #36926): 1. Before finalizing, raise the flags listed in step 1 of the pub-refresh case, in the same way, so that the slot doesn't expire while its consumer is stopped. 2. Stop the consumer. 3. Finalize the upgrade. 4. Run `yb-admin prepare_replication_slots_for_transactional_ddl`. It adds the catalog tables and invalidates the masters' caches. If it reports an unreachable master, investigate why it is unreachable. Once that is fixed, rerun the command; it is safe to rerun. 5. Start the consumer. 6. Once the consumer has caught up, revert all the flags from step 1, `cdc_intent_retention_ms` before `cdc_wal_retention_time_secs`. ### Follow ups 1. A YBA precheck at finalize that warns about pub-refresh slots that aren't paused, since prepare makes them unusable. ## Upgrade/Rollback safety New fields on `SysCDCStreamEntryPB` (`cdcsdk_implicit_conversion_barrier_ht`, `cdcsdk_paused_for_upgrade`, `cdcsdk_unusable_with_transactional_ddl`) and `CDCStreamInfoPB`, plus three new master RPCs and yb-admin commands. All are additive. - **Rollback before finalize:** 1. Stop the consumer of each slot that has a barrier. A consumer started after the barrier streams nothing after T until it restarts, even once the slot is released. 2. While the masters still run the new binary, run `yb-admin release_replication_slot <slot>` for each of those slots. It removes the barrier time and the pause marker from the stream, deletes the slot's sys catalog cdc_state entry, and releases the sys catalog retention barriers unless another slot still needs them. If it fails, rerun it. 3. Roll back the upgrade. 4. Restart the consumers. 5. Once the consumers have caught up, revert all the flags from step 1, `cdc_intent_retention_ms` before `cdc_wal_retention_time_secs`. - **After finalize:** conversion and unusable marking happen only in the prepare command, which requires the transactional DDL flag, which is set only after finalize. - **New yb-admin, old master:** the RPCs fail with no such method before anything changes. ## Test plan ``` ./yb_build.sh debug --cxx-test cdcsdk_consumption_consistent_changes-test --gtest_filter 'CDCSDKConsumptionConsistentChangesTest.TestConvertPausedPubRefreshSlotToImplicit' ./yb_build.sh debug --cxx-test cdcsdk_consumption_consistent_changes-test --gtest_filter 'CDCSDKConsumptionConsistentChangesTest.TestReleaseAndExpirePausedPubRefreshSlot' ./yb_build.sh debug --cxx-test cdcsdk_consumption_consistent_changes-test --gtest_filter 'CDCSDKConsumptionConsistentChangesTest.TestConvertedSlotExpiresIfConsumerResumesLate' ./yb_build.sh debug --cxx-test cdcsdk_consumption_consistent_changes-test --gtest_filter 'CDCSDKConsumptionConsistentChangesTest.TestPrepareSlotForTransactionalDdl' ./yb_build.sh debug --cxx-test cdcsdk_consumption_consistent_changes-test --gtest_filter 'CDCSDKConsumptionConsistentChangesTest.TestPausedPubRefreshSlotRetention' ./yb_build.sh debug --cxx-test cdcsdk_consumption_consistent_changes-test --gtest_filter 'CDCSDKConsumptionConsistentChangesTest.TestStoppedImplicitSlotRetention' ./yb_build.sh debug --cxx-test ysql_cdc_upgrade-test --gtest_filter 'CdcUpgradeFrom2026_1Test.*' ./yb_build.sh debug --cxx-test master_xrepl-test ./yb_build.sh debug --cxx-test cdcsdk_ysql-test --gtest_filter 'CDCSDKYsqlTest.TestValidationAndSyncOfCDCStateEntriesAfterUserTableRemoval*' ``` - [x] `TestConvertPausedPubRefreshSlotToImplicit`: a consumer started after the barrier gets the change committed before it but not one committed after it, and the slot pauses once its restart time reaches T. After the conversion, that held-back change comes first, with an LSN past the last one sent before the pause. `pause_replication_slot` with a 1-second timeout fails on a slot whose consumer hasn't streamed up to the barrier. The converted slot decodes a DROP COLUMN committed while it was paused, which only `pg_attribute` shows. Releasing a slot keeps the sys catalog barriers that the paused slot needs. A slot that wasn't paused is reported as skipped and VWAL init refuses it. - [x] `TestReleaseAndExpirePausedPubRefreshSlot`: release releases the sys catalog WAL, intent and history barriers when no other slot needs them, and a released slot streams as a pub-refresh slot again. Release also deletes a sys catalog cdc_state entry that the stream has no barrier recorded for. A paused slot whose barrier expired is marked unusable instead of being converted, and release then rejects it. - [x] `TestConvertedSlotExpiresIfConsumerResumesLate`: `validate_and_sync_cdc_state_table_entries_on_change_data_stream` on a paused slot keeps its barrier entry, and the slot is still converted. A converted slot whose consumer resumes after `cdc_intent_retention_ms` fails VWAL init. With the user tablet's active time refreshed, it still fails on the sys catalog tablet. - [x] `TestPrepareSlotForTransactionalDdl` still passes. - [x] `TestPausedPubRefreshSlotRetention` and `TestStoppedImplicitSlotRetention`: with time-based WAL and history retention off, a slot stopped longer than the `cdc_intent_retention_ms` it was created with, after that flag is raised, keeps everything it needs on its user tablet and the sys catalog tablet through flushes, full compactions and WAL GC. Each compaction's history cutoff stays before the first change made while the consumer was stopped, both tablets keep intents, and the consumer then gets every change, the before images of `REPLICA IDENTITY FULL` updates, and a DROP COLUMN. - [x] `CdcUpgradeFrom2026_1Test.PausedPubRefreshSlotConvertedToImplicit` (upgrade from 2026.1.2.0): pub-refresh slots are created on the old version. One is paused before finalize, with a walsender consumer started after the barrier. Its restart time then equals the barrier time, and after the prepare it decodes DDL committed while it was paused. The other becomes unusable. `ImplicitSlotDecodesTxnDdlAfterUpgrade` still passes. - [x] `master_xrepl-test` 22/22. - [x] `TestValidationAndSyncOfCDCStateEntriesAfterUserTableRemovalOnNonConsistentSnapshotStream` and `...OnConsistentSnapshotStream` still pass. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

Commit:4c1bd14
Author:Sergei Politov
Committer:GitHub

[#36288] DocDB: Keep the vectors of rows whose DELETE or UPDATE is rolled back to a savepoint (#37985) ## Summary A DELETE or UPDATE of a row with a vector puts the ids of the vectors it removes into a delete list record of its batch, and the transaction apply tombstones them in the vector reverse mapping. The record had no subtransaction id, so it was applied even when its subtransaction was rolled back, and the index lost the vector of a live row for good. Besides `ROLLBACK TO SAVEPOINT`, READ COMMITTED statement retries, plpgsql `EXCEPTION` blocks and client autosave roll back subtransactions too. The record now starts with a new value type for vector index records, `kVectorIndexRecord`, followed by the record type `VectorIndexRecordType::kDeleteList` and the subtransaction id of its batch, and the apply skips the records of rolled back subtransactions. A later record format takes a new record type rather than a new value type. The readers that only skip these records (transaction loading, lock status and CDC) recognize both the old and the new record through `dockv::IsVectorIndexRecord`. Also moves the subtransaction id encoding of intent values to `EncodeSubTransactionId`, shared by the intents and the external intents writers, which writes the same bytes. ## Upgrade/Rollback safety An older binary does not know the new record: its apply looks the record up as an intent key, does not find it and skips it, so the vectors of the rows deleted by the transaction stay in the index, and its transaction loader logs a decode failure (both `DFATAL`). So the leader writes the new record only after the new AutoFlag `vector_index_delete_list_with_subtransaction` (`kLocalPersisted`) is promoted, and replicates its choice in the write batch (`delete_vector_ids_with_subtransaction`, a new field of `KeyValueWriteBatchPB`), so all replicas and WAL replay write the same record. Promoting the AutoFlag is the point of no return: after it, an older binary leaves the vectors of the rows deleted by transactions that hold the new record. Until the promotion, rolled back subtransactions still lose vectors. `yb-ts-cli verify_vector_indexes` detects the vectors lost before it, and rebuilding the index or rewriting the affected rows restores them. ## Test plan - [x] `./yb_build.sh debug --cxx-test pgwrapper_pg_vector_index-test --gtest_filter PgVectorIndexUtilTest.RolledBackSubtransactionKeepsVectors`: without the fix rows 1, 3, 20 and 1000 are missing from the index search results. - [x] `LoadTransactionWithDeleteList`, `DeleteListsWithBothEncodings`, `SameTxnReplacedVector` and `SameTxnDeletedVector` of `PgVectorIndexUtilTest` pass. - [ ] Jenkins.

Commit:1de1ac9
Author:IshanChhangani
Committer:GitHub

[BACKPORT 2026.1][#16670] DocDB: dist-trace: (yb.rpc.trace) field option and tags for tablet, table and object-lock request fields (#36757) Backport stack for 2026.1, part 1 of 4, in master order: #34146 → #34156 → #34194 → #35037. ## Summary Adds a `(yb.rpc.trace)` field option to `lightweight_message.proto` and tags the first production fields with it. Tagging a request field with `[(yb.rpc.trace) = {enabled: true}]` makes gen_yrpc emit `TracingAttributes()` for the message, and the generated proxy stub drains it onto the outbound RPC's client span, gated on an active trace context so untraced requests build nothing. Nested and repeated messages are followed, keys are `req.<field>[.<sub>]`, enums by name, `bytes` hex-encoded unless `bytes_as_string`. A repeated field yields one key with `*` in place of the index (`req.ops.*.write.table_id`) whose value lists the distinct values sorted, so the attribute count does not grow with the batch. The shared-memory exchange never goes through a proxy stub, so `PgClientData::StartSharedMemorySpan` drains the same attributes by hand before starting the simulated client span. Tagged: tablet and table ids across tserver, master, consensus, CDC and backup RPCs, and the object-lock oids and lock mode on `PgClientService.AcquireObjectLock`. The pb_ stub can only call `TracingAttributes(req)` from a generated `.messages.cc`, so `master_client`, `master_admin`, `tserver_admin` and `cdc_service` move to `MESSAGES_PROTO_FILES`; without that a tag compiles and emits nothing. Perform, tablet Read and tablet Write spans also get `rpc.table_names`, one `name(id)` entry per distinct table the request touches including the index behind `index_request`, so the raw id attributes can be read without a catalog lookup. Design: https://docs.google.com/document/d/1nwb-V1dIa-8mLM5dvZwsDgoQ1ODLUA9TUFFqjXUeplo/edit?tab=t.0#heading=h.k8r70zn36j4j ## Upgrade/Rollback safety Descriptor-only change: a new `FieldOptions` extension and options on existing fields. Options are never serialized, no field numbers or message shapes change. Generating messages files for four more protos adds lightweight C++ types but changes nothing on the wire. Mixed-version clusters and rollback are unaffected. Original commit: ad0e95cfc93cbc82cd00c8ba8968367a46e57e03 / #34146 ## Test plan - [x] `LWProtoTest.TracingAttributes` tags fields in `rtest.proto` covering string, hex bytes, bytes_as_string, uint32, enum, bool, repeated scalars, nested, repeated nested, pointer and self-referential messages, and checks the lightweight and pb_ forms against one expected list. `./yb_build.sh release --cxx-test lwproto-test` passes. - [x] `dist_trace-test` asserts the attributes end to end on both paths: `req.ops.*.read.table_id` and `req.ops.*.write.table_id` on shared-memory Perform spans, `req.lock_oid.*` and `req.lock_type` on the shared-memory object-lock span, `req.ops.*.read.table_id` on the RPC Perform span plus `req.tablet_id` and `rpc.table_names` on the tserver's Read and Write spans. `./yb_build.sh release --cxx-test dist_trace-test` passes. <!-- Reviewable:start --> - - - This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/yugabyte/yugabyte-db/36757) <!-- Reviewable:end --> ## Merge conflicts - `src/yb/tserver/pg_client.proto` `ObjectLockOid`: kept 2026.1's `uint64` oid types and added the `(yb.rpc.trace)` tags. Co-authored-by: ellabaron-code <ella.baron@shopify.com>

Commit:ad13cf6
Author:Bvsk Patnaik
Committer:Bvsk Patnaik

[#37328] YSQL: Say what the object lock request check catches The check in DoAcquireObjectLock fires when a lock request comes from a transaction that does not use object locks locally, which means a pggate sender skipped UsingObjectLocksLocally(). Its message, copied from master, said "Table Locking feature not enabled.", which points at the gflags; in release builds that text reaches the client. Say what the check found instead. State the field's contract in its proto comment: pggate sets it from the same predicate that decides whether the transaction sends its locks. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Commit:1814f49
Author:Bvsk Patnaik
Committer:Bvsk Patnaik

[#37328] YSQL: Shorten the using_table_locks_locally comment and scale the test timeouts - pg_client.proto: the field comment points to PgSession::AcquireObjectLock for why a transaction without global object locks takes local locks, instead of repeating it with one of its two issues. - IsTxnUsingLocalTableLocks becomes IsTxnUsingTableLocksLocally, to match the field. - CheckFastPathWriteDeadlockIsDetected scales its two statement timeouts by kTimeMultiplier. The ALTER's timeout decides the test, so a slow build must not read as a missed cycle. - The rolling restart test moves next to the other tests of its fixture. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

Commit:724cb05
Author:Bvsk Patnaik
Committer:Bvsk Patnaik

[#37328] YSQL: Tag legacy-mode fast path writes with their object lock owner Replays #37359's commits (074ed6508a4..eb4ffe3042a) as one commit on #37358's renamed request field (using_table_locks_globally) and infra flag (ysql_yb_enable_global_object_locking_infra): the fast path owner tag, PgPerformOptionsPB field 42 is_using_local_table_locks and its setter, the check that object lock requests come from a lock-taking transaction, and FastPathWriteDeadlockIsDetected. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Commit:e38fc75
Author:Bvsk Patnaik
Committer:Bvsk Patnaik

[#37328] YSQL: Rename is_using_local_table_locks to using_table_locks_locally Match its neighbor using_table_locks_globally: the pair now reads as the same question at two scopes, set from UsingObjectLocksLocally() and UsingObjectLocksGloballyForCurrentTxn(). The tag stays 42, so the wire format does not change, and the field is new in this PR. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Commit:f23c0ff
Author:Bvsk Patnaik
Committer:GitHub

[#36572] YSQL: Keep new indexes and foreign keys intact for writers that predate object locking (#37358) ## Summary ### Background While the `ysql_yb_enable_global_object_locking_infra` AutoFlag is being promoted, some transactions latched it on and some off: each transaction latches the flag when it starts. DDLs wait for concurrent DMLs only through object locks since 7dd192f0d83 (#31758), which replaced CREATE INDEX's `YbWaitForBackendsCatalogVersion` barriers with `WaitForLockers(ShareLock)`. - **object locks** (`UsingObjectLocksLocally()`): the transaction sends PG's locks to the tserver. Off in initdb, binary upgrade, parallel workers, autonomous DDLs, transactions that read the catalog as of `yb_read_time`, and with `enable_object_locking_for_table_locks` off. - **global object locks** (`UsingObjectLocksGloballyForCurrentTxn()`): object locks, and the transaction latched the infra flag on. Its DDLs take global locks and run in the transaction block. This is safe because the flag is promoted only once every tserver runs the new binary. - **autonomous DDL**: a DDL of a transaction that latched the flag off runs in its own DDL transaction. | | DML object locks | DDL object locks | mode of catalog operations | catalog cache refresh | |---|---|---|---|---| | latched off, before | none | none | legacy mode of catalog operations | between transactions | | latched off, after | local (RowExclusive or weaker) | none (autonomous DDL) | PG mode of catalog operations | on a new lock (AcceptInvalidationMessages) | | latched on | local | global | PG mode of catalog operations | on a new lock (AcceptInvalidationMessages) | ### Problem A transaction that started before the promotion runs next to DDLs that started after it. 1. **A DDL does not wait for an older writer.** CREATE INDEX misses the writer's row: ```sql -- ts-1, started before the promotion BEGIN; UPDATE test SET v = 42 WHERE k = 1; -- ts-2, after the promotion CREATE INDEX test_v_idx ON test(v); -- does not wait; backfill reads v = 0 -- ts-1 COMMIT; -- index keeps (0, k = 1) SELECT yb_index_check('test_v_idx'::regclass); -- inconsistent index row ``` ADD FOREIGN KEY does not wait for an older DELETE in the same way. 2. **An older writer uses a catalog cache from before the DDL.** It read `test`, CREATE INDEX ran elsewhere, and its INSERT is planned without the index. Today the tablet rejects it with 40001 (schema version mismatch); after #33037 it would commit with no index entry, or skip a new foreign key check. ### Root cause 1. `PgSession::AcquireObjectLock` returned before any RPC when the transaction latched the infra flag off, so `WaitForLockers` and the DDL's global locks saw no holder. Since 7dd192f0d83. 2. `AcceptInvalidationMessages` refreshed only under `YBCIsObjectLockingEnabled()`, which reads the latched infra flag, so such a transaction never refreshed mid-transaction, unlike PG. ### Fix Two predicates, and every caller asks one of them: - `UsingObjectLocksLocally()` gates the locks a transaction sends, the refresh on a new lock and the mode of catalog operations, legacy or PG (`YBCIsLegacyModeForCatalogOps()`). - `UsingObjectLocksGloballyForCurrentTxn()` = that && the latched infra flag. It gates global and session locks, `WaitForLockers`, DDL in a transaction block, `using_table_locks_globally` (the skip of the tablets' catalog version check and of the DDL abort), and the DDL-side callers in PG. A transaction that latched the flag off sends locks of mode RowExclusive or weaker and no session locks; PG still records every lock locally, as upstream does. It releases them at commit, abort and ROLLBACK TO SAVEPOINT, and keeps both old fences. ### Why the fix works Only the non-obvious parts: - **The refresh and the local locks ship together.** After a refresh the transaction's requests carry the new catalog version, so the tablets' catalog version check no longer stops them. What stops a reader from reading a table rewritten after its snapshot is then its AccessShare lock, which the rewrite waits for, exactly as for a transaction with global object locks. ### Known limitations 1. A transaction that latched the infra flag off loses the transparent single-statement retry on "The catalog snapshot used for this transaction has been invalidated" (postgres.c, keyed on the legacy mode of catalog operations). 2. An autonomous DDL sends no locks, but PG still records the ones it takes. A later statement of the same transaction that asks for the same lock (a SELECT after CREATE VIEW on the same table) finds it held, so PG does not ask YB and the lock is never sent; a DDL with global object locks does not wait for that reader. `enable_object_locking_for_table_locks` keeps its meaning: off means no object locks, local or global. So a running cluster moves between global object locking and no object locks in two rolling restarts, with #37334's `disable_global_object_locks` flag as the middle step: - To turn object locks off: first set `disable_global_object_locks=true`, then set `enable_object_locking_for_table_locks=false`. - To turn them on: first set `enable_object_locking_for_table_locks=true` with `disable_global_object_locks=true`, then set `disable_global_object_locks=false`. One rolling restart is not safe in either direction (#37355): while it runs, a node with `enable_object_locking_for_table_locks` off takes no local object locks, so a DDL with global object locks on another node does not wait for that node's writers. #37958 tracks the docs for this procedure. `ysql_yb_enable_object_locking_infra` (GUC `yb_enable_object_locking_infra`) is renamed `ysql_yb_enable_global_object_locking_infra`: with local locks now independent of it, it gates global object locks and what needs them (transactional DDL, concurrent DDL, the tablets' catalog version skip). It is only on master and the unreleased 2026.2, so it gets no deprecated alias. The request field `is_using_table_locks` is renamed `using_table_locks_globally` for the same reason: a DML-only transaction with it set holds only local locks. ## Upgrade/Rollback safety No new wire field: the renamed request fields keep their field numbers. The AutoFlag `ysql_yb_enable_object_locking_infra` is renamed with no alias. The old name is only on master and the 2026.2 branch. This PR will be backported to 2026.2, and no release has been cut from that branch so far, so no released cluster has the old name. ## Test plan New tests in `src/yb/yql/pgwrapper/pg_object_locks-test.cc`. Unless noted, each runs with the older transaction's node promoted and not promoted before the DDL (`PgObjectLocksTestLegacyTxnVsNewDdl`). - `CreateIndexWaitsForLegacyWriter`: two older writers, one REPEATABLE READ and one READ COMMITTED, each have an UPDATE open; CREATE INDEX on another node must wait for both, and the index must have the updated values. Before this PR: CREATE INDEX does not wait and the index is corrupt (`inconsistent index row`). - `AddForeignKeyWaitsForLegacyDelete`: ADD FOREIGN KEY must wait for an older DELETE of a parent row. Before this PR: it does not wait. - `LegacyWriteMaintainsNewIndex`: an RR and an RC older transaction read the table, CREATE INDEX runs elsewhere, then each inserts a row; the index must have both. Before this PR: the INSERT fails with 40001 `schema version mismatch`. - `LegacyDeleteChecksNewForeignKey`: after ADD FOREIGN KEY elsewhere, an older transaction's DELETE of a referenced parent row must fail with a foreign key violation. Before this PR: the DELETE is planned without the new foreign key, because the transaction never refreshes. - `LegacyReadOnlyTxnReleasesLocksAtCommit`, `LegacyReadOnlyTxnReleasesLocksAtRollbackToSavepoint`, `LegacyWriterReleasesLocksAtRollback`: a DDL must not wait for an older transaction after its COMMIT, ROLLBACK TO SAVEPOINT or ROLLBACK. Before this PR: such a transaction took no locks, so there was nothing to release. With this PR's locks but without the matching release change, the DDL hits its statement timeout. - `TableRewriteWaitsForLegacyReader`: ALTER COLUMN TYPE (a table rewrite) must wait for an older reader's AccessShare lock, and the reader must still read its 10 rows. Before this PR: the ALTER does not wait, and the reader keeps its old catalog. - `NewDdlDoesNotWaitForLockFirstTakenInDdlMode`: pins known limitation 2. The older transaction runs CREATE VIEW on the table, then SELECTs it; the SELECT's AccessShare is not sent, so ALTER TABLE on another node does not wait for it, and the reader still commits. Before this PR: no locks were sent at all, so the ALTER did not wait either. - `LegacyCreateIndexSucceeds`: a CREATE INDEX in a transaction that latched the flag off must succeed. It takes a session lock through `LockRelationIdForSession`, which FATALs unless PG holds the transaction lock in its own lock table, so the locks pggate does not send must still be recorded by PG. Before this PR: passes; no locks were sent. - `LegacyDdlLeavesNoStaleCatalogEntry` (not parameterized): an autonomous DDL must not refresh its cache and keep stale entries after it ends. Before this PR: such a DDL never refreshed. Making the new refresh apply to autonomous DDLs fails it with `column "w" does not exist`. - `LegacyRewriteNeverEmptiesLegacyReader` (not parameterized): an autonomous table rewrite takes no locks, so nothing makes it wait for an older reader; the reader must get an error or the rows it saw before, never the empty new table. Before this PR: the reader never refreshed. - `LegacyCreateIndexAcrossPromotion` (not parameterized): an autonomous CREATE INDEX blocks between phases (`ysql_yb_test_block_index_phase`), the infra flag is promoted, and the CREATE INDEX must finish without a FATAL. Before this PR: no object locks were sent, before or after the promotion. - `ObjectLockingPromotionUpgradeTest.CreateIndexWaitsForLegacyWriterAcrossPromotion` (`upgrade-tests`): upgrade from 2025.1.1.0, open a writer before the AutoFlags are promoted, then run CREATE INDEX. It must wait for the writer, the index must have the writer's row, and the count must go through the new index (EXPLAIN). Before the promotion, `pg_locks` on the writer's node must show its granted RowExclusiveLock. Release builds only: debug builds have object locking off by default. Before this PR: the writer holds no lock, so CREATE INDEX does not wait. - `AutonomousDdlKeepsLegacyTxnLocks`: an older reader holds AccessShare on a table, then runs an autonomous DDL in the same transaction; a table rewrite with global object locks must still wait for the reader. Without the tserver's skip of the release for autonomous DDL sessions, the rewrite does not wait. - `LegacyCursorStoppedByCatalogVersionCheck` (not parameterized): an older transaction reads from a cursor opened before a REVOKE; the read takes no new lock, so it reaches the tablet with the old catalog version and must fail the tablets' catalog version check. Sending `using_table_locks_globally` for a transaction with only local locks makes the read succeed. - `CursorWithGlobalLocksReadsToEndAfterRevoke` (not parameterized): the same steps for a transaction with global object locks. The tablets skip the check and PG checks privileges only when the cursor is opened, so the cursor reads to the end, as in PG; the transaction's next statement fails with `permission denied`. Waits in `CreateIndexWaitsForLegacyWriter`, `AddForeignKeyWaitsForLegacyDelete`, `TableRewriteWaitsForLegacyReader`, `LegacyDdlLeavesNoStaleCatalogEntry` and the upgrade test wait on the condition (`pg_locks`, ASH, `pg_stat_activity`), not on a fixed sleep. Changed tests: - `TestCatalogOpModeStableAcrossPromotion` (#29766): a transaction with object locks now reads the catalog with a catalog snapshot read time from its first statement, so the test expects that read path both before and after the infra flag is promoted under the open transaction. Keying catalog reads on global object locks again makes it fail at the pre-promotion read. - `LegacyTxnKeepsCatalogVersionCheck` (#32224) is removed. The older reader now refreshes on its next lock, so PG's privilege check stops it (`permission denied`) before the read reaches the tablet, and the test no longer shows the tablets' catalog version check. `LegacyCursorStoppedByCatalogVersionCheck` shows it instead. - `PgLocksV76UpgradeTest.TestPgLocksViewAdvisoryLocksSupport`: after finalize, the open transaction's `pg_locks` read refreshes on its new lock and sees the new row type when table locks are on; with table locks off it takes no locks and keeps the stale row type. The test reads `enable_object_locking_for_table_locks` from the tserver to pick the expected outcome: release builds default it to on and debug builds to off, so each build type covers one outcome. ``` ./yb_build.sh release --cxx-test pgwrapper_pg_object_locks-test \ --gtest_filter '*LegacyTxnVsNewDdl*:PgObjectLocksTestMixModeDuringPromotion.*:*AbortTxnsInMixedMode*' ./yb_build.sh release --cxx-test integration-tests_pg_locks_v76_upgrade-test ``` Results, release, each test in its own process, on the head merged with master: - `pgwrapper_pg_object_locks-test`: 108 passed. - `pgwrapper_pg_read_time-test` (with #36925's `yb_read_time` tests): 167 passed. - `integration-tests_object_locking_promotion_upgrade-test`: pass. - `integration-tests_pg_locks_v76_upgrade-test`: table locks off passes (`Returned row contains 24 attributes, but query expects 21`); table locks on failed once at the first pg_locks read with `Couldn't fetch old transactions`, the #37761 flake that #37814 fixes; it passed on earlier runs. - Mutation checks: each new or changed test above fails with the behavior it pins reverted. Part of stack #37903, bottom to top: #37358, #37334, #37359. Jenkins: no brass Fixes #36572 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

Commit:2e06e07
Author:Yamen Haddad
Committer:GitHub

[#37241] DocDB: ThinClientService: consistent-prefix reads with a server-computed staleness (#37242) Fixes #37241 ### Summary Every read through `ThinClientService.Perform` is STRONG, so the tablet leader serves it; for a caller spread across zones (amp's pageserver reading page rows) that crosses an AZ two thirds of the time. This lets a batch ask for a consistent-prefix read instead, at a read time the **tserver** derives from its own clock minus a caller-supplied staleness, so only the cluster's `max_clock_skew_usec` matters, as for YSQL follower reads. - `thin_client.proto`: `ThinReadTimeOptionsPB` gains `bool read_from_followers = 2` and `optional uint32 follower_read_staleness_ms = 3`, additively. - `thin_client_service.cc`: with `read_from_followers`, every read op of the batch runs at `CONSISTENT_PREFIX`. With a staleness and no pinned `read_time` the session reads at `now - staleness`, through `SetFollowerReadTime`, which moves from `pg_client_session.cc` to `pg_client_session_util.{h,cc}` so both follower-read paths share it. A pinned `read_time` (a paging continuation) keeps its snapshot and only takes the consistency level. `InvalidArgument` for: a staleness on a write batch, a staleness without `read_from_followers` (the leader would serve a historical snapshot nobody asked for), a staleness of 0 (it would read at `now`, which no follower can serve) or below `2 * max_clock_skew_usec` (1000 ms by default; a follower usually cannot serve that either, and YSQL refuses the same range in `is_staleness_acceptable`), and `read_from_followers` with neither a `read_time` nor a staleness. `read_from_followers` on a write batch is ignored. - `thin_client_service.cc`: when the batch's read point holds a time after the flush (pinned, staleness-derived, or stamped by the batcher for a multi-tablet batch), it goes into every op paging state that has none. DocDB binds `read_time` only when it stops inside a tablet (`PgsqlReadOperation::SetPagingState`); the paging state that moves a scan to the next tablet (`Tablet::CreatePagingStateForRead`) has none. Without this, a follower scan whose first page ended at a tablet boundary reached the shim with no snapshot, and its continuation was rejected. DocDB reads `paging_state.read_time` only together with `next_row_key`, so the stamp is a report to the client and changes nothing in the tablet. An unpinned single-tablet leader read leaves the read point empty and is unchanged, so such a scan still moves to a newer snapshot at each tablet boundary; that predates this PR, is tracked as #38112, and the `ybthin_read_async` comment now says so. - `yb_thin_client.{h,cc}`: `ybthin_read_async` takes a `const ybthin_read_options*` in place of its bare `read_time_ht`; NULL or a zeroed struct is the leader read it was before. This changes the function's signature (see the notes below). The shim refuses a fresh follower batch with a zero staleness as `YBTHIN_INVALID` before sending it. The paging envelope (version 4) carries the scan's `read_from_followers` next to its read time, so every page of a follower scan asks for a follower at the first page's snapshot without the caller re-passing anything. A scan keeps the replica choice of its first page: follower options on the continuation of a scan that started at the leader are ignored, and a continuation whose paging state holds no read time is a leader read. ### What the follower handoff costs A follower behind the read time hands the read to another replica through existing code: `ReadQuery::WaitForSafeTime` returns `IllegalState` under `ysql_follower_reads_avoid_waiting_for_safe_time`, and `TabletInvoker` retries the next replica. The caller still sees one Perform, but the handoff is not free: that `IllegalState` carries no `TabletServerError` code, so `TabletInvoker::FailToNewReplica` reaches `MarkReplicaFailed` and the local replica stays failed in the tserver's shared meta cache until its next refresh (`lookup_cache_refresh_secs`, 60 s by default). Every follower read of that tablet through that tserver goes to a remote replica for up to that long. This is pre-existing YSQL follower-read behaviour; this PR changes neither `read_query.cc` nor `tablet_rpc.cc`. A follow-up could return a `STALE_FOLLOWER`-coded status so the invoker retries without marking the replica failed, but that changes YSQL behaviour and needs its own discussion with the DocDB owners. ### Notes for the owners - An unpaged follower batch reports `used_read_time_ht = 0`, because the service only echoes the read time inside a paging state (pre-existing). A follow-up could echo it in an additive response field. - C ABI: `ybthin_read_async`'s signature changes in place instead of gaining a second entry point, as agreed with the owners. The library has one caller, which pins a shim release and ships the library in the same image as the binaries built against it, so the two never differ at run time. The header and README say this policy holds while the library has a single caller. `yb_thin_client.h` and the README said the ABI was additive-only; they now say it may change between shim releases, and that only the wire to the tserver is kept backward-compatible. A caller that declares the functions by hand has to update its declaration with the header, because a stale declaration of the old signature still links. ## Upgrade/Rollback safety No AutoFlag or gflag. Nothing is persisted. The wire change is two appended optional fields on `ThinReadTimeOptionsPB`, plus an existing field (`PgsqlPagingStatePB.read_time`) that is now set in one more case. A thin-client paging state is held in client memory, and its continuation is meant for the tserver process that issued it, where its session lives. Usually a restarted tserver answers `SESSION_LOST`: the shim reports that page as `YBTHIN_NETWORK`, and a retry of the continuation as `YBTHIN_READ_RESTART`. That is not guaranteed, though: session ids are a per-process counter, so a restarted tserver can hand the same id to a new session, and a continuation can then be served by a different process than its first page. - **Can an old tserver read what a new thin client sends?** Yes. It ignores both fields and serves the batch STRONG at the leader: unpinned for a fresh batch, at the pinned `read_time` for a continuation. A leader read at that time is never staler than the follower read that was requested, so the shim does not depend on the receiver honouring the fields. REVIEW.md asks for a gate only in that case. The header documents this fallback. - **Can a new tserver read what an old thin client sends?** Yes, and it behaves as before: an old shim sets neither field, so none of the new checks can fire. The one difference an old shim sees is the stamped `paging_state.read_time` at a tablet boundary of a batch whose read point was fixed (pinned, or a multi-tablet batch the batcher stamped). Old shims already read that field mid-tablet and pin the next page to it, so their scan now keeps its snapshot across that boundary instead of moving forward. - **Mixed versions inside the cluster.** The tserver-to-tserver hop is an ordinary `Read` RPC with the existing `consistency_level = CONSISTENT_PREFIX` and `read_time` fields, which every supported tserver already serves, as it does for YSQL follower reads. No new field crosses between tservers. - **Rollback after thin clients have started sending the new fields.** Nothing to undo, because nothing was written. A rolled-back tserver ignores the fields as above. Take a scan whose first page was a follower read on a new tserver: rolling that tserver back restarts it, so the continuation usually fails as above and the caller restarts the scan, which the old binary serves at the leader. If the continuation instead reaches the rolled-back process under a reissued session id, it carries the scan's pinned `read_time` (a new tserver always reports one for a follower page), which old tservers honour, plus `read_from_followers`, which they ignore: a leader read at the scan's snapshot. - **Upgrade with a scan in flight.** The reverse case: a scan whose first page an old tserver served at the leader, unpinned, ending at a tablet boundary, so its paging state holds no read time. Its continuation can reach an upgraded process under a reissued session id. The new tserver refuses `read_from_followers` without a read time, so the shim never sends the flag on a continuation whose paging state holds no read time, whatever the scan asked for: such a page could not be a follower read anyway. The request is then the same as before this PR, whichever process receives it (`FollowerContinuationWithoutAReadTimeReadsAtTheLeader`). - The shim's paging envelope bumps to version 4. It is process-local (it embeds the shim's session index and generation), so it crosses no process boundary. - The C ABI change is not additive: the fourth parameter of `ybthin_read_async` changes from `uint64_t read_time_ht` to `const ybthin_read_options*`. It has no upgrade or rollback consequence for a running cluster, because the library is not part of the tserver: its caller takes the new signature when it moves its pinned shim release, in the same build that compiles its binaries. ### Test Plan `yb_thin_client-itest`, `PgThinClientFollowerReadTest.*` (RF3 mini cluster). The follower assertions use the tablet's `consistent_prefix_read_requests` counter per replica, which `ReadQuery` bumps for every CONSISTENT_PREFIX read that reaches a replica (before the safe-time decision), so it is process-scoped and transport-independent; no `TEST_assert_reads_served_by_follower`. No fixed sleeps: the tests `WaitFor` the follower's `SafeTime(RequireLease::kFalse)` to pass the Perform tserver's clock minus the staleness, and the rows to be older than that. - `StaleReadIsServedByAFollower`: a Perform sent to a follower's tserver with the flag and a staleness returns the rows; that follower's counter rises by one, the leader's and the other follower's by zero. - `LaggingFollowerHandsTheReadOver`: the follower the Perform lands on has `UpdateConsensus` paused for longer than the staleness; the read still returns the rows well under the RPC deadline; the paused follower's counter rises by exactly one (the refused attempt) and exactly one other replica's rises by one. - `PagedFollowerScanHoldsItsSnapshot`: rows written between page one and page two are not seen; every page reports the same `used_read_time_ht`; continuations pass no options; the follower's counter rises by one per page, the others' not at all. - `PagedFollowerScanCrossesTablets`: a range-sharded table split at 10/100/150, with no leader on the Perform's tserver. Page one (limit 32) ends at the 10-row first tablet, so its paging state comes from `CreatePagingStateForRead`. The scan completes in 8 pages, every page reports page one's nonzero `used_read_time_ht`, rows written into the last tablet after page one are not seen, and the reader's replicas take exactly one consistent-prefix read per page, the other tservers none. Before the fix, page one reported 0 and the continuation failed with `YBTHIN_INVALID`. - `ReadWithoutOptionsStaysOnTheLeader`: `ybthin_read_async` with no options sees rows written a moment ago, and the leader's `Read` RPC count rises while the Perform's own tserver's does not. - `FollowerContinuationWithoutAReadTimeReadsAtTheLeader`: a leader scan of a table split at 10 ends its first page at the tablet boundary with no read time; the test sets the follower flag in the shim's envelope, as a first page served by an old tserver would leave it. The continuation succeeds and no replica takes a consistent-prefix read. Without the fix the server refuses it with `InvalidArgument`. - `ContinuationKeepsTheScansReplicaChoice`: a scan whose first page had no options is continued with follower options; every page succeeds and no replica's counter moves. - `StalenessOutsideAFollowerReadIsRejected`, `TooSmallStalenessIsRejectedByTheService`: batches built on the proxy directly get `InvalidArgument` for a staleness on a write batch, a staleness on a read batch without `read_from_followers`, a zero staleness, and a staleness of 1 ms and of one below `2 * max_clock_skew_usec`. - `ZeroStalenessIsInvalidAtTheShim`: `YBTHIN_INVALID` for a fresh follower batch with staleness 0; the same options pinned to an earlier batch's `used_read_time_ht` succeed. All 38 `yb_thin_client-itest` tests pass, including the four `PgThinClientExternal*` tests, which run tservers as separate processes. --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>

Commit:fcf9d5d
Author:Piyush Jain
Committer:GitHub

[#30797] YSQL: Revert "Set index birth_time for indexes built nonconcurrently" (#38235) ## Summary Reverts 4e76682e276ae29c20eaee89ca8019811ecad5b7 (#37103). With that commit, an index created inside a DDL transaction stays `BIRTH_TIME_PENDING_DDL_COMMIT` until the transaction commits, and the tserver read guard answers reads of it with a retryable `LeaderNotReadyToServe` error. A statement that reads the index in the same transaction that created it is therefore retried until the RPC deadline (about 600s) and fails. This was a known limitation (#37091) for which only three `yb.orig.ddl_txn_*` regress reads were disabled, but extension scripts and tests do the same thing, so these fail on every attempt on master: - `DocumentDBTest.*` and `BasicDocumentDBTests#testBasicOperations`: `CREATE EXTENSION documentdb CASCADE` installs pg_cron, creates an index on `cron.job` and reads `cron.job` in the same transaction. - `PgPartmanTest.*/FlagOn`: `BEGIN; CREATE TABLE ... UNIQUE; ALTER TABLE ... ADD FOREIGN KEY; INSERT`, where the foreign key check reads the new unique index. The revert is the exact inverse of the original commit. The change will be re-landed together with the fix for #37091. #30797 (silently incomplete results for a snapshot read that precedes a nonconcurrent index's commit) is open again until then. Test failures: #38144, #38145, #38179, #38180, #38193, #38194, #38195, #38197, #38198 ## Upgrade/Rollback safety This removes the AutoFlag `enable_ysql_index_birth_time_state`, the proto fields `IndexInfoPB.birth_time_state`, `IndexStatusPB.birth_time_state` and `ChangeMetadataRequestPB.mark_non_concurrent_index_done`, and the `MarkNonConcurrentIndexDone` tserver admin RPC. They exist only in master builds 2.33.0.0-b54 to b57; no release branch contains them. Moving a cluster that ran one of those builds with the AutoFlag promoted to a build with this revert is not handled; recreate such a cluster. ## Test plan - [x] `./yb_build.sh release --cxx-test pg_partman-test --gtest_filter 'PgPartmanTest.TestIdNative/FlagOn'` passes (258s). 🤖 Generated with [Claude Code](https://claude.com/claude-code) cloud.mekodata.ai://efcaf569-041e-4415-a0da-a4023c93197c;adlc/proj/yb_concurrency_control meko_conversation_id:c1be8b03a9da4a9ea78ebfe7d4e01570

Commit:563a75e
Author:Hari Krishna Sunder

Rust master layer 3, web UI: a Rust leader serves the leader pages A Rust leader answered every leader page with 503. The new crate master-web (src/yb/rust/master/web) serves them on a ready leader: - Every C++ /api/v1 JSON page and /dump-entities, byte-compatible with the C++ leader (json.rs ports the rapidjson-based JsonWriter, including JsonWriter::Protobuf). Column types and partition bounds come from a new host call, WebTableLayout. RustMasterTest.RustLeaderServesWebJson diffs every page against a C++ leader. - JSON for the pages C++ only renders as HTML: /api/v1/overview, /api/v1/load-distribution, /api/v1/tasks, /api/v1/tablet-server-clocks. - A React/TypeScript web UI (master/web/ui) at every old page URL, built like yugabyted-ui: yb_build.sh --build-master-ui runs npm and Vite into $BUILD_ROOT/master-ui, which master-web embeds. Without it the JSON is served and the page URLs say the UI was not built. On a master that is not a ready leader a route answers yb_web::PassToHost and the host fetches the page from the leader, as before. /ObjectLockManager is rendered by the host's lock manager. webserver_password_file, webserver_authentication_domain and webserver_redirect_http_to_https are no longer refused: yb-web does squeasel's HTTP Digest authentication and HTTP-to-HTTPS redirect. (The C++ Webserver passes the password file to squeasel under an option name squeasel rejects, so a C++ server with that flag does not start.) master_path_handlers-itest's HTML-table helpers read the JSON when the leader is a Rust master. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015zgRh2KXCyjcsuAeffjY4n

Commit:bae76ff
Author:Ella Baron
Committer:Ella Baron

tserver: send a tablet's storage metadata and full compaction status to the master only when changed Both per-tablet sections were resent in every metrics heartbeat. The tserver now keeps what the master confirmed and sends an entry only when it differs; the master sets tablet_metadata_processed in the response once it applied every entry, so an entry that found no replica is resent. One flag, tserver_heartbeat_metrics_send_tablet_metadata_on_change, gates both.

Commit:3ccb716
Author:Vivian Dai
Committer:Vivian Dai

[BACKPORT 2026.1][#33766] DocDB: Prefer a remote bootstrap source that has finished post-split compaction Summary: While a split child has outstanding post-split compaction (PSC), the tserver reports `should_disable_lb_move` for that replica and the cluster balancer won't move the tablet off that tserver. The check is per tserver so once the replica on the source tserver finished PSC, the tablet could be moved even if other replicas hadn't. The Raft leader picks the remote bootstrap source (itself or the closest peer, per `remote_bootstrap_from_leader_only`), so the new replica could be copied from one that still holds the parent's data and then has to run PSC itself and the parent's data is copied and compacted a second time. This change makes the leader prefer a remote bootstrap source that has finished PSC: - Followers set a new `pending_post_split_compaction` field in their UpdateConsensus response while PSC is outstanding, using the same check as `should_disable_lb_move`. The leader gets its own value from its tablet. - When choosing the remote bootstrap source, the leader prefers one that has finished PSC, but won't pick one outside the new replica's region for it. When no replica has PSC outstanding, it picks the closest eligible peer, with the leader winning ties, as it did before this change. This applies to every remote bootstrap, including moves off blacklisted or misplaced tservers and replacing missing replicas. The cluster balancer is unchanged, so a replica that is slow to finish PSC only blocks moves off its own tserver, not the tablet's other replicas. - As this is best effort, the new replica can still get the parent's data when no replica in its region has finished PSC, or when `remote_bootstrap_from_leader_only` is set and the leader hasn't finished PSC. - After a config change, the master rebuilt the tablet's replica list from one tserver's report and reset every other replica's `should_disable_lb_move` to false. It now keeps their previous values. - The PSC check runs on each UpdateConsensus response and when the leader picks a remote bootstrap source. The field is only sent when set. **Upgrade/Rollback safety:** The new `pending_post_split_compaction` response field is optional, so an older follower never sets it and is treated as having finished PSC, which matches current behavior, while an older leader ignores it. No AutoFlag is needed, since a mixed-version cluster only affects which peer is picked as the remote bootstrap source. No new gflags. Merge conflicts: - src/yb/consensus/consensus_queue.cc:1102 — 2025.2 takes queue_lock_ in an inner scope in GetRemoteBootstrapRequestForPeer and fills the request outside it; kept that structure, read PendingPostSplitCompaction() before the scope and set local_peer_->pending_post_split_compaction first thing inside it - src/yb/consensus/test_consensus_context.h:53 — took only this change's setter and member; dropped CheckReplicateAllowed/ShouldRelinquishLeadership overrides, which aren't in 2025.2's ConsensusContext - src/yb/master/tablet_split_manager.cc:542 — kept 2025.2's AllReplicasHaveFinishedCompaction; on master this change only reworded a comment in GetReplicasWithOutstandingCompaction, which 2025.2 doesn't have Original commit: 5bf5ef5e7fa2fb02a8a1909a20295f9e40a61098 / D58768 Test Plan: ``` ./yb_build.sh debug --cxx-test consensus_queue-test --gtest_filter ConsensusQueueTest.RbsSourceSkipsPeerWithPendingPostSplitCompaction ./yb_build.sh debug --cxx-test consensus_queue-test --gtest_filter ConsensusQueueTest.RbsSourceStaysInRegionDespitePendingPostSplitCompaction ./yb_build.sh debug --cxx-test consensus_queue-test --gtest_filter ConsensusQueueTest.RbsSourcePrefersCompactedLeaderInSameZone ./yb_build.sh debug --cxx-test consensus_queue-test --gtest_filter ConsensusQueueTest.RbsSourceSkipsLeaderWithPendingPostSplitCompaction ./yb_build.sh debug --cxx-test master_heartbeat-itest --gtest_filter MasterHeartbeatITest.ConfigChangePreservesOtherReplicasShouldDisableLbMove ./yb_build.sh debug --cxx-test remote_bootstrap-itest --gtest_filter RemoteBootstrapITest.TestRemoteBootstrapFromClosestPeer ./yb_build.sh debug --cxx-test tablet-split-itest --gtest_filter TabletSplitITest.TestLoadBalancerAndSplit ./yb_build.sh debug --cxx-test tablet-split-itest --gtest_filter AutomaticTabletSplitITest.RemoteBootstrapAfterPostSplitDoesNotHoldOutstandingSplit ``` Reviewers: bkolagani Reviewed By: bkolagani Subscribers: neera.mital, bkolagani, ybase Differential Revision: https://phorge.dev.yugabyte.com/D60038

Commit:c08e241
Author:Vivian Dai
Committer:Vivian Dai

[BACKPORT 2025.2][#33766] DocDB: Prefer a remote bootstrap source that has finished post-split compaction Summary: While a split child has outstanding post-split compaction (PSC), the tserver reports `should_disable_lb_move` for that replica and the cluster balancer won't move the tablet off that tserver. The check is per tserver so once the replica on the source tserver finished PSC, the tablet could be moved even if other replicas hadn't. The Raft leader picks the remote bootstrap source (itself or the closest peer, per `remote_bootstrap_from_leader_only`), so the new replica could be copied from one that still holds the parent's data and then has to run PSC itself and the parent's data is copied and compacted a second time. This change makes the leader prefer a remote bootstrap source that has finished PSC: - Followers set a new `pending_post_split_compaction` field in their UpdateConsensus response while PSC is outstanding, using the same check as `should_disable_lb_move`. The leader gets its own value from its tablet. - When choosing the remote bootstrap source, the leader prefers one that has finished PSC, but won't pick one outside the new replica's region for it. When no replica has PSC outstanding, it picks the closest eligible peer, with the leader winning ties, as it did before this change. This applies to every remote bootstrap, including moves off blacklisted or misplaced tservers and replacing missing replicas. The cluster balancer is unchanged, so a replica that is slow to finish PSC only blocks moves off its own tserver, not the tablet's other replicas. - As this is best effort, the new replica can still get the parent's data when no replica in its region has finished PSC, or when `remote_bootstrap_from_leader_only` is set and the leader hasn't finished PSC. - After a config change, the master rebuilt the tablet's replica list from one tserver's report and reset every other replica's `should_disable_lb_move` to false. It now keeps their previous values. - The PSC check runs on each UpdateConsensus response and when the leader picks a remote bootstrap source. The field is only sent when set. **Upgrade/Rollback safety:** The new `pending_post_split_compaction` response field is optional, so an older follower never sets it and is treated as having finished PSC, which matches current behavior, while an older leader ignores it. No AutoFlag is needed, since a mixed-version cluster only affects which peer is picked as the remote bootstrap source. No new gflags. Merge conflicts: - src/yb/consensus/consensus_queue.cc:1102 — 2025.2 takes queue_lock_ in an inner scope in GetRemoteBootstrapRequestForPeer and fills the request outside it; kept that structure, read PendingPostSplitCompaction() before the scope and set local_peer_->pending_post_split_compaction first thing inside it - src/yb/consensus/test_consensus_context.h:53 — took only this change's setter and member; dropped CheckReplicateAllowed/ShouldRelinquishLeadership overrides, which aren't in 2025.2's ConsensusContext - src/yb/master/tablet_split_manager.cc:542 — kept 2025.2's AllReplicasHaveFinishedCompaction; on master this change only reworded a comment in GetReplicasWithOutstandingCompaction, which 2025.2 doesn't have Original commit: 5bf5ef5e7fa2fb02a8a1909a20295f9e40a61098 / D58768 Test Plan: ``` ./yb_build.sh debug --cxx-test consensus_queue-test --gtest_filter ConsensusQueueTest.RbsSourceSkipsPeerWithPendingPostSplitCompaction ./yb_build.sh debug --cxx-test consensus_queue-test --gtest_filter ConsensusQueueTest.RbsSourceStaysInRegionDespitePendingPostSplitCompaction ./yb_build.sh debug --cxx-test consensus_queue-test --gtest_filter ConsensusQueueTest.RbsSourcePrefersCompactedLeaderInSameZone ./yb_build.sh debug --cxx-test consensus_queue-test --gtest_filter ConsensusQueueTest.RbsSourceSkipsLeaderWithPendingPostSplitCompaction ./yb_build.sh debug --cxx-test master_heartbeat-itest --gtest_filter MasterHeartbeatITest.ConfigChangePreservesOtherReplicasShouldDisableLbMove ./yb_build.sh debug --cxx-test remote_bootstrap-itest --gtest_filter RemoteBootstrapITest.TestRemoteBootstrapFromClosestPeer ./yb_build.sh debug --cxx-test tablet-split-itest --gtest_filter TabletSplitITest.TestLoadBalancerAndSplit ./yb_build.sh debug --cxx-test tablet-split-itest --gtest_filter AutomaticTabletSplitITest.RemoteBootstrapAfterPostSplitDoesNotHoldOutstandingSplit ``` Reviewers: bkolagani Reviewed By: bkolagani Subscribers: ybase, bkolagani, neera.mital Differential Revision: https://phorge.dev.yugabyte.com/D60037

Commit:8690d55
Author:Ella Baron
Committer:Ella Baron

tserver: send a tablet's storage metadata and full compaction status to the master only when changed Both per-tablet sections were resent in every metrics heartbeat. The tserver now keeps what the master confirmed and sends an entry only when it differs; the master sets tablet_metadata_processed in the response once it applied every entry, so an entry that found no replica is resent. One flag, tserver_heartbeat_metrics_send_tablet_metadata_on_change, gates both.

Commit:d87cb52
Author:Ella Baron
Committer:Ella Baron

master: hand tservers the tserver list as TSHeartbeat deltas Every TSHeartbeat response carried the full live tserver list; at 2000 tservers that was 451 MB/s out of the master leader and most of the TSHeartbeat handler time. The master now versions its list (leader term in the high 32 bits, a per-leader change counter in the low 32), each tserver echoes the version it holds, and the response carries only the tservers stamped after it plus tombstones for removals. A term change, an unknown version, one older than the tombstone horizon (master_tserver_list_history_sec, 600) or one ahead of the master's counter gets a full list.

Commit:4e76682
Author:Piyush Jain
Committer:GitHub

[#30797] YSQL: Set index birth_time for indexes built nonconcurrently (#37103) ## Summary A CREATE INDEX run inside a transaction block is built nonconcurrently and skips the backfill that sets the index birth_time, so birth_time stays 0. The docdb read guard then does not throw a kSnapshotTooOld error for a snapshot read whose read time precedes the index commit, and the index rows (written at the commit) are not visible to that read, so it silently returns an incomplete result. This is the data loss seen in the Jepsen si.append-table test. On a successful nonconcurrent index create, set the index birth_time to the master clock time (>= the DDL commit time, see below) and propagate it to the tablets via a new MarkNonConcurrentIndexDone tserver admin RPC. To reject reads until birth_time is final, introduce a `birth_time_state` enum in the index metadata (IndexInfoPB), set by the master at index creation: - BIRTH_TIME_FINAL (default): birth_time, if set, is final. - BIRTH_TIME_PENDING_BACKFILL: online (backfilled) index, until the backfill completes. - BIRTH_TIME_PENDING_DDL_COMMIT: nonconcurrent index created in a DDL transaction, until the transaction commits. The backfill path was relying on the `retain_delete_markers` flag as a proxy for this; post this change it can rely on `birth_time_state` too (both are checked). The read guard returns two different errors: - While the state is pending on the tablet (birth_time 0): `LeaderNotReadyToServe`. The tserver's client retries it until the RPC deadline, so the read waits until the tablet is finalized. - Once birth_time is set, for a read whose read time is before it: `SnapshotTooOld` (SQLSTATE 72000). The query layer does not retry it, so the application has to retry the transaction. Details: - Online indexes: the state is cleared together with setting birth_time when the backfill completes (master AllowCompactionsToGCDeleteMarkers, tablet BackfillDone). They never take the nonconcurrent path. - Nonconcurrent indexes: the DDL transaction verifier (HandleSuccessfulYsqlDdlTxn) sets birth_time, clears the state and posts AsyncMarkNonConcurrentIndexDone (new monitored task type) to each tablet. The tablet replicates a ChangeMetadata op with the new mark_non_concurrent_index_done field, which sets birth_time and clears the state. This is separate from BackfillDone / mark_backfill_done so that the two paths are not conflated, and each path only clears its own pending state. A finalize op for an index already removed from the tablet (e.g. a DROP INDEX on a colocated table racing with the RPC) is skipped instead of failing the replicated op; this also applies to BackfillDone. - birth_time >= commit time: the transaction coordinator picks the commit time, so the master's clock is moved past it before birth_time is picked. ReportYsqlDdlTxnStatus now carries the reporting tserver's hybrid time (which the commit response moved past the commit time), and when the master polls the transaction instead, its clock is updated from the GetTransactionStatus response. - Recovery of a lost finalize RPC: the master does not resend it (e.g. after a master failover or when a task gives up). The tablet validator also checks indexes whose local state is not final: a tablet is queued when it opens, and at the next periodic run (`tablet_validator_retain_delete_markers_validation_period_sec`, 60s by default) the validator asks the master (GetBackfillStatus now returns the master's birth_time_state) and finalizes the tablet metadata once the master reports the index final. Reads through the index retry until then. The validator asks only once per tablet open, so if that check ran while the index was still pending on the master, a later lost RPC is not repaired until the tablet is reopened, and not even then within an hour, since the validator reuses its cached "pending" answer for that long. The same gap already exists for BackfillDone; both are tracked in #37622. - xCluster targets: the birth time is not propagated yet (#33155). However, the target re-creates the index via DDL replication, so it will set the pending state and clear it on commit to allow reads (without stamping a local birth_time, so it stays 0 as before). - Known limitation: statements in the transaction that created a nonconcurrent index and read that index (index scans, `INSERT ... ON CONFLICT` on a unique index, foreign key checks against a UNIQUE constraint, `yb_index_check()`) are retried until statement_timeout, since the index only becomes final once that same transaction commits. This includes `CREATE TABLE ... UNIQUE` followed by an upsert in the same transaction. Tracked in #37091, to be fixed before a release containing this change. The three regress reads that do this (yb.orig.ddl_txn_{triggers,event_triggers,indices}) are commented out until then. Fixes #30797 ## Upgrade/Rollback safety - New AutoFlag enable_ysql_index_birth_time_state (kLocalPersisted). The master sets a non-final birth_time_state only when it is on; while it is off, behavior is unchanged (every index is final). It is an AutoFlag so that no index is left pending forever: an old master that becomes leader before the CREATE INDEX transaction is verified would not finalize the index, and after a rollback, indexes that were still pending would stay pending with nothing to finalize them on re-upgrade. - The new fields (IndexInfoPB.birth_time_state, IndexStatusPB.birth_time_state, ChangeMetadataRequestPB.mark_non_concurrent_index_done) and the new RPC are only used for indexes with a non-final state, which exist only once the AutoFlag is promoted, i.e. after all tservers run the new version. Older tservers never receive them. - A missing IndexInfoPB.birth_time_state means final, so indexes created before the upgrade keep the current behavior. An older binary that reads persisted metadata containing the field ignores it, and only loses the new read guard. A GetBackfillStatus response without IndexStatusPB.birth_time_state (from an older master) is treated as not final, so the tablet validator does not finalize anything from it. - New optional ReportYsqlDdlTxnStatusRequestPB.propagated_hybrid_time: an older master ignores it and an older tserver does not send it, which is the behavior before this change in both cases. ## Test plan - [x] Ran (release): ``` ./yb_build.sh release --cxx-test pgwrapper_pg_concurrent_ddls-test --gtest_filter 'PgConcurrentDDLsTest.SnapshotRead*:PgConcurrentDDLsTest.NonConcurrentIndex*:PgConcurrentDDLsTest.PendingIndexRejectsStaleRead:PgConcurrentDDLsTest.OnlineIndex*' ./yb_build.sh release --cxx-test pgwrapper_pg_index_backfill-test --gtest_filter 'PgIndexBackfillTest.RetainDeleteMarkers*:PgIndexBackfillColocated.ColocatedRetainDeleteMarkers*' ./yb_build.sh release --cxx-test master_multi_step_monitored_task-test ./yb_build.sh release --java-test 'org.yb.pgsql.TestDdlTransactionBlocks' ``` New tests in pg_concurrent_ddls-test: SnapshotReadFreshNonConcurrentIndexMatrix, PendingIndexRejectsStaleRead, OnlineIndexRejectsStaleReadBeforeBackfillDone, OnlineIndexBirthTimeFinalOnMaster, NonConcurrentIndexWithDropColumnInSameTxn, NonConcurrentIndexFinalizeAfterDropIndex.

Commit:1ef2d1b
Author:Hari Krishna Sunder

Rust master layer 3, W11: the xCluster target, failover and the RPC coverage gate A Rust leader now serves the target universe's side of xCluster, so DB-scoped replication runs end to end between Rust-led universes. - Inbound replication groups (master-xcluster): SetupUniverseReplication and its IsDone, Alter/Delete/SetEnabled, Get*, GetReplicationStatus, the safe time RPCs, the consumer registry updates sent by source tservers, persistence blocking, AutoFlags revalidation, the automatic-mode schema RPCs and the legacy BootstrapProducer/IsBootstrapRequired/WaitForReplicationDrain. Setup tasks, replication status and safe-time state stay in memory, as in C++. - New target tables join their DB-scoped group in post-tablet-create work; table-lifecycle keeps a post-create failure and IsCreateTableDone returns it with done=true. disable_xcluster_db_scoped_new_table_processing skips the work. - The controller runs on every catalog change (dropped target tables, split parents, hidden tables' streams), with safe time on its interval. - XClusterFailover / IsXClusterFailoverDone, through a FailoverSnapshots trait over the snapshot coordinator and split manager; a failover cut off by a leader change is marked aborted. - The source fails on every error the target returns: XClusterClient's "ignore AlreadyPresent" branch is dead in C++ because SyncLeaderMasterRpc already fails on response errors. - A dropped database's persistence block entry is erased in the write that marks it DELETED. - A namespace snapshot waits on the sys catalog leader's MVCC safe time (RequireLease::kTrue, as C++ WaitForSafeTime does). The follower path it used never advances on a single master without sys catalog writes, which hung failover's CreateSnapshot. - PITR on an automatic-mode target skips the sys catalog restore (RestoreHooks::is_automatic_xcluster_target). - Tablet splitting of xCluster tables follows enable_tablet_split_of_xcluster_replicated_tables and enable_tablet_split_of_xcluster_bootstrapping_tables instead of refusing. - New host calls in master_rust_peer.proto for the C++ helpers the target reuses (tablet mapping, schema equivalence, AutoFlags, safe time table, split remapping, historical packings). - rpc-handlers' every_master_method_is_served gates RPC coverage. Never served, with reasons in leader::NEVER_SERVED: the YEDIS methods and SetupNamespaceReplicationWithBootstrap(+IsDone), whose flow needs the C++ SnapshotTransferManager and whose tests are all disabled. Test plan: - cargo fmt --check, clippy -D warnings, cargo test --workspace - build-support/lint.py on the changed files - YB_RUST_MASTER=1 xcluster_db_scoped-test: 32 of 36 pass. Delete, CreateReplicationWithHiddenTables and both AddRemoveNamespace variants read CatalogManager from the test body and wait for W14. DeleteWhenTargetIsDown timed out on the target's safe time in two runs before the safe-time fix and passed after it. - YB_RUST_MASTER=1 xcluster_failover-test: all 11 pass. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015zgRgxsdPtsouTvhhiWf55

Commit:698ba54
Author:Hari Krishna Sunder

Rust master layer 3, W11: xCluster outbound replication groups on the source A new crate, master-xcluster, serves the source side of DB-scoped xCluster on a Rust leader: XClusterCreate/Delete OutboundReplicationGroup, XClusterAdd/RemoveNamespace..., IsXClusterBootstrapRequired, GetXClusterStreams, GetXClusterOutboundReplicationGroups(Info), the repair RPCs, DeleteXClusterWalAnchorStreams, XClusterEnsureSequenceUpdatesAreInWal, PauseResumeXClusterProducerStreams, and CreateXClusterReplication with its IsCreate/AddNamespaceTo/IsAlter companions, which call the target. A group is its XCLUSTER_OUTBOUND_REPLICATION_GROUP row, rebuilt from the latest snapshot and written with the stream rows it creates. The xcluster controller starts one checkpoint task per CHECKPOINTING namespace per term: in automatic mode it creates sequences_data, advances the OID counters, creates the DDL replication extension and sets the namespace's role; it then creates INITIATED streams and has a tserver bootstrap them (BootstrapProducer) until the namespace is READY. A table created in a group's namespace gets its stream, checkpointed at op 0, before it runs: the assigner calls the new XreplHooks::post_tablet_create, which master-xrepl forwards to master-xcluster (XreplExtension). A dropped table with xCluster streams stays hidden until its streams are deleted or it has been hidden for cdc_wal_retention_time_secs. The other universe's masters are reached through a new host call, XClusterRemoteCall (K9): the host keeps an XClusterRemoteClientHolder per replication group, with its certificates, and forwards serialized requests; Rust retries leader changes. The xCluster config's producer registry, namespace roles and the target's safe times go to tservers by heartbeat. MiniCluster::WaitForLoadBalancerToStabilize and XClusterTestBase::GetCDCStreamForTable ask a Rust leader over RPC rather than through its CatalogManager. The target side (inbound replication groups) is not served yet. New flags read: enable_xcluster_api_v2, xcluster_enable_ddl_replication, max_xcluster_streams_to_checkpoint_in_parallel, xcluster_ysql_statement_timeout_sec, enable_xcluster_wal_anchor_stream(_infra), xcluster_enable_target_applied_filter, skip_fields_moved_to_xcluster_guarded_info and the TEST flags of these paths. Test plan: - cargo fmt --check, clippy -D warnings, cargo test --workspace - build-support/lint.py on the changed files - With YB_RUST_MASTER=1, xcluster_db_scoped-test CreateTable: the source's namespace is checkpointed to READY through the tservers; the test then fails at the target, whose SetupUniverseReplication is not served yet Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015zgRgxsdPtsouTvhhiWf55

Commit:379844b
Author:Hari Krishna Sunder

Rust master layer 3, W11: CDCSDK table membership, splits and the remaining CDCSDK RPCs Serves the rest of the CDCSDK side of MasterReplication on a Rust leader: GetUDTypeMetadata and GetTableSchemaFromSysCatalog (which tservers call while streaming), RemoveUserTableFromCDCSDKStream, RemoveTablesFromCDCSDKStream, ValidateAndSyncCDCStateEntriesForCDCSDKStream and CleanupStaleCDCStreams. Tables join and leave streams by reconciling the catalog each xrepl pass, with the rules C++ rebuilds its in-memory queues from at load. Work inside DDL reaches master-xrepl through an XreplHooks trait in table-lifecycle: CreateTablet sets cdc_sdk_set_retention_barriers for logical replication streams and the replica's safe op id becomes the new tablet's cdc_state rows; a split registers the children's rows; a split parent of a streamed table is hidden and kept until every stream is done with it. Tables in CDCSDK streams can therefore be split (xCluster-streamed ones still not), subject to enable_tablet_split_of_cdcsdk_streamed_tables and enable_tablet_split_of_replication_slot_streamed_tables. With enable_table_rewrite_for_cdcsdk_table, a dropped table that a logical replication stream has is hidden (Retainer::active_cdcsdk) until each stream's row is gone or its slot passed the hide time, as C++ does; it stays in the streams until its tablets are deleted. Colocated tables are not retained. TRUNCATE and rewrites (old_rewrite_table_id) of streamed tables apply CDCSDKAllowTableRewrite; DROP DATABASE marks the database's CDCSDK streams DELETING. The host registers the master's CDC service under ysql_yb_enable_implicit_dynamic_tables_logical_replication, through which logical replication reads the PG catalog tables from the sys catalog tablet; master-xrepl enables it once a CDCSDK stream exists. Without it, GetChanges on the sys catalog tablet failed ("Service yb.cdc.CDCService not registered on Master"). Stream creation follows C++'s batching (cdc_create_stream_alter_table_dispatch_batch_size/_delay_ms), bounds each batch by the RPC deadline (a late tablet rolls the stream back), and rolls back any failure after the row is written. ListCDCStreams filtered by the id_type proto default when the request left it unset, hiding every CDCSDK stream from such requests; C++ filters only on an explicit id_type. Ineligible tables (an index added under TEST_cdcsdk_add_indexes_to_stream) leave streams; as C++, they are looked for once per leader term. Split children get rows only for streams not removing the table. The C++ test sync points of these paths, and DeleteTableInternal::End, reach the Rust leader through a host call (TestSyncPointCallback), only under TEST_running_test, so itests that inject failures or order themselves with LoadDependency run unchanged. CDCSDKYsqlTest::SplitTablet sends MasterAdmin.SplitTablet to a Rust leader instead of calling its CatalogManager. Host: ReadUDTypeMetadata, ReadTableSchema, EnableCDCService, TestSyncPointCallback; CdcStateRead returns every column, and every row without a stream id. New flags read: cdc_create_stream_alter_table_dispatch_batch_size, cdc_create_stream_alter_table_dispatch_delay_ms, cdcsdk_table_processing_limit_per_run, cdcsdk_enable_dynamic_table_addition_with_table_cleanup, cdc_intent_retention_ms, enable_truncate_cdcsdk_table, enable_table_rewrite_for_cdcsdk_table, enable_tablet_split_of_cdcsdk_streamed_tables, enable_tablet_split_of_replication_slot_streamed_tables and the TEST_cdcsdk_skip_* removal flags; max_concurrent_alter_table_rpcs is not applied. Test plan: - cargo fmt --check, clippy -D warnings, cargo test --workspace - build-support/lint.py on the changed files - With YB_RUST_MASTER=1 (failures that need xCluster or call C++ master internals, W14, are listed): - cdcsdk_consistent_snapshot-test: 50/50 (rollback and late-alter tests rerun on the final build) - cdcsdk_stream-test: 18/19 (CDCWithXclusterEnabled: xCluster) - cdcsdk_tablet_split-test: 57/57 run (the two that call XReplValidateSplitCandidateTable on CatalogManager excluded) - cdcsdk_consumption_consistent_changes-test: 9 hidden-table, drop and rewrite tests pass; 3 call MiniMaster::catalog_manager() - cdcsdk_ysql-test: 17 truncate, rewrite, drop-table, removal, ineligible-table and split-during-removal tests pass; 4 DB-drop restart tests and TestPopulationOfDroppedTableListInStreamMetadata call C++ master internals Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015zgRgxsdPtsouTvhhiWf55

Commit:d692db8
Author:Hari Krishna Sunder

Rust master layer 3, W11: CDC streams and the CDCSDK stream lifecycle The new master-xrepl crate serves the CDC stream RPCs on a Rust leader: GetCDCStream, GetCDCDBStreamInfo, ListCDCStreams, UpdateCDCStream, DeleteCDCStream, DisableDynamicTableAdditionOnCDCSDKStream and CreateCDCStream for CDCSDK streams (logical replication slots and gRPC streams, with and without a consistent snapshot). CreateCDCStream of an xCluster stream answers NotSupported until the xCluster part of W11. The cdc_state table stays behind the host's cdc::CDCStateTable (new host calls CdcStateUpsert, CdcStateRead, CdcStateDelete, CdcStateDeleteStreams, CdcStateCreateTableRequest). Retention barriers are AlterSchema RPCs the crate sends itself; with a consistent snapshot option each tablet's answer becomes its cdc_state checkpoint, and catalog tables on the sys catalog tablet get theirs from a CHANGE_METADATA_OP the host submits (SetCatalogRetentionBarrier). An xrepl controller deletes DELETING streams (cdc_state rows, then the rows, the xCluster config's paused streams and outbound groups) and removes dropped tables from CDCSDK streams. Also fixes two W10 clone bugs found by the full yb-admin-snapshot-schedule-test run: - GetBackupPgTableInfo without a read time read the PG catalog at time 0, so ImportSnapshotMeta and YSQL clones never matched a table by schema name; it now reads the latest. - Repartitioned (cloned) tablets get SetupTabletInfo's initial consensus state, so seeding their peers writes a valid row. Test plan: - cargo fmt --check, cargo clippy --workspace --all-targets -D warnings, cargo test --workspace (258 passed); build-support/lint.py clean. - YB_RUST_MASTER=1 cdcsdk_stream-test: 18 of 19 pass; CDCWithXclusterEnabled needs xCluster stream creation. - YB_RUST_MASTER=1 yb-admin-snapshot-schedule-test: CloneYcql, CloneConcurrentClearMetacacheCallbacks, CloneAbortsWhenClearMetacacheFails, CloneAbortsWhenEnableDbConnsFails, SysCatalogRetentionWithClone, CreatingSnapshotMarkerHealedAfterFailover; the first 66 tests of the full suite passed apart from those clone tests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015zgRgxsdPtsouTvhhiWf55

Commit:663000c
Author:Hari Krishna Sunder

Rust master layer 3, W10: snapshots, PITR, import, clone and encryption A Rust leader now serves every MasterBackup RPC and ChangeEncryptionInfo. The new master-snapshot-coordinator crate leads what every master's C++ MasterSnapshotCoordinator applies (K5): SysCatalogTap passes that coordinator leader term -1, and the Rust leader submits CREATE/DELETE_ON_MASTER and RESTORE_SYS_CATALOG through a new host call, sends TabletSnapshotOp itself and writes the snapshot, schedule, restoration and clone rows as UPDATEs the C++ apply half decodes. ImportSnapshotMeta and CloneNamespace run in Rust; schema comparison, PG catalog reads and reading a snapshot's sys catalog files stay in the host. The W10 refusals in table-lifecycle and the split manager are gone: drops of tables a schedule or snapshot needs, and retired split parents, are hidden and later cleaned up by the coordinator. Behaviour matched to C++ that the tests exposed: - Tablet RPCs propagate Clock::Now; MaxGlobalNow only gives snapshot times. Propagating it pushed tserver clocks forward until they crashed on clock skew. - After a catalog reload while leading, every tserver sends a full tablet report; a PITR restores tablets once those reports were processed (at most 10 s), so deleted split children do not serve stale reads. - Heartbeats set snapshots_info.last_restorations_update_ht (the host's coordinator never does), so tablets leave the restore state without RESTORE_FINISHED. - Fresh tablets (split children, clones) get hinted elections for 5 minutes. - A clone's schema-version bump on PREPARING tables is pushed when they become RUNNING. New flags read: enable_db_clone, ysql_clone_pg_schema_rpc_timeout_ms and the snapshot coordinator, import and TEST flags of the C++ coordinator (wiring.rs). Test plan: - cargo fmt --check, cargo clippy --workspace --all-targets -D warnings, cargo test --workspace (249 passed); build-support/lint.py clean. - YB_RUST_MASTER=1 yb-admin-snapshot-schedule-test, passing: Basic, Delete, EditInterval, EditIntervalAndRetention, ListRestorationsAfterFailover, CreatingSnapshotMarkerHealedAfterFailover, UndeleteIndex, AlterTable, ConsistentRestore, ConsistentTxnRestore, ConsistentRestoreFailover, CreateDuplicateSchedules, UndeleteTable, UndeleteTableWithRestart, CleanupDeletedTablets, TestGCHiddenTables, DropKeyspaceAndSchedule, RestoreToBeforePreviousRestoreAt, RestoreToAfterPreviousCompleteAt, DisallowForwardRestore, BeforeCreateFinishes, TestTruncateDisallowedWithPitr, DroppedTableMarkedHiddenAfterFailover, RestoreAfterSplit, SysCatalogRetention, UndeleteIndexToBackfillTime, RestoreDuringBackfill, DDLsDuringRestore, DeleteIndexOnRestore, VerifyRestoreWithDeletedTablets, SplitDisabledDuringRestore, ReadsWithSnapshotScheduleAndSplit, WritesAfterRestoreWithSplit, CatalogLoadRace, TestSnapshotBootstrap, PgsqlSingleColumnUpdate, PgsqlTestMonotonicSequence, PgsqlTestPerDbCatalogVersion, CloneYcql, DeleteRowsFromCloneYcql. - PgsqlRestoreDroppedTableWithTablespace fails with C++ and Rust masters alike on the test host (every tserver under 5% free disk, so the balancer adds no replicas). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015zgRgxsdPtsouTvhhiWf55

Commit:c0946bb
Author:Bvsk Patnaik
Committer:Bvsk Patnaik

[#37328] YSQL: Tag legacy-mode fast path writes with their object lock owner Replays #37359's commits (074ed6508a4..eb4ffe3042a) as one commit on #37358's renamed request field (using_table_locks_globally) and infra flag (ysql_yb_enable_global_object_locking_infra): the fast path owner tag, PgPerformOptionsPB field 42 is_using_local_table_locks and its setter, the check that object lock requests come from a lock-taking transaction, and FastPathWriteDeadlockIsDetected. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Commit:c58274a
Author:Bvsk Patnaik
Committer:Bvsk Patnaik

[#37328] YSQL: Rename is_using_local_table_locks to using_table_locks_locally Match its neighbor using_table_locks_globally: the pair now reads as the same question at two scopes, set from UsingObjectLocksLocally() and UsingObjectLocksGloballyForCurrentTxn(). The tag stays 42, so the wire format does not change, and the field is new in this PR. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Commit:c8c1b02
Author:Bvsk Patnaik
Committer:Bvsk Patnaik

[#37328] YSQL: Say what the object lock request check catches The check in DoAcquireObjectLock fires when a lock request comes from a transaction that does not use object locks locally, which means a pggate sender skipped UsingObjectLocksLocally(). Its message, copied from master, said "Table Locking feature not enabled.", which points at the gflags; in release builds that text reaches the client. Say what the check found instead. State the field's contract in its proto comment: pggate sets it from the same predicate that decides whether the transaction sends its locks. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Commit:6902424
Author:Hari Krishna Sunder

Rust master layer 3, W7/W9: YCQL indexes, roles and UDTs, vector indexes, YSQL splits Finishes the leader RPCs of W7-W9 on a Rust master, and fixes what the pg_regress schedules found. YCQL: - Secondary indexes: create (DELETE_ONLY with backfill on), the permission steps in the `table-alters` controller (`ClassifyIndexes`, `ApplyYcqlActions`: next permission, delete at INDEX_UNUSED, batched and deferred backfills, a former leader's job resumed alone), DROP INDEX through the removing permissions, `LaunchBackfillIndexForTable`, `GetBackfillJobs`, `IsCreateTableDone`/`IsDeleteTableDone` for indexes, and index requests from clients before 2.0 (`IndexInfoBuilder`). - Roles and permissions (`roles.rs`): CreateRole, AlterRole, DeleteRole, GrantRevokeRole, GrantRevokePermission, GetPermissions; creators get permissions on new keyspaces, tables and roles, and drops remove them. - UDTs (`udtypes.rs`): CreateUDType, DeleteUDType, GetUDTypeInfo, ListUDTypes. YSQL: - Vector indexes, joined to the indexed table's tablets as colocated tables, with flush of vector indexes only and the partition list version moved by splits. - Tablet splitting of YSQL tables, with C++'s remaining candidate checks (uncommitted tables, vector indexes, sequences, xCluster DDL tables, partitioning version, backfill jobs). - Backfill row counts for `reltuples`, and the requester liveness check. - `GetTabletsMetadata`; the sequences table is not user-created. - Lock releases carry catalog versions and invalidation messages, and wait for the transaction's DDL verification. - `AddTransactionStatusTablet` and transaction table autoscaling; legacy colocated databases, including joining a tablet whose row lists its tables; PG-layer verification of tables created without DDL atomicity. Admin: `DumpState` on every master, `DdlLog` (DDL log rows on drops and alter steps), `GetCompactionStatus`, `AreNodesSafeToTakeDown`, `GetMasterHeartbeatDelays`, `GetObjectLockStatus`, `TestRetry`, `AccessYsqlBackendsManagerTestRegister`, and the live/dead tserver and follower heartbeat delay metrics. Fixes: - An alter aborts other transactions only for the version a YSQL `AlterTable` request made, not index or verifier alters. - Tablets of a table dropped by a rollback to a savepoint are deleted without aborting the DDL's own transaction. - A backfill finishing after a drop no longer brings the table back to ALTERING. - A read replica placement with no tservers gets no observers instead of failing assignment. - Colocation ids follow `TEST_sequential_colocation_ids` and `TEST_colocation_ids`. - Master errors always carry the `errors` terminator, so a QLError status reaches CQL with its message. - Table locations report each replica's state. - Alters finish on the last tablet acknowledgement, cascade drops to indexes, and status messages carry C++'s ": CODE" suffix. Tested on Rust masters: the 21 pg_regress schedules except as noted in testing.md, pg_ddl_atomicity-test (external), TestPgRegressTabletSplit, TestPgRegressSplitOptions, the pgvector schedule, TestRoles, TestUserDefinedTypes, TestAuthorizationEnforcement, TestIndex except testTruncate (52 of 53; testOptimizedIndexUpdate_Transactional failed once and passed on two reruns; testTruncate exceeds this machine's per-user thread limit on C++ masters too), AdminCliTest.DdlLog, yb-admin-are-nodes-safe-test, tablet_health_manager-itest, and the Rust unit tests. Not here: the W10 snapshot tests, and W14 MiniCluster tests (cql-index-test, PgDdlAtomicity*MiniClusterTest). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015zgRgzB6ta2bvKQKKQNe7m

Commit:ad6f5e1
Author:myang2021
Committer:GitHub

[#34309] YSQL: Send a connection away from a fill of its own session (#37789) ## Summary Problem: Two TODO(#34309) markers left by 510da0f5d37, independent of each other. A connection start's prefetch asks the tserver not to queue it behind a fill already under way, by setting send_away_if_fill_in_flight on the request. AnotherFillInFlight, the check behind that ask, counted only fills whose session id differs from the asker's, since the session id is the only handle on "another connection" a Perform carries. Parallel workers adopt their leader's session id and each run a connection start of their own, so the workers of one query -- siblings -- all prefetch under a single session id. A sibling's fill therefore carries the asking worker's own session id, and AnotherFillInFlight skipped it as the asker's own rather than counting it. A prefetch is several paged rounds, so the second worker waits out the rounds the first still has in flight instead of reading the version this node already holds -- the wait 510da0f5d37 exists to remove. Fix: Count every fill in the key group, the asking session's own included. AnotherFillInFlight loses its session id argument and becomes FillInFlight, a plain lookup of the key group: [[nodiscard]] bool AnotherFillInFlight(KeyGroup group, uint64_t session_id) REQUIRES(mutex_) { auto it = fills_in_flight_.find(group); if (it == fills_in_flight_.end()) { return false; } return std::ranges::any_of( it->second, [session_id](const auto& e) { return e.first != session_id; }); } becomes [[nodiscard]] bool FillInFlight(KeyGroup group) REQUIRES(mutex_) { return fills_in_flight_.contains(group); } Skipping the asker's own session was meant to stop a connection being sent away from a fill of its own. Outside a parallel query it almost never had anything to skip. send_away_if_fill_in_flight is set only when the master leader is not shedding, and setting it and starting at the stale read are exclusive branches, so a connection that asks begins at the trust_cache attempt. Its opening round is the one request that carries the ask, and it reaches this check before the connection has filled anything. A send-away clears send_away_if_fill_in_flight, and the connection then either reads the older version it was sent to or, where that could not be prepared, runs trust_cache a second time -- an attempt that no longer reaches this check. A connection to template1 is the exception. The auth prefetch fills under Template1DbOid with an ordinary token, and that token lives in its setter until the request object goes away, which is past the response that lets the backend send its opening round. The skip was hiding a fill of that connection's own, and counting it costs such a connection a stale attempt or a rerun, not a wrong read. Siblings are the case the skip decided wrongly, and the one this change is for. With the session filter gone nothing reads the session level of fills_in_flight_, so it becomes a count per key group, and session_id goes from FillToken, NoteFillStarted, NoteFillFinished, TakeFillToken and Get. What ties a caller to another's claim is sharing its token slot, which is what parallel workers do: prefetch_fill_token_ is a PgClientSession member and a worker adopts its leader's. Descriptions of the check that still gave the old rule now match it: the FillInFlight comment, the pg_response_cache_send_aways help text an operator reads, the YBCStartSysTablePrefetching contract and the send-away branch in YbRunWithPrefetcher. AConnectionIsNotSentAwayFromAFillOfItsOwn pinned that skip, asserting that a second round of one session is not sent away while the first is still filling. It becomes ASiblingIsSentAwayFromAFillOfTheSameSession, which asserts the opposite and reads those two requests as two siblings rather than one connection asking twice -- pggate asks on the opening round only. Their keys differ, so only FillInFlight can turn the second away, and what makes them siblings is now the token slot they share. AGroupStaysBusyAcrossOverlappingRounds reached the skip the same way, with SendAway::kTrue on its later round; that round now carries SendAway::kFalse, which is what pggate sends. FillInFlight counts the whole key group, so it already covers most callers that ask for the very entry being filled. The group can still be uncounted while that entry fills: the entry is marked as filling a moment before the setter takes the token that counts the group, and siblings share one token, so one sibling's completing round can release it while another's round is still filling -- a round's response is stored after the session lock is dropped, so the two overlap. A caller landing in either gap is turned away on the entry itself, which is what ACallerOnTheEntryBeingFilledIsSentAway covers. This change leaves that send-away alone, but the test no longer reached it: the caller asked on an entry its own session was filling, and FillInFlight now catches that first. Its setup is rebuilt on the token-release gap. The other TODO is unrelated to the check above. ComesUpWhenSentAwayWithNoUsableOlderVersion could not tell a connection that never failed from one whose retry succeeded. It now connects through PGConnSettings::should_stop, which ends the retry loop after the first attempt. ## Upgrade/Rollback safety No wire-format change. The only `.proto` edit is a comment on the existing `CachingInfoPB.send_away_if_fill_in_flight` (field 7); no field is added, removed, renumbered, or reinterpreted. That flag travels only between a node's own PG backends and its tserver, which come from the same binary, and the response cache it drives is local to that tserver. A mixed-version cluster therefore sees no interaction here: each node answers the flag with whatever check its own build carries. Rollback is a plain revert. The send-away decision is taken per request and leaves no persistent state, so there is nothing to migrate in either direction. ## Test plan ``` ./yb_build.sh release --skip-pg-parquet --sj --cxx-test pg_response_cache-test ./yb_build.sh release --skip-pg-parquet --sj --cxx-test pg_catalog_perf-test \ --gtest_filter 'PgStaleCatalogPrefetch*:PgUnbridgeable*:PgNarrowMessageQueue*' build-support/lint.sh --rev origin/master ```

Commit:f246663
Author:Bvsk Patnaik

[#36572] YSQL: Rename the global object locking flag and request field - ysql_yb_enable_object_locking_infra (GUC yb_enable_object_locking_infra) becomes ysql_yb_enable_global_object_locking_infra: it gates only global object locks now, since a transaction without it still takes local locks. The flag first landed in 7659e8b4f18 and is in no release branch (only master and the unreleased 2026.2), and previous_auto_flags.json does not list it, so it is renamed without a deprecated alias. - is_using_table_locks becomes using_table_locks_globally in the PgsqlWrite and PgsqlRead requests and the pg_client.proto options. It is set from UsingObjectLocksGloballyForCurrentTxn(): the transaction runs with global object locking, though it may hold only local locks itself, as a DML-only transaction does. Proto field names are not on the wire. - Field comments say that and what the tablet does with it: skip the catalog version check. - Drop the repeated list of backends that take no object locks from the YbUsingObjectLocksLocally comment; pggate.cc defines it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Commit:cffd7fa
Author:Hari Krishna Sunder

Rust master layer 3, W9: YSQL major catalog upgrade, DDL atomicity fixes A Rust leader now runs the YSQL major catalog upgrade and its rollback. Rust keeps the `YsqlInitDBAndMajorUpgradeHandler` state machine (`ysql-manager/src/major_upgrade.rs`) and the catalog DDL that initdb and pg_upgrade drive (`table-lifecycle/src/major_upgrade.rs`). The C++ host runs what is PG or process code: initdb in upgrade mode, pg_upgrade (C++'s `PerformPgUpgrade` is now the free function `PerformYsqlPgUpgrade`, shared by both masters), `CreateYsqlSysTable`, the catalog version and transaction steps, and the tserver version checks. Rust sends the host the upgrade state on every change, which gates catalog reads and writes as C++ does. The host also: - registers `PgClientService` over `MasterTabletServer`, since initdb runs PG with the master as its tserver, and answers PG's per-database catalog version lookup; - forwards `ClientUpgradeYsql` (`yb-admin finalize_upgrade`) to the closest live tserver; - removes the pg_upgrade socket directory a killed master left behind, at startup, as `Master::Init` does. During the upgrade a DELETING database is kept until the upgrade ends, and a namespace interrupted in `NEXT_VER_PREPARING` becomes `NEXT_VER_FAILED` at term start. `TEST_pause_before_upsert_ysql_sys_table` and `TEST_fail_yugabyte_namespace_creation_on_second_attempt` are honoured. The upgrade test base runs the old release on C++ masters and restarts into `yb-master-rs` with `YB_RUST_MASTER`. DDL atomicity: a DDL blocked by another transaction's verification gets `TABLE_SCHEMA_CHANGE_IN_PROGRESS` with the master error code encoded in its status (`tagged_error`), so PG retries it, and transaction ids are printed as UUIDs. `TEST_pause_ddl_rollback` holds the tables' DDL state, as in C++. Tests on Rust masters (`YB_RUST_MASTER=1`): ysql_major_upgrade-test 41/41, ysql_major_upgrade_rpcs-test 14/14, pg_ddl_atomicity-test 42 passed. The 3 `PgDdlAtomicitySnapshotTest` tests need snapshot schedules (W10), and the MiniCluster tests run C++ masters in process. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015zgRgzB6ta2bvKQKKQNe7m

Commit:e6a757b
Author:Bvsk Patnaik
Committer:Bvsk Patnaik

Merge origin/master into the #36572 layer 1 head Conflicts with 2fe29a63f0d (#36925: a transaction that starts with yb_read_time set runs without object locks). Resolved by keeping both latched facts: YBCSetObjectLockingForCurrTxn now also latches whether the transaction reads the catalog as of yb_read_time, and UsingObjectLocksLocally() is false for such a transaction, so it takes no local locks, reads the catalog the legacy way and keeps the tserver catalog version check, as #36925 intends. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Commit:f61e7bf
Author:Bvsk Patnaik

[#36572] YSQL: Keep the master names of the sequence and tablet request fields Restores is_using_table_locks on PgsqlWriteRequestPB, PgsqlReadRequestPB and the sequence requests; a15064887a6 renamed them skip_catalog_version_check. Field numbers never changed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Commit:43da497
Author:Bvsk Patnaik
Committer:Bvsk Patnaik

[#36572] YSQL: Key object locking on local locks and global object locks Rework the object locking predicates in pggate and PG around two questions: - IsObjectLockingEnabledForCurrentTxn(): does the transaction send object locks at all (only local DML locks unless it has global object locks)? The backend can take object locks (not initdb, binary upgrade or a parallel worker), enable_object_locking_for_table_locks is on, and the transaction is not an autonomous DDL. An autonomous DDL takes no object locks. This replaces the per-transaction local-lock latch and IsTakingObjectLocks(). - IsGlobalObjectLockingEnabledForCurrentTxn(): does the transaction also take global DDL locks? IsObjectLockingEnabledForCurrentTxn() and the latched yb_enable_object_locking_infra. enable_object_locking_for_table_locks keeps its master meaning again: off means no object locks at all. This reverts the hunks that decoupled the local lock manager, the lease, pg_locks and the tserver lock RPCs from it. YBCIsLegacyModeForCatalogOps() answers only how the transaction reads the catalog (through the legacy catalog session when it takes no object locks). The DDL-side callers (autonomous DDL, CREATE INDEX phases, ANALYZE, FK batching, catalog version locks, lmgr clamps, DATABASEOID flush) use the new YBCIsGlobalObjectLockingEnabledForCurrentTxn(). The CREATE INDEX WaitForLockers choice and the VALIDATE CONSTRAINT schema version skip now include the backend check, as pggate's WaitForLockers does. pg_cron's catalog check follows the refresh gate in AcceptInvalidationMessages, as it did on master. TestCatalogOpModeStableAcrossPromotion now expects the transactional catalog read path both before and after the infra flag is promoted under an open transaction: a transaction with object locks reads the catalog that way from its first statement, so the path cannot switch mid-way. Drop PgPerformOptionsPB.is_using_local_table_locks; the perform option keeps its master name is_using_table_locks. Comments say "with/without global object locks" and "autonomous DDL" instead of new/legacy mode. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Commit:64d2010
Author:John Meehan
Committer:John Meehan

[#33444] DocDB: Persist the released marked-write watermark and expose an index-backfill downgrade-safety probe The tserver half of the downgrade fence for deferred unique-index verification. A pre-feature binary replays marked backfill writes with positional write IDs -- byte-divergent replicas -- so a cluster must not be allowed to downgrade while any tablet still holds an active ordering generation or marked WAL entries that have not been flushed below the regular-DB frontier. This change makes that per-tablet condition provable and convergent and exposes it over an admin RPC; the master-side veto that consumes it (refusing to demote the capability AutoFlag) lands with the flag itself and the activation work, so the flag cannot ship ahead of the behavior it protects. Two release changes make the condition convergent: - Release persists released_marked_write_watermark in the generation superblock record: the highest marked-write Raft index the tablet has applied (tracked in memory, monotonic, re-seeded by bootstrap replay, which re-applies exactly the marked writes still above the flushed frontier). Anchoring to the last marked WRITE rather than the release op matters: metadata ops never advance the RocksDB frontier, so a release-op watermark would never converge on an idle tablet. - Release schedules an async regular-DB flush (FlushReason::kIndexBackfillGenerationRelease), pushing the marked writes below the frontier within seconds instead of waiting for an organic flush. A marked write that was appended after the release op but applies after it (the leader admits marked writes until the release applies) sits above the persisted watermark; applying it under the released generation raises the watermark and schedules the same flush, so the fence keeps waiting for exactly that write. The released record now survives in the superblock (previously only active generations were persisted) until the next activation. The tablet validator's straggler heal records the watermark and schedules the flush too, deferring its heal to a later cycle if the tablet is not running (the watermark lives on the Tablet). Because that heal re-checks long-active generations periodically (tablet_validator_index_backfill_generation_recheck_sec), a generation stranded by a lost job converges to released on the running binary; a drain never depends on a restart. CheckIndexBackfillDowngradeSafety (tserver admin RPC) reports, per server, the number of tablets with an active ordering generation and the number of released tablets whose watermark is still above the flushed frontier. It fails closed while a tablet cannot report a frontier (bootstrapping, shutting down) and skips tombstoned/deleted peers (WAL removed) and completed split parents (flushed before the split checkpoint), so husks on the master's cleanup schedule cannot keep a drain from converging. A released tablet it finds still unflushed gets the flush scheduled again, so an idle tablet whose release-time flush was lost converges on the next probe rather than never. Test Plan: ./yb_build.sh release --cxx-test pg_index_backfill-test --gtest_filter 'PgIndexBackfillVerifier.DowngradeSafetyProbeReportsHeldGenerations/0' (release blocked by the fixture: the probe counts the held generations) ./yb_build.sh release --cxx-test pg_index_backfill-test --gtest_filter 'PgIndexBackfillVerifierReleased.DowngradeSafetyProbeClearsAfterRelease/0' (release + release flush: both counts converge to zero within seconds) ./yb_build.sh release --cxx-test pg_index_backfill-test --gtest_filter 'PgIndexBackfillVerifierReleasedNoFlush.DowngradeSafetyProbeReportsUnflushedMarkedWal/0' (release flush disabled via TEST_skip_index_backfill_generation_release_flush: no generation is active, yet the unflushed marked WAL is still reported) ./yb_build.sh release --cxx-test tablet_peer-test (watermark persistence, replay re-seeding, release record survival) ./yb_build.sh release --cxx-test fixed_hybrid_time_write_id-itest

Commit:a15f929
Author:John Meehan
Committer:John Meehan

[#33444] DocDB: Add a keyed, per-job violation fingerprint to unique-index verification Verification failure reports have been value-free by design: encoding classes and counts, never the duplicate key. That satisfies the privacy contract but leaves support blind to correlation -- "is this the same duplicate as the last retry?" This adds the design document's keyed, scope-limited diagnostic fingerprint: a non-reversible token identifying the violating index key, safe for low-cardinality values because confirming a candidate requires the key, and the key's scope limits how long confirmation is possible at all. Mechanics: - Per-job secret (32 bytes, OpenSSL RAND_bytes; a CSPRNG failure fails the job launch instead of the master) generated in BackfillTable::Launch for SKIP_ALL jobs (other modes never verify and carry no key), persisted in BackfillJobPB (field 9, latched-per-job like the mode and the gate), and deleted with the job: fingerprints correlate across the job's retries and master failover, and are unconfirmable once the job is gone. - The verifier exposes the first violating group's raw encoded DocKey in-process only (excluded from ToString; never serialized). The tserver handler renders the token -- "<jobtag8hex>-<digest16hex>", both truncated HMAC-SHA256 under the job key (yb/util/keyed_fingerprint.{h,cc}) -- into a structured response field and a "fingerprint=" suffix on the still-value-free reason. The job tag is derived from the key alone, so tokens from different jobs are visibly incomparable. - The reason rides the existing paths into the persisted verification state, the master outcome log, and (in gating mode) backfill_error_message -> the CREATE INDEX error. The structured token is also persisted (UniqueIndexVerificationStatePB field 5) for support tooling. - The key never leaves the master/tserver trust domain. One helper, ScrubBackfillJobSecrets (catalog_manager_util), is applied at every SysTablesEntryPB egress: GetBackfillJobs and the admin sys-catalog dump (DumpSysCatalogEntries, which otherwise renders the table entry verbatim; both pinned by test), CatalogManager::DumpState, the catalog loader's and the create/alter/delete paths' VLOGs of whole table entries, and the snapshot export repack (a backup taken during a backfill would otherwise carry the key off the cluster). Both request-log sites (master send VLOG, tserver receive DVLOG) share RedactedDebugString -- which also elides start_key, a raw index DocKey. Test Plan: ./yb_build.sh release --cxx-test keyed_fingerprint-test --gtest_filter 'KeyedFingerprintTest.TokenFormatAndDeterminism' ./yb_build.sh release --cxx-test keyed_fingerprint-test --gtest_filter 'KeyedFingerprintTest.TagIdentifiesKeyScope' (token format, determinism under one key, shared scope tag per key, cross-key incomparability) ./yb_build.sh release --cxx-test unique_index_verifier-test --gtest_filter 'UniqueIndexVerifierTest.ViolationExposesGroupPrefixInMemoryOnly' (the privacy pin: the raw group prefix is exposed in memory for the caller and excluded from ToString) ./yb_build.sh release --cxx-test pg_index_backfill-test --gtest_filter 'PgIndexBackfillFailClosedVerification.ViolationFailsCreateIndex/0' (extended: the CREATE INDEX error now carries "fingerprint=<token>"; verified rendering end-to-end: fingerprint=2d929cea-dbf259e3ff0d7297) ./yb_build.sh release --cxx-test pg_index_backfill-test --gtest_filter 'PgIndexBackfillMultiMaster.UniqueCheckModePersistedAcrossMasterFailover/0' (the key is absent from GetBackfillJobs and from the sys-catalog dump of the table entry while the job is live) Full unique_index_verifier-test 20/20. Regressions: shadow violation (waiter matches the enriched line), shadow clean multi-tablet, failover resume -- all green.

Commit:ee332f7
Author:John Meehan
Committer:John Meehan

[#33444] DocDB: Add verification scan accounting and master outcome counters Verification/job observability for deferred unique-index verification, plus the accounting fix deferred from the verifier-core review: versions_scanned was inconsistent on the bounded-memory fallback path (the forward pass stopped counting at buffer overflow, then the reverse walk re-counted its re-visits -- over- and under-counting the same group). The forward pass now owns the accounting -- every physical version in a scanned group is counted exactly once regardless of replay path -- and the new fallback_groups counter records the reverse walk's extra work instead (a fallback group is read roughly twice). Plumbing and aggregation: - VerifyUniqueIndexTabletResponsePB gains fallback_groups (field 8); the handler fills it from the verifier result. - The shadow coordinator aggregates the job's scan totals from every response and logs them with the recorded outcome: "... VERIFY_CLEAN [observational] (dockey_groups=300, versions=300, fallback_groups=0)". - Five cluster-level master counters (backfill_aborted pattern): unique_index_verification_outcome_{clean,violation,inconclusive} (one per job: counted on the transition into the terminal state, which is final -- a coordinator failure after a recorded verdict neither overwrites it nor counts again) and unique_index_verification_{versions_scanned,fallback_groups} (aggregated from responses; late responses dropped by the single-winner guard still count -- the scan happened). A persistently high fallback_groups rate is the signal to raise unique_index_verify_max_buffered_versions_per_group; the counter description says so. Test Plan: ./yb_build.sh release --cxx-test unique_index_verifier-test --gtest_filter 'UniqueIndexVerifierTest.FallbackAccountingMatchesBuffered' (the accounting pin: identical data verified through the buffered and fallback paths reports identical versions_scanned and dockey_groups_scanned; the fallback run reports fallback_groups = 1) ./yb_build.sh release --cxx-test pg_index_backfill-test --gtest_filter 'PgIndexBackfillShadowVerification.CleanOutcomeRecordedMultiTablet/0' (extended: asserts the master outcome counter == 1 -- exact on a fresh cluster, catches double-counting -- and versions_scanned > 0 via the cluster metric entity) Full unique_index_verifier-test 19/19. Regressions: ViolationRecordedButDoesNotBlockPublication (waiter matches the enriched log line), ViolationFailsCreateIndex (gating path through Record), failover resume, paginated clean -- all green.

Commit:b41175a
Author:John Meehan
Committer:John Meehan

[#33444] DocDB: Gate unique-index publication on the deferred verification outcome The last functional piece of deferred uniqueness verification: behind ysql_index_backfill_fail_closed_verification, the verification phase's outcome decides publication instead of merely being recorded. The publication order becomes the design document's: backfill completes -> stay at DO_BACKFILL -> verify all index tablets -> persist the outcome -> RWD -> publish birth_time -> release generation + retention -> indisvalid The gate defaults to ON. Observational (record-and-publish) mode is the explicit opt-out, never the default polarity of a safety property; the flag is inert while no job selects SKIP_ALL, and the mode selector remains hardcoded to CHECK_ALL, so nothing changes for any cluster until activation. Tests drive the gate through the master-side TEST override. In gating mode, backfill-success marking for the job's unique index is deferred from chunk completion to verification completion: a clean index is then marked successful (including the xCluster backfill-completed notification, which must not fire for an index that may still fail) and proceeds through the terminal funnel to READ_WRITE_AND_DELETE and birth_time publication. A VIOLATION or unresolvable INCONCLUSIVE outcome fails the index through the existing backfill failure path -- the persisted backfill status carries a value-free reason, CREATE INDEX returns it, the index is never READ_WRITE_AND_DELETE and never indisvalid, and the existing invalid-index cleanup applies. Since bae5985de0c/#33837 that status reaches the backend with its error codes intact, so a VIOLATION is tagged PgsqlError(YB_PG_UNIQUE_VIOLATION): CREATE UNIQUE INDEX fails with SQLSTATE 23505 exactly as a CHECK_ALL duplicate (or vanilla PostgreSQL) does. INCONCLUSIVE and coordinator failures carry no code (XX000): they are infrastructure failures, not statements about the data. A coordinator failure records INCONCLUSIVE against the job's index (resolvable from the job even when the phase failed before selecting its target) and fails it rather than publishing; an index dropped before the phase could verify it takes the same failure path. A non-unique index has nothing to verify and is marked successful at chunk completion as before. The marking is filtered through indexes_to_build() (the index while still IN_PROGRESS), and a resume that finds a persisted terminal outcome re-applies the marking and finishes without re-scanning: the job stays persisted through the release wait, so a failover after publication would otherwise re-run the phase and mark the index successful a second time -- including a second xCluster backfill-completed checkpoint. YSQL backfills exactly one index per job (MultiStageAlterTable never batches or defers YSQL indexes), so the gate reasons about "the job's index" rather than a per-index set. The gating decision is latched per job and persisted in BackfillJobPB, mirroring the uniqueness-check mode: failover, retries, and runtime flag flips cannot reinterpret an active job. Reading the flag live at each decision point would misbehave in both directions -- gating->observational after the deferred marking leaves the index IN_PROGRESS at the funnel; observational->gating mid-phase would fail an index whose success (and xCluster notification) already fired. The funnel warns if it ever sees an IN_PROGRESS index (reachable when the terminal-state marking itself failed, e.g. on a lost leader epoch); the pre-existing mapping publishes such an index as failed. The gate implies the verification phase runs even when ysql_index_backfill_shadow_verification is off. Observational mode is untouched -- outcome logs now carry a [gating] / [observational] tag. Test Plan: ./yb_build.sh release --cxx-test pg_index_backfill-test --gtest_filter 'PgIndexBackfillFailClosedVerification.CleanBuildPublishes/0' ./yb_build.sh release --cxx-test pg_index_backfill-test --gtest_filter 'PgIndexBackfillFailClosedVerification.ViolationFailsCreateIndex/0' ./yb_build.sh release --cxx-test pg_index_backfill-test --gtest_filter 'PgIndexBackfillFailClosedInconclusive.InconclusiveFailsCreateIndex/0' ./yb_build.sh release --cxx-test pg_index_backfill-test --gtest_filter 'PgIndexBackfillFailClosedPhaseFailure.CoordinatorFailureFailsCreateIndex/0' ./yb_build.sh release --cxx-test pg_index_backfill-test --gtest_filter 'PgIndexBackfillShadowVerificationFailover.DuplicateWrittenBeforeFailoverIsStillReported/0' ./yb_build.sh release --cxx-test pg_index_backfill-test --gtest_filter 'PgIndexBackfillShadowVerification.ViolationRecordedButDoesNotBlockPublication/0' The violation test is the feature's end state: a real pre-existing duplicate survives the SKIP_ALL build and fails CREATE INDEX with a verification error carrying SQLSTATE 23505; pg_index shows indisvalid = false, the base table stays usable, and the invalid index is droppable. The inconclusive test pins fail-closed = "not proven clean is not clean": an unresolvable verification outcome fails the build even though the data is clean. The phase-failure test injects a coordinator failure and asserts the uniform INCONCLUSIVE (coordinator error) failure with no 23505. The failover test pins that re-activation (which moves the generation's base_op_index above every marked write of the first leader) does not hide a duplicate written before the failover. The observational test now opts out of the default-on gate explicitly and still records-and-publishes. Also run: the SKIP_ALL / verifier / shadow / multi-master fixtures with the gate at its new default. Three pre-existing tests change with the default: the verifier fixture (which drives the RPC itself over builds that contain duplicates) and the mode plumbing test UniqueCheckModePlumbedFromMasterToWritePath (which deliberately publishes an index over a duplicate to prove the write path skipped the checks) opt out explicitly, and UniqueCheckModePersistedAcrossMasterFailover now asserts the verification VIOLATION -- a resumed job that wrongly re-selected CHECK_ALL fails earlier with the foreground "duplicate key" error instead, so the assertion still distinguishes the two.

Commit:bb4c22f
Author:John Meehan
Committer:John Meehan

[#33444] DocDB: Add the shadow verification coordinator with durable job state The first production caller of the verification RPC: after a SKIP_ALL unique-index backfill completes -- while the ordering generations are still active, before the terminal funnel releases them -- the master runs the deferred uniqueness verification scan observationally. The outcome is persisted and logged but never gates index publication; the fail-closed verify-before-publication gate is a separate, later capability. Disabled by default (ysql_index_backfill_shadow_verification). The phase is single-index by construction: SKIP_ALL is YSQL-only and MultiStageAlterTable backfills exactly one YSQL index per job (no batching, no deferral), so the job's unique index is the verification target. The coordinator holds one state, one target tablet set and one outcome, and a job that violates the invariant fails closed (DFATAL in debug builds, INCONCLUSIVE in release). Durable state (BackfillJobPB.unique_index_verification, one message): verification state, verify_upper_ht, the target tablet set, tablets confirmed clean, and a value-free reason. verify_upper_ht is chosen ONCE from cluster hybrid time (never wall clock) and persisted before any scan -- retries and failover resume verify the same window. Clean tablets are durable progress at tablet granularity; finer intra-tablet resume keys are deliberately not persisted, since re-scanning one tablet after failover is idempotent. All persistence goes through the indexed table's write lock and a sys-catalog upsert under the job's leader epoch. Tablet-set stability. Splits of the index are allowed during the build (children inherit the ordering generation), so verification establishes the stable set itself: persisting IN_PROGRESS arms a durable split fence (the tablet-split manager refuses to split an index whose job's verification state is not VERIFY_NONE -- it reads sys-catalog state, so it holds across failover until the funnel clears the job after release), then the coordinator disables automatic splitting of the index, drains any split in flight (the same wait the backfill applies to the indexed table), resolves the tablets and persists them as the target set. A resume that finds a different set reports VERIFY_INCONCLUSIVE instead of scanning a moving set. Coordinator mechanics (VerifyUniqueIndexForTablet, the GetSafeTimeForTablet task shape): - Bounded per-tablet concurrency (index_backfill_shadow_verification_max_concurrent_tablets). - Pagination: index_backfill_verify_dockey_groups_per_rpc caps each RPC's group budget; a tablet resumes from the returned resume key without touching the join counts. - Short-circuit under a single-winner protocol: the terminal check and the launch-next / index-done decision are one critical section, so a CLEAN callback racing a short-circuiting VIOLATION can neither overwrite the recorded outcome nor double-finish the phase; the slow clean-tablet persist happens after the join decision. Late responses cannot change a recorded outcome. The join guards on job failure, not done() -- the whole phase runs in the job's kSuccess state. Aborted tasks join too: the task is registered on the index table, which DROP INDEX tears down while the phase waits on it, and a join that never arrives would strand the job; an abort is an inconclusive tablet (a leadership-loss abort then fails the epoch-checked sys-catalog writes, and the new leader resumes the phase). - Fail-open on coordinator errors: the phase is on the CREATE INDEX critical path, so every coordinator failure (task launch, persistence, pagination resume) degrades to VERIFY_INCONCLUSIVE and continues into publication -- no failure may strand the job short of the terminal funnel, and a latch keeps the funnel single-entry when a failure after the recorded outcome degrades through the same path. The tserver returns typed error codes for deterministic failures (generation mismatch, non-unique-index tablet, cutoff violations) so the coordinator does not burn its retry budget on requests that cannot succeed. - An index with no tablets records VERIFY_INCONCLUSIVE: an empty tablet set is not evidence of uniqueness. - Master failover mid-phase resumes with the persisted window. Generic lost-backfill resume (load-time enqueue, resuming a job whose table has left ALTERING) is upstream since #33207/c830a71489 -- this part originally carried equivalent machinery and now relies on upstream's, adding only the verification-phase awareness: a resumed job whose indexes are all SUCCESS skips straight to the completion phase (and from there into verification) instead of relaunching empty backfill chunks. - No generation_base_op_index in the requests, on purpose: the base is per-tablet (the activation op's own Raft index) and a re-activated generation's higher base is still the generation this job's marked writes live under; active + index-table match is the semantic guard. Each verification RPC gets its own deadline (index_backfill_verify_rpc_timeout_ms, default 60s), mirroring the backfill chunks' ysql_index_backfill_rpc_timeout_ms instead of inheriting the generic 30s master_ts_rpc_timeout_ms: the deadline sizes one page of the scan (the tserver stops a grace margin early and returns a resume key), and pages should be sizable independently of unrelated master RPCs. Test Plan: ./yb_build.sh release --cxx-test pg_index_backfill-test --gtest_filter 'PgIndexBackfillShadowVerificationSlow.SplitFencedDuringVerification/0' (split refused by the durable fence while the scan runs; accepted again after release) ./yb_build.sh release --cxx-test pg_index_backfill-test --gtest_filter 'PgIndexBackfillShadowVerification.CleanOutcomeRecordedMultiTablet/0' ./yb_build.sh release --cxx-test pg_index_backfill-test --gtest_filter 'PgIndexBackfillShadowVerification.ViolationRecordedButDoesNotBlockPublication/0' ./yb_build.sh release --cxx-test pg_index_backfill-test --gtest_filter 'PgIndexBackfillShadowVerificationPaginated.CleanAcrossManyRpcs/0' ./yb_build.sh release --cxx-test pg_index_backfill-test --gtest_filter 'PgIndexBackfillVerifier.GenerationMismatchFailsCleanly/0' ./yb_build.sh release --cxx-test pg_index_backfill-test --gtest_filter 'PgIndexBackfillShadowVerificationFailover.ResumesWithPersistedWindowAcrossFailover/0' The violation test is the shadow-semantics pin: a real duplicate is recorded (NOT CLEAN: VERIFY_VIOLATION in the master log) while CREATE INDEX still succeeds and the index serves reads. The multi-tablet test exercises the fan-out/join; the paginated fixture forces one DocKey group per RPC through the resume path. GenerationMismatchFailsCleanly covers the 4b review's L1 (table-id and base mismatches fail without scanning). The failover test holds the verification RPCs open with a retryable rejection, steps the master leader down mid-phase, and asserts the new leader resumes with the persisted window before completing the build. ./yb_build.sh release --cxx-test pg_index_backfill-test --gtest_filter 'PgIndexBackfillShadowVerificationDeadlinePaginated.CleanAcrossDeadlinePages/0' The deadline-paginated test is the regression for the measurement-run defect: a slowed scan against a 3s verify budget and 1s grace margin must complete CLEAN through several deadline-bounded pages; before the margin (4b) and this budget existed, the page response always arrived after the coordinator's deadline and the tablet failed INCONCLUSIVE at the retry ceiling. Regressions: shadow-flag-off skip_all e2e, CHECK_ALL mode test, 4b verifier e2e -- all green. The failover-resume e2e now passes through upstream