Proto commits in MaterializeInc/materialize

These commits are when the Protocol Buffers files have changed: (only the last 100 relevant commits are shown)

2026-07-07

storage: version the PostgreSQL oid source cast per export (#37458) Follow-up to https://github.com/MaterializeInc/materialize/pull/37440 Sorry for only realizing so late that it's an issue after all! Without this PR `bin/mzcompose --find platform-checks run default --scenario=UpgradeEntireMz --check=PgCdcOidRetraction` fails like this after the upgrade: ``` > SELECT id, o FROM oid_retraction_table ORDER BY id; 2026-07-06T13:19:56.907798Z WARN mz_testdrive::action::consistency: No Catalog state on disk, skipping consistency check ^^^ +++ 2:1: error: executing query failed: db error: ERROR: Source error: source must be dropped and recreated due to failure: failed to cast raw column: invalid input syntax for type oid: number too large to fit in target type: "4294967295" ``` Since the source is not an errored state it's possible customers could run into this. `parse_oid` was widened to the full `u32` range (#37440), but the same helper backs `CastStringToOid` in the PostgreSQL source, whose eval behavior must stay stable across releases. Replication re-casts the old tuple on delete, so a value ingested pre-upgrade as a `CastError` would be retracted post-upgrade as a value. That leaves the error stuck and adds a phantom negative row. The fix versions the cast per export: * `CastStringToOid` is kept on a frozen `parse_oid_legacy` that accepts only the `i32` range, so exports created before the widening keep retracting cleanly across upgrades. The SQL, COPY, and pgwire paths keep the widened `parse_oid`. * Newly purified exports persist a `cast_oid_full_range` flag in their statement `DETAILS` and plan a new `CastStringToOidFullRange` variant that accepts the full `u32` range, so fresh sources ingest valid oid values above `i32::MAX`. Column casts are regenerated from the persisted details on every plan, so deriving the choice from the details (rather than the binary version) keeps it stable for the lifetime of an export. Tests: a unit snapshot pinning the frozen legacy behavior, error snapshot and SQL-cast parity tests for the full-range variant, a platform-check reproducing the cross-upgrade retraction, and a pg-cdc testdrive file covering snapshot, replication, and retraction of above-`i32::MAX` oids in scalar and array columns. --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

2026-06-23

repr: Harden proto/Row decoding against malformed input (#36984) Hardens `mz-repr` proto/Row decoding against malformed/untrusted bytes — these paths are reachable from persisted state and the wire, so a panic is an availability bug. Replaces panics/asserts with proper decode errors and validates ranges (Date, CheckedTimestamp, ProtoNumeric, ProtoRange, ProtoRelationDesc, ProtoRow dict ordering, uuid parsing, leap-second truncation, Avro decimal). Found by the cargo-fuzz suite ([separate infra PR](https://github.com/MaterializeInc/materialize/pull/36982)). Each fix has a regression test. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> Co-authored-by: Moritz Hoffmann <antiguru@gmail.com>

2026-06-18

[SS-187] Add support for replication from a physical postgres replica (#37020) ### Motivation We would like to be able to set up replication against a physical Postgres replica. This fails hard today because you can't read pg_current_wal_lsn from a standby. Relevant issue: https://linear.app/materializeinc/issue/SS-187/replication-from-postgres-replica This is adapted from https://github.com/MaterializeInc/materialize/pull/36923. ### Description - Record during purification whether the connection is to a physical replica - Use the appropriate method of loading the LSN depending on whether it's a physical replica or not ### Tradeoffs to call out - Replicating from a primary without `hot_standby_feedback=on` on the replica risks snapshots timing out (potentially quickly) and increases the risk of the replication slot being terminated. - In the test case where we stall, we'll hold a slot open on the replica. With `hot_standby_feedback=on` this risks bloating the disk of the primary. I think setting max_slot_wal_keep_size to a reasonably high level mitigates the scariest versions of this. This style of tradeoff seems somewhat similar to existing trade-offs around max_slot_wall_keep_size settings. - RTR is no longer accurate relative to the primary postgres replica. This should be ok and similar to cascading logical replication which we support today. - Materialize sources will require a fresh snapshot in the case of failover -- this is similar to logical replication today where a timeline id change produces an error. This protects us from corruption. ### Verification - Testdrive tests covering reading from a replica, basic RTR queries, and ensuring we stall if the same replica is recovered from a more recent backup

Revert "Remove the Fivetran destination (#36919)" (#37117) This reverts commit 1a79aef641832be2d9b3efe4578bc93c917e7707.

2026-06-09

Remove the Fivetran destination (#36919) Delete the Fivetran Destination integration in its entirety: the mz-fivetran-destination crate and its vendored SDK protos, the fivetran-destination-tester image, the dedicated test suite, the testdrive `fivetran-destination` built-in command and its CLI/state plumbing, the Python mzcompose services, the CI pipeline step, and the user-facing documentation. Also regenerate buf.yaml (dropping the fivetran proto exclude) and prune the now-unused tonic/prost build dependencies from mz-testdrive. Co-authored-by: Claude <noreply@anthropic.com>

2026-05-04

[mysql source versioning] add binlog setting to mysql source details at purification (#36253) Now adds a check at sql purification time to see what the binlog setting of the upstream mysql db is, and saves that setting in the mysql source details, so that we can check the original value during replication, and maintain a consistent decoding strategy --------- Co-authored-by: Copilot <copilot@github.com>

2026-04-24

compute: serialize DataflowError on dataflow edges (#35776) ## Summary - Introduce `DataflowErrorSer`, a newtype over proto-encoded bytes, to replace `DataflowError` on compute-internal dataflow edges (streams, arrangements). - Make `persist_source` generic over the error type `E`, so compute callers can choose `DataflowErrorSer` while storage callers stay on `DataflowError`. - Switch all compute rendering, arrangement types (`ErrSpine`, `RowErrSpine`, etc.), `CollectionBundle`, sinks, and logging to use `DataflowErrorSer`. ## Motivation `DataflowError` is a complex enum with boxed variants (`EvalError` alone has 50+ sub-variants). It requires a 700-line hand-crafted `Columnation` impl. In compute rendering, errors are **never inspected** — they flow opaquely through concat, consolidate, arrange, and are only displayed (`.to_string()`) at sinks. Replacing it with opaque serialized bytes: - Eliminates the need for the complex `Columnation` impl in compute (delegates to trivial `Vec<u8>` columnation) - Removes complex structures from dataflow edges - Paves the way for eventually using `Row` to represent serialized errors ## Design **Serialization boundary:** Errors are serialized at the `persist_source` boundary when entering compute, and deserialized only at exit points (MV sink and continual task for persist writes; subscribe and copy-to-s3 via `Display`). **Canonicality invariant:** The MV sink's self-correction logic computes `desired - persist` and relies on equality for cancellation. Proto3 encoding via prost is deterministic for messages without `map` fields (verified: none in the error proto chain), so byte-equality implies semantic equality. **Persist boundary unchanged:** `SourceData` already serializes errors as proto bytes in Arrow columnar encoding. This change does not affect the persist format. ## Testing - Added proptest verifying round-trip canonicality: `ser(deser(ser(e))) == ser(e)` for all `DataflowError` variants. - Added test verifying `Display` equivalence between `DataflowError` and `DataflowErrorSer`. - All existing mz-compute and mz-storage-operators tests pass. ## Future work - Direct `EvalError → proto bytes` (avoids a `Box::new` allocation in the `From<EvalError>` path) - `SourceData::append_err_bytes` to skip deserialize/reserialize at MV sink - Error code + optional description format for compute-created errors - `Columnar` impl for `DataflowErrorSer` (enables newer spine infrastructure) 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

2026-04-21

`repeat_row`: Make subquery guard aware of negative diffs (#36185) This is the 4. PR in the `repeat_row` productionization work stream. This resolves https://github.com/MaterializeInc/materialize/pull/32978#discussion_r2200866973. It was not important originally, but now that we are productionizing `repeat_row`, the original error message would become confusing: it would say that "more than one record produced in subquery", when in fact there was a negative number of rows. I didn't add a test, because I wasn't able to find a query where this error deterministically happens. If I just naively put a `repeat_row(-1)` inside a subquery, then sometimes this error happens, but sometimes a `Distinct` gives the negative accumulation error message.

2026-04-07

catalog: convert mz_materialized_views into a view over the catalog (#35819) This PR converts the `mz_materialized_views` builtin table into a materialized view over the catalog. It includes some pre-work to make this possible: * Introduces a constant builtin view `mz_builtin_materialized_views` that exposes... builtin materialized views. The `mz_materialized_views` query joins against that view to augment the catalog contents (which only include user MVs) with all the builtin MVs. * Introduces two new internal SQL functions: * `parse_catalog_create_sql`, to extract information from the MV `create_sql` * `redact_sql`, to derived `redacted_create_sql` from the MV `create_sql` Alternatives to having `mz_builtin_materialized_views` as a constant builtin view: * Make it a builtin table that gets updated during bootstrap. We want to move away from builtin tables as they complicate having concurrent envds, so this seems like a step in the wrong direction. Builtin views pose no issues for concurrent envds. * Make it a builtin table function. This doesn't work because SQL funcs are defined in `mz-expr`, while builtins are defined in `mz-catalog`, and the dependency relationship is the wrong way round. We could introduce a way to add table functions that get evaluated during planning time, but afaict that infrastructure doesn't exist yet. * Insert builtins into the catalog so we can simply treat them like user MVs. This might be nice but is also a lot of work, and might also complicate 0dt upgrades by adding more state to the catalog that will need to be migrated. ### Motivation Part of SQL-118

2026-03-25

adapter: make mz_databases a view over the catalog (#35449) This PR replaces the `mz_databases` builtin table with a builtin materialized view over `mz_catalog_raw`. As part of this, it adds some internal SQL functions for parsing parts of the catalog JSON, representing IDs and privileges, into the format of the existing builtin tables. This is only converting a single table for now, to try out the concept. If we don't find issues, the plan is to convert most of the other builtin tables as well. ### Motivation Part of SQL-118

2026-03-05

persist: Add group commit consensus service PoC Introduces two new components to batch independent cross-shard CAS writes into a single durable S3 Express One Zone PUT per flush interval, making cost O(1/batch_window) instead of O(shards): - `RpcConsensus` (gRPC client implementing the `Consensus` trait) - `persist-consensus-svc` (group commit service with actor-based state machine, WAL, snapshot, and recovery) The actor processes CAS, truncate, scan, head, and list_keys commands on a single thread. Writes are batched and flushed to S3 WAL periodically, with snapshots every N batches for bounded recovery. Includes 27 unit tests covering CAS semantics, group commit batching, read operations, truncate, WAL integration, and snapshot intervals.

2026-02-20

Remove fivetran submodule (#35097) Inspired by https://materializeinc.slack.com/archives/C0246GEHL8N/p1771525524278419 git submodules are a pain in my experience, this is the last remaining one.

2025-12-10

catalog-protos: remove the old proto definitions This commit removes the now unused proto definitions and cleans up the crate dependencies.

2025-11-17

Merge pull request #34140 from bkirwi/batch-match [persist] Check schemas in batch append

sql_server: add additional upstream constraint support (#33991) Adds support for unique constraints and isolates constraint detection from table discovery. This also fixes an issue where a column appears multiple times in the source description if it is a member of multiple unique/PK constraints. ### Motivation Implements https://github.com/MaterializeInc/database-issues/issues/9723 ### Tips for reviewer Functional changes: - `src/sql-server-util/src/inspect.rs` (changes SQL Server queries: simplify table desc query, break out constraint query) - `src/sql/src/pure/sql_server.rs` (building out SourceExportStatementDetails) ### Checklist - [ ] This PR has adequate test coverage / QA involvement has been duly considered. ([trigger-ci for additional test/nightly runs](https://trigger-ci.dev.materialize.com/)) - [ ] This PR has an associated up-to-date [design doc](https://github.com/MaterializeInc/materialize/blob/main/doc/developer/design/README.md), is a design doc ([template](https://github.com/MaterializeInc/materialize/blob/main/doc/developer/design/00000000_template.md)), or is sufficiently small to not require a design. <!-- Reference the design in the description. --> - [ ] If this PR evolves [an existing `$T ⇔ Proto$T` mapping](https://github.com/MaterializeInc/materialize/blob/main/doc/developer/command-and-response-binary-encoding.md) (possibly in a backwards-incompatible way), then it is tagged with a `T-proto` label. - [ ] If this PR will require changes to cloud orchestration or tests, there is a companion cloud PR to account for those changes that is tagged with the release-blocker label ([example](https://github.com/MaterializeInc/cloud/pull/5021)). <!-- Ask in #team-cloud on Slack if you need help preparing the cloud PR. --> - [ ] If this PR includes major [user-facing behavior changes](https://github.com/MaterializeInc/materialize/blob/main/doc/developer/guide-changes.md#what-changes-require-a-release-note), I have pinged the relevant PM to schedule a changelog post.

sql_server: add additional upstream constraint support (#33991) Adds support for unique constraints and isolates constraint detection from table discovery. This also fixes an issue where a column appears multiple times in the source description if it is a member of multiple unique/PK constraints. ### Motivation Implements https://github.com/MaterializeInc/database-issues/issues/9723 ### Tips for reviewer Functional changes: - `src/sql-server-util/src/inspect.rs` (changes SQL Server queries: simplify table desc query, break out constraint query) - `src/sql/src/pure/sql_server.rs` (building out SourceExportStatementDetails) ### Checklist - [ ] This PR has adequate test coverage / QA involvement has been duly considered. ([trigger-ci for additional test/nightly runs](https://trigger-ci.dev.materialize.com/)) - [ ] This PR has an associated up-to-date [design doc](https://github.com/MaterializeInc/materialize/blob/main/doc/developer/design/README.md), is a design doc ([template](https://github.com/MaterializeInc/materialize/blob/main/doc/developer/design/00000000_template.md)), or is sufficiently small to not require a design. <!-- Reference the design in the description. --> - [ ] If this PR evolves [an existing `$T ⇔ Proto$T` mapping](https://github.com/MaterializeInc/materialize/blob/main/doc/developer/command-and-response-binary-encoding.md) (possibly in a backwards-incompatible way), then it is tagged with a `T-proto` label. - [ ] If this PR will require changes to cloud orchestration or tests, there is a companion cloud PR to account for those changes that is tagged with the release-blocker label ([example](https://github.com/MaterializeInc/cloud/pull/5021)). <!-- Ask in #team-cloud on Slack if you need help preparing the cloud PR. --> - [ ] If this PR includes major [user-facing behavior changes](https://github.com/MaterializeInc/materialize/blob/main/doc/developer/guide-changes.md#what-changes-require-a-release-note), I have pinged the relevant PM to schedule a changelog post.

2025-11-14

Stash the encoded schemas in Batch

2025-11-13

Add a metadata map to the hollow batch part

Add a metadata map to the RunMeta struct

2025-11-06

Auth: migrate hash string to be directly compatible with pg/pgbouncer (#33714) When I originally wrote auth I used a postgres inspired string format to store the hashes, but didn't index on making it 1:1 compatible since we had originally decided not to surface the hash to the user... That didn't last. Some want to be able to use pgbouncer. It would be easy enough to give instructions/a script to make the existing hash format into the one pgbouncer wants in its config file but... we could just not do that. (This PR also adds platform-checks for passwords to ensure functionality across versions) <!-- Describe the contents of the PR briefly but completely. If you write detailed commit messages, it is acceptable to copy/paste them here, or write "see commit messages for details." If there is only one commit in the PR, GitHub will have already added its commit message above. --> ### Motivation <!-- Which of the following best describes the motivation behind this PR? * This PR fixes a recognized bug. [Ensure issue is linked somewhere.] * This PR adds a known-desirable feature. [Ensure issue is linked somewhere.] * This PR fixes a previously unreported bug. [Describe the bug in detail, as if you were filing a bug report.] * This PR adds a feature that has not yet been specified. [Write a brief specification for the feature, including justification for its inclusion in Materialize, as if you were writing the original feature specification.] * This PR refactors existing code. [Describe what was wrong with the existing code, if it is not obvious.] --> ### Tips for reviewer <!-- Leave some tips for your reviewer, like: * The diff is much smaller if viewed with whitespace hidden. * [Some function/module/file] deserves extra attention. * [Some function/module/file] is pure code movement and only needs a skim. Delete this section if no tips. --> ### Checklist - [ ] This PR has adequate test coverage / QA involvement has been duly considered. ([trigger-ci for additional test/nightly runs](https://trigger-ci.dev.materialize.com/)) - [ ] This PR has an associated up-to-date [design doc](https://github.com/MaterializeInc/materialize/blob/main/doc/developer/design/README.md), is a design doc ([template](https://github.com/MaterializeInc/materialize/blob/main/doc/developer/design/00000000_template.md)), or is sufficiently small to not require a design. <!-- Reference the design in the description. --> - [ ] If this PR evolves [an existing `$T ⇔ Proto$T` mapping](https://github.com/MaterializeInc/materialize/blob/main/doc/developer/command-and-response-binary-encoding.md) (possibly in a backwards-incompatible way), then it is tagged with a `T-proto` label. - [ ] If this PR will require changes to cloud orchestration or tests, there is a companion cloud PR to account for those changes that is tagged with the release-blocker label ([example](https://github.com/MaterializeInc/cloud/pull/5021)). <!-- Ask in #team-cloud on Slack if you need help preparing the cloud PR. --> - [ ] If this PR includes major [user-facing behavior changes](https://github.com/MaterializeInc/materialize/blob/main/doc/developer/guide-changes.md#what-changes-require-a-release-note), I have pinged the relevant PM to schedule a changelog post.

The documentation is generated from this commit.

2025-09-30

*: remove more dead protobuf/proptest code

2025-09-19

pgcopy: remove unused protobuf code

rocksdb-types: remove unused protobuf code

tracing: remove unused protobuf code

service: remove unused protobuf code

dyncfg: remove unused protobuf code

2025-09-17

Remove most protos from mz-expr (#33608) As it says on the box, remove protos and their Rust bindings from mz-expr. Also remove some `Arbitrary` implementations that aren't used. We can't remove all `Arbitrary` implementations because we construct expressions in many tests. Signed-off-by: Moritz Hoffmann <mh@materialize.com>

storage-types: remove proto code This commit removes unused protobuf definitions, as well as any code to support them from `mz-storage-types`. Not all proto definitions can be deleted. Some are used for serializing into persist or the catalog.

compute-types: remove proto code This commit removes the protobuf definitions, as well as any code to support them from `mz-compute-types`. We have to keep most of the `Arbitrary` impls because adapter's expression cache code wants to use them in its tests.

storage-client: remove proto code This commit removes the protobuf definitions, as well as any code to support them from `mz-storage-client`.

compute-client: remove proto code This commit removes the protobuf definitions, as well as any code to support them from `mz-compute-client`.

2025-09-16

compute: use unified introspection for operator hydration statuses This commit changes `mz_compute_operator_hydration_statuses` from a compute controller introspection collection fed by `Status` compute responses to a unified introspection collection fed by a subscribe reading from an introspection index. Apart from better reusing existing infrastructure, the change also fixes a bug where operator hydration statuses would be missing after a successful compute reconciliation.

2025-09-05

sql: Unicode normalization (#33507) Mirrors the postgres normalize function, which takes a string to normalize and optionally the unicode form, defaulting to NFC. - The parser parses the form keywords as strings, inserting the default NFC when the sql call is unary. - The expression layer passes through the implementation to the `unicode-normalization` crate, which is already a transitive dependency. For details, see: - [Unicode docs](https://unicode.org/reports/tr15/#Norm_Forms) - [Postgres](https://www.postgresql.org/docs/current/functions-string.html) - [Issue 7930](https://github.com/MaterializeInc/database-issues/issues/7930).

Parse the normalization form as a keyword, not a string So it's `SELECT normalize('한글', NFC)`, not `SELECT normalize('한글', 'NFC')`

2025-09-04

sql: Unicode normalization Mirrors the postgres normalize function. Provides a unary function defaulting to the NFC form, as well as a binary variant taking the form to use (one of NFC, NFD, NFKC, or NFKD). For details, see: - [Issue 7930](https://github.com/MaterializeInc/database-issues/issues/7930) - [Unicode docs](https://unicode.org/reports/tr15/#Norm_Forms) - [Postgres](https://www.postgresql.org/docs/current/functions-string.html)

Remove yugabyte (#33494) title says it all ### Motivation https://github.com/MaterializeInc/database-issues/issues/9641 ### Checklist - [ ] This PR has adequate test coverage / QA involvement has been duly considered. ([trigger-ci for additional test/nightly runs](https://trigger-ci.dev.materialize.com/)) - [ ] This PR has an associated up-to-date [design doc](https://github.com/MaterializeInc/materialize/blob/main/doc/developer/design/README.md), is a design doc ([template](https://github.com/MaterializeInc/materialize/blob/main/doc/developer/design/00000000_template.md)), or is sufficiently small to not require a design. <!-- Reference the design in the description. --> - [ ] If this PR evolves [an existing `$T ⇔ Proto$T` mapping](https://github.com/MaterializeInc/materialize/blob/main/doc/developer/command-and-response-binary-encoding.md) (possibly in a backwards-incompatible way), then it is tagged with a `T-proto` label. - [ ] If this PR will require changes to cloud orchestration or tests, there is a companion cloud PR to account for those changes that is tagged with the release-blocker label ([example](https://github.com/MaterializeInc/cloud/pull/5021)). <!-- Ask in #team-cloud on Slack if you need help preparing the cloud PR. --> - [ ] If this PR includes major [user-facing behavior changes](https://github.com/MaterializeInc/materialize/blob/main/doc/developer/guide-changes.md#what-changes-require-a-release-note), I have pinged the relevant PM to schedule a changelog post.

sql: Unicode normalization Mirrors the postgres normalize function. Provides a unary function defaulting to the NFC form, as well as a binary variant taking the form to use (one of NFC, NFD, NFKC, or NFKD). For details, see: - [Issue 7930](https://github.com/MaterializeInc/database-issues/issues/7930) - [Unicode docs](https://unicode.org/reports/tr15/#Norm_Forms) - [Postgres](https://www.postgresql.org/docs/current/functions-string.html)

2025-08-26

Add migration to set LOGIN for mz_system (#33423) For existing installations from before we supported self managed auth the LOGIN bit wouldn't be set for mz_system making it impossible to login. This migration fixes that <!-- Describe the contents of the PR briefly but completely. If you write detailed commit messages, it is acceptable to copy/paste them here, or write "see commit messages for details." If there is only one commit in the PR, GitHub will have already added its commit message above. --> ### Motivation <!-- Which of the following best describes the motivation behind this PR? * This PR fixes a recognized bug. [Ensure issue is linked somewhere.] * This PR adds a known-desirable feature. [Ensure issue is linked somewhere.] * This PR fixes a previously unreported bug. [Describe the bug in detail, as if you were filing a bug report.] * This PR adds a feature that has not yet been specified. [Write a brief specification for the feature, including justification for its inclusion in Materialize, as if you were writing the original feature specification.] * This PR refactors existing code. [Describe what was wrong with the existing code, if it is not obvious.] --> ### Tips for reviewer <!-- Leave some tips for your reviewer, like: * The diff is much smaller if viewed with whitespace hidden. * [Some function/module/file] deserves extra attention. * [Some function/module/file] is pure code movement and only needs a skim. Delete this section if no tips. --> ### Checklist - [ ] This PR has adequate test coverage / QA involvement has been duly considered. ([trigger-ci for additional test/nightly runs](https://trigger-ci.dev.materialize.com/)) - [ ] This PR has an associated up-to-date [design doc](https://github.com/MaterializeInc/materialize/blob/main/doc/developer/design/README.md), is a design doc ([template](https://github.com/MaterializeInc/materialize/blob/main/doc/developer/design/00000000_template.md)), or is sufficiently small to not require a design. <!-- Reference the design in the description. --> - [ ] If this PR evolves [an existing `$T ⇔ Proto$T` mapping](https://github.com/MaterializeInc/materialize/blob/main/doc/developer/command-and-response-binary-encoding.md) (possibly in a backwards-incompatible way), then it is tagged with a `T-proto` label. - [ ] If this PR will require changes to cloud orchestration or tests, there is a companion cloud PR to account for those changes that is tagged with the release-blocker label ([example](https://github.com/MaterializeInc/cloud/pull/5021)). <!-- Ask in #team-cloud on Slack if you need help preparing the cloud PR. --> - [ ] If this PR includes major [user-facing behavior changes](https://github.com/MaterializeInc/materialize/blob/main/doc/developer/guide-changes.md#what-changes-require-a-release-note), I have pinged the relevant PM to schedule a changelog post.

sql_server: handle upstream restores (#33409) Adds detection and handling for SQL Server restore. If MZ detect the database configured as a source has been restored on the upstream SQL Server via `msdb.dbo.restorehistory`, emit a DefiniteError for all exports. ### Motivation Implements https://github.com/MaterializeInc/database-issues/issues/9591 ### Checklist - [x] This PR has adequate test coverage / QA involvement has been duly considered. ([trigger-ci for additional test/nightly runs](https://trigger-ci.dev.materialize.com/)) - [ ] This PR has an associated up-to-date [design doc](https://github.com/MaterializeInc/materialize/blob/main/doc/developer/design/README.md), is a design doc ([template](https://github.com/MaterializeInc/materialize/blob/main/doc/developer/design/00000000_template.md)), or is sufficiently small to not require a design. <!-- Reference the design in the description. --> - [ ] If this PR evolves [an existing `$T ⇔ Proto$T` mapping](https://github.com/MaterializeInc/materialize/blob/main/doc/developer/command-and-response-binary-encoding.md) (possibly in a backwards-incompatible way), then it is tagged with a `T-proto` label. - [ ] If this PR will require changes to cloud orchestration or tests, there is a companion cloud PR to account for those changes that is tagged with the release-blocker label ([example](https://github.com/MaterializeInc/cloud/pull/5021)). <!-- Ask in #team-cloud on Slack if you need help preparing the cloud PR. --> - [ ] If this PR includes major [user-facing behavior changes](https://github.com/MaterializeInc/materialize/blob/main/doc/developer/guide-changes.md#what-changes-require-a-release-note), I have pinged the relevant PM to schedule a changelog post.

2025-08-25

catalog: migration for removed disk flag in cluster/replica configs

adapter: remove replica disk flag This commit removes the `disk` flag associated with replica definitions. Whether a replica gets a disk now solely depends on the `disk_limit` configuration in its replica size: If the `disk_limit` is 0, no disk is attached, otherwise the replica gets a disk. The change is intended to simplify the logic around disk assignment, which previously was complicated for historical reasons. For backward compatibility reasons, the `CREATE CLUSTER REPLICA ... WITH (DISK = false)` syntax still works for non-cc clusters, but has no effect now, except from showing a deprecation notice.

2025-08-22

Rewrite `WITH ORDINALITY` (which `ROWS FROM` also relies on)

2025-08-19

catalog: migration for removed replica options

adapter: remove unneeded replica options Now that the Timely runtime parameters are passed as command line flags to the clusterd processes, these SQL options passed when creating unorchestrated replicas are dead code*: * `STORAGE ADDRESSES` * `COMPUTE ADDRESSES` * `WORKERS` This commit removes them from the adapter. It does not yet remove the SQL parser support, to avoid breaking tests that want to run the same replica DDL against different versions of Materialize. *: Well, almost: The `COPY TO` sequencing code would like to know the number of workers to decide, how many partitions to create. It's probably not worth to keep the replica option just for that, given that unorchestrated replicas are a test-only feature. So we just default to the number of processes in this case.

2025-08-13

Remove optional types from LIR (#33309) Our LIR types had space for optional types that were meant to inform specialization. There is no specialization and no concrete plans to implement such from type information. In particular, the type system itself may evolve, and until it stabilizes it makes some sense to extract these and re-introduce them once we have a clearer sense of what they would be for (vs maintaining them). ### Motivation <!-- Which of the following best describes the motivation behind this PR? * This PR fixes a recognized bug. [Ensure issue is linked somewhere.] * This PR adds a known-desirable feature. [Ensure issue is linked somewhere.] * This PR fixes a previously unreported bug. [Describe the bug in detail, as if you were filing a bug report.] * This PR adds a feature that has not yet been specified. [Write a brief specification for the feature, including justification for its inclusion in Materialize, as if you were writing the original feature specification.] * This PR refactors existing code. [Describe what was wrong with the existing code, if it is not obvious.] --> ### Tips for reviewer <!-- Leave some tips for your reviewer, like: * The diff is much smaller if viewed with whitespace hidden. * [Some function/module/file] deserves extra attention. * [Some function/module/file] is pure code movement and only needs a skim. Delete this section if no tips. --> ### Checklist - [ ] This PR has adequate test coverage / QA involvement has been duly considered. ([trigger-ci for additional test/nightly runs](https://trigger-ci.dev.materialize.com/)) - [ ] This PR has an associated up-to-date [design doc](https://github.com/MaterializeInc/materialize/blob/main/doc/developer/design/README.md), is a design doc ([template](https://github.com/MaterializeInc/materialize/blob/main/doc/developer/design/00000000_template.md)), or is sufficiently small to not require a design. <!-- Reference the design in the description. --> - [ ] If this PR evolves [an existing `$T ⇔ Proto$T` mapping](https://github.com/MaterializeInc/materialize/blob/main/doc/developer/command-and-response-binary-encoding.md) (possibly in a backwards-incompatible way), then it is tagged with a `T-proto` label. - [ ] If this PR will require changes to cloud orchestration or tests, there is a companion cloud PR to account for those changes that is tagged with the release-blocker label ([example](https://github.com/MaterializeInc/cloud/pull/5021)). <!-- Ask in #team-cloud on Slack if you need help preparing the cloud PR. --> - [ ] If this PR includes major [user-facing behavior changes](https://github.com/MaterializeInc/materialize/blob/main/doc/developer/guide-changes.md#what-changes-require-a-release-note), I have pinged the relevant PM to schedule a changelog post.

2025-08-09

Rewrite `WITH ORDINALITY` (which `ROWS FROM` also relies on)

2025-08-08

Fix warnings reported by Rust 1.89 (#33270) As it says on the box, fix warnings generated by Rust 1.89. The PR doesn't upgrade to Rust 1.89 yet. --------- Signed-off-by: Moritz Hoffmann <antiguru@gmail.com>

2025-07-29

Merge pull request #33145 from teskje/upsert-remove-autospill storage/upsert: remove the autospill backend

2025-07-28

Add source table support for SQL server (#32987) Adds support for `CREATE TABLE .. FROM SOURCE` for SQL Server. ### Motivation implements https://github.com/MaterializeInc/database-issues/issues/9202 ### Checklist - [ ] This PR has adequate test coverage / QA involvement has been duly considered. ([trigger-ci for additional test/nightly runs](https://trigger-ci.dev.materialize.com/)) - [ ] This PR has an associated up-to-date [design doc](https://github.com/MaterializeInc/materialize/blob/main/doc/developer/design/README.md), is a design doc ([template](https://github.com/MaterializeInc/materialize/blob/main/doc/developer/design/00000000_template.md)), or is sufficiently small to not require a design. <!-- Reference the design in the description. --> - [ ] If this PR evolves [an existing `$T ⇔ Proto$T` mapping](https://github.com/MaterializeInc/materialize/blob/main/doc/developer/command-and-response-binary-encoding.md) (possibly in a backwards-incompatible way), then it is tagged with a `T-proto` label. - [ ] If this PR will require changes to cloud orchestration or tests, there is a companion cloud PR to account for those changes that is tagged with the release-blocker label ([example](https://github.com/MaterializeInc/cloud/pull/5021)). <!-- Ask in #team-cloud on Slack if you need help preparing the cloud PR. --> - [ ] If this PR includes major [user-facing behavior changes](https://github.com/MaterializeInc/materialize/blob/main/doc/developer/guide-changes.md#what-changes-require-a-release-note), I have pinged the relevant PM to schedule a changelog post. --------- Co-authored-by: Petros Angelatos <petrosagg@gmail.com>

2025-07-25

storage/upsert: remove the autospill backend This commit rips out the autospill backend, which would start of using the in-memory hashmap and then start spilling to RocksDB once some size threshold was reached. We determined that this is an unnecessary optimization and we can avoid some complexity by deleting it. This also paves the way for replacing the in-memory hashmap with RocksDB+MemEnv. Previously doing so would have been kind of weird because the autospill backend would start "spilling" to an in-memory RocksDB instance once the threshold is reached.

2025-07-22

Merge pull request #32891 from teskje/remove-createtimely-2 controller,cluster: replace `CreateTimely` with a `Hello` command

2025-07-14

persist: add len field to RunMeta for tracking updates (#32992) Towards incremental compaction <!-- Describe the contents of the PR briefly but completely. If you write detailed commit messages, it is acceptable to copy/paste them here, or write "see commit messages for details." If there is only one commit in the PR, GitHub will have already added its commit message above. --> ### Motivation <!-- Which of the following best describes the motivation behind this PR? * This PR fixes a recognized bug. [Ensure issue is linked somewhere.] * This PR adds a known-desirable feature. [Ensure issue is linked somewhere.] * This PR fixes a previously unreported bug. [Describe the bug in detail, as if you were filing a bug report.] * This PR adds a feature that has not yet been specified. [Write a brief specification for the feature, including justification for its inclusion in Materialize, as if you were writing the original feature specification.] * This PR refactors existing code. [Describe what was wrong with the existing code, if it is not obvious.] --> ### Tips for reviewer <!-- Leave some tips for your reviewer, like: * The diff is much smaller if viewed with whitespace hidden. * [Some function/module/file] deserves extra attention. * [Some function/module/file] is pure code movement and only needs a skim. Delete this section if no tips. --> ### Checklist - [ ] This PR has adequate test coverage / QA involvement has been duly considered. ([trigger-ci for additional test/nightly runs](https://trigger-ci.dev.materialize.com/)) - [ ] This PR has an associated up-to-date [design doc](https://github.com/MaterializeInc/materialize/blob/main/doc/developer/design/README.md), is a design doc ([template](https://github.com/MaterializeInc/materialize/blob/main/doc/developer/design/00000000_template.md)), or is sufficiently small to not require a design. <!-- Reference the design in the description. --> - [ ] If this PR evolves [an existing `$T ⇔ Proto$T` mapping](https://github.com/MaterializeInc/materialize/blob/main/doc/developer/command-and-response-binary-encoding.md) (possibly in a backwards-incompatible way), then it is tagged with a `T-proto` label. - [ ] If this PR will require changes to cloud orchestration or tests, there is a companion cloud PR to account for those changes that is tagged with the release-blocker label ([example](https://github.com/MaterializeInc/cloud/pull/5021)). <!-- Ask in #team-cloud on Slack if you need help preparing the cloud PR. --> - [ ] If this PR includes major [user-facing behavior changes](https://github.com/MaterializeInc/materialize/blob/main/doc/developer/guide-changes.md#what-changes-require-a-release-note), I have pinged the relevant PM to schedule a changelog post.

Merge main

2025-07-11

Add support for configuring encryption for sql server (#32892) Adds support for configuring TLS encryption and custom CA certificate with MS SQL Server The following options are added to CONNECTION FOR SQL SERVER: - `SSL CERTIFICATE AUTHORITY` - is one more more certificates in PEM format. - `SSL MODE` - `disabled` - no encryption - `required` - encryption required, no certificate validation - `verify` - encryption required, validate server certificate using system certs - `verify_ca` - encryption required, validate server certificate using provided CA cert (requires `SSL CERTIFICATE AUTHORITY` ) Implements https://github.com/MaterializeInc/database-issues/issues/9205 - [x] This PR has adequate test coverage / QA involvement has been duly considered. ([trigger-ci for additional test/nightly runs](https://trigger-ci.dev.materialize.com/)) - [ ] This PR has an associated up-to-date [design doc](https://github.com/MaterializeInc/materialize/blob/main/doc/developer/design/README.md), is a design doc ([template](https://github.com/MaterializeInc/materialize/blob/main/doc/developer/design/00000000_template.md)), or is sufficiently small to not require a design. <!-- Reference the design in the description. --> - [ ] If this PR evolves [an existing `$T ⇔ Proto$T` mapping](https://github.com/MaterializeInc/materialize/blob/main/doc/developer/command-and-response-binary-encoding.md) (possibly in a backwards-incompatible way), then it is tagged with a `T-proto` label. - [ ] If this PR will require changes to cloud orchestration or tests, there is a companion cloud PR to account for those changes that is tagged with the release-blocker label ([example](https://github.com/MaterializeInc/cloud/pull/5021)). <!-- Ask in #team-cloud on Slack if you need help preparing the cloud PR. --> - [ ] If this PR includes major [user-facing behavior changes](https://github.com/MaterializeInc/materialize/blob/main/doc/developer/guide-changes.md#what-changes-require-a-release-note), I have pinged the relevant PM to schedule a changelog post.

controller: remove `ClusterStartupEpoch` This type is not used for anything anymore, so we can delete it and a bunch of epoch plumbing code in the controllers.

controller: replace `CreateTimely` epoch with a nonce This commit replaces the epoch transmitted in `CreateTimely` commands with a nonce. On the replica we need a reliable way to differentiate compute protocol iterations, which the epoch doesn't provide as it doesn't increase in read-only environments.

persist: add len field to RunMeta for tracking updates

Guard subquery size with a TableFunc (#32978) Our subquery lowering has been incorrect for a while, with a combination of a filter that retains subqueries with counts greater than one, and a map to introduce an error. SQL (and our MIR) has no guaranteed evaluation order, and the error would often escape, because the filter does not ensure that the error cannot be picked up as an expression by surrounding logic. This PR introduces a `TableFunc::GuardSubquerySize` which takes the count as an argument, and either produces an error in the output column if the count is greater than one, or produces no output at all if the count is one. We also add an optimization to prune the expression when the count is the constant literal one, as happens when we can statically determine that the subquery cannot have more than one result. Several expected plan changes, and a few moments where plans that end up only operating on the error stream (the subquery contents can be pruned) end up with different join plans. We can discuss those deviations, but tbh I'm not sure on what basis we evaluate join plans with collections that are either empty or errors. ### Motivation <!-- Which of the following best describes the motivation behind this PR? * This PR fixes a recognized bug. [Ensure issue is linked somewhere.] * This PR adds a known-desirable feature. [Ensure issue is linked somewhere.] * This PR fixes a previously unreported bug. [Describe the bug in detail, as if you were filing a bug report.] * This PR adds a feature that has not yet been specified. [Write a brief specification for the feature, including justification for its inclusion in Materialize, as if you were writing the original feature specification.] * This PR refactors existing code. [Describe what was wrong with the existing code, if it is not obvious.] --> ### Tips for reviewer <!-- Leave some tips for your reviewer, like: * The diff is much smaller if viewed with whitespace hidden. * [Some function/module/file] deserves extra attention. * [Some function/module/file] is pure code movement and only needs a skim. Delete this section if no tips. --> ### Checklist - [ ] This PR has adequate test coverage / QA involvement has been duly considered. ([trigger-ci for additional test/nightly runs](https://trigger-ci.dev.materialize.com/)) - [ ] This PR has an associated up-to-date [design doc](https://github.com/MaterializeInc/materialize/blob/main/doc/developer/design/README.md), is a design doc ([template](https://github.com/MaterializeInc/materialize/blob/main/doc/developer/design/00000000_template.md)), or is sufficiently small to not require a design. <!-- Reference the design in the description. --> - [ ] If this PR evolves [an existing `$T ⇔ Proto$T` mapping](https://github.com/MaterializeInc/materialize/blob/main/doc/developer/command-and-response-binary-encoding.md) (possibly in a backwards-incompatible way), then it is tagged with a `T-proto` label. - [ ] If this PR will require changes to cloud orchestration or tests, there is a companion cloud PR to account for those changes that is tagged with the release-blocker label ([example](https://github.com/MaterializeInc/cloud/pull/5021)). <!-- Ask in #team-cloud on Slack if you need help preparing the cloud PR. --> - [ ] If this PR includes major [user-facing behavior changes](https://github.com/MaterializeInc/materialize/blob/main/doc/developer/guide-changes.md#what-changes-require-a-release-note), I have pinged the relevant PM to schedule a changelog post.

2025-07-08

Add support for configuring encryption for sql server (#32892) Adds support for configuring TLS encryption and custom CA certificate with MS SQL Server The following options are added to CONNECTION FOR SQL SERVER: - `SSL CERTIFICATE AUTHORITY` - is one more more certificates in PEM format. - `SSL MODE` - `disabled` - no encryption - `required` - encryption required, no certificate validation - `verify` - encryption required, validate server certificate using system certs - `verify_ca` - encryption required, validate server certificate using provided CA cert (requires `SSL CERTIFICATE AUTHORITY` ) ### Motivation Implements https://github.com/MaterializeInc/database-issues/issues/9205 ### Checklist - [x] This PR has adequate test coverage / QA involvement has been duly considered. ([trigger-ci for additional test/nightly runs](https://trigger-ci.dev.materialize.com/)) - [ ] This PR has an associated up-to-date [design doc](https://github.com/MaterializeInc/materialize/blob/main/doc/developer/design/README.md), is a design doc ([template](https://github.com/MaterializeInc/materialize/blob/main/doc/developer/design/00000000_template.md)), or is sufficiently small to not require a design. <!-- Reference the design in the description. --> - [ ] If this PR evolves [an existing `$T ⇔ Proto$T` mapping](https://github.com/MaterializeInc/materialize/blob/main/doc/developer/command-and-response-binary-encoding.md) (possibly in a backwards-incompatible way), then it is tagged with a `T-proto` label. - [ ] If this PR will require changes to cloud orchestration or tests, there is a companion cloud PR to account for those changes that is tagged with the release-blocker label ([example](https://github.com/MaterializeInc/cloud/pull/5021)). <!-- Ask in #team-cloud on Slack if you need help preparing the cloud PR. --> - [ ] If this PR includes major [user-facing behavior changes](https://github.com/MaterializeInc/materialize/blob/main/doc/developer/guide-changes.md#what-changes-require-a-release-note), I have pinged the relevant PM to schedule a changelog post.

expr: also encode MSE column names when serializing with protobuf This commit makes the `RustType for MirScalarExpression` also serialize column names, instead of ignoring them. Having column names show up in dataflow operators can be useful, and the fact that protobuf serialization doesn't roundtrip all MSE values is surprising. Also, in the future we would like to rely on derived serialization logic, rather than a hand-rolled one, and this special case makes that harder. The bulk of the changes is adjusting the tests. For SLTs, we just use the automatic rewriting. For testdrive that isn't available so we want to be a bit more careful to not make the testdrive output too susceptible to optimizer changes. In particular, wherever we match on the name of a dataflow-internal arrangement, we want to replace the column name because it may change if the optimizer's column name derivation logic changes. If we are matching on the name of an index export arrangement, using the plain column name is fine because it is unlikely to change as long as the index definition remains unchanged.

2025-07-02

Store len on RunMeta to aid in calculating new len during incremental compaction

2025-07-01

cluster-client: remove `TimelyConfig` protobuf impls This commit removes the protobuf support for the `TimelyConfig` type, as it is now dead code.

cluster*: replace `CreateTimely` cluster commands This commit replaces the `CreateTimely` command from both the storage and compute protocols, since the Timely runtime is now always initialized at process startup. We still need the protocol nonce, so we can't simply remove the `CreateTimely` command. We replace it with a new `Hello` command instead.

2025-06-30

controller: remove `ClusterStartupEpoch` This type is not used for anything anymore, so we can delete it and a bunch of epoch plumbing code in the controllers.

controller: replace `CreateTimely` epoch with a nonce This commit replaces the epoch transmitted in `CreateTimely` commands with a nonce. On the replica we need a reliable way to differentiate compute protocol iterations, which the epoch doesn't provide as it doesn't increase in read-only environments.

2025-06-24

Uniquely identified runs with UUIDs (#32783) Towards: https://github.com/MaterializeInc/database-issues/issues/9194 <!-- Describe the contents of the PR briefly but completely. If you write detailed commit messages, it is acceptable to copy/paste them here, or write "see commit messages for details." If there is only one commit in the PR, GitHub will have already added its commit message above. --> ### Motivation <!-- Which of the following best describes the motivation behind this PR? * This PR fixes a recognized bug. [Ensure issue is linked somewhere.] * This PR adds a known-desirable feature. [Ensure issue is linked somewhere.] * This PR fixes a previously unreported bug. [Describe the bug in detail, as if you were filing a bug report.] * This PR adds a feature that has not yet been specified. [Write a brief specification for the feature, including justification for its inclusion in Materialize, as if you were writing the original feature specification.] * This PR refactors existing code. [Describe what was wrong with the existing code, if it is not obvious.] --> ### Tips for reviewer <!-- Leave some tips for your reviewer, like: * The diff is much smaller if viewed with whitespace hidden. * [Some function/module/file] deserves extra attention. * [Some function/module/file] is pure code movement and only needs a skim. Delete this section if no tips. --> ### Checklist - [ ] This PR has adequate test coverage / QA involvement has been duly considered. ([trigger-ci for additional test/nightly runs](https://trigger-ci.dev.materialize.com/)) - [ ] This PR has an associated up-to-date [design doc](https://github.com/MaterializeInc/materialize/blob/main/doc/developer/design/README.md), is a design doc ([template](https://github.com/MaterializeInc/materialize/blob/main/doc/developer/design/00000000_template.md)), or is sufficiently small to not require a design. <!-- Reference the design in the description. --> - [ ] If this PR evolves [an existing `$T ⇔ Proto$T` mapping](https://github.com/MaterializeInc/materialize/blob/main/doc/developer/command-and-response-binary-encoding.md) (possibly in a backwards-incompatible way), then it is tagged with a `T-proto` label. - [ ] If this PR will require changes to cloud orchestration or tests, there is a companion cloud PR to account for those changes that is tagged with the release-blocker label ([example](https://github.com/MaterializeInc/cloud/pull/5021)). <!-- Ask in #team-cloud on Slack if you need help preparing the cloud PR. --> - [ ] If this PR includes major [user-facing behavior changes](https://github.com/MaterializeInc/materialize/blob/main/doc/developer/guide-changes.md#what-changes-require-a-release-note), I have pinged the relevant PM to schedule a changelog post.

2025-06-21

Uniquely identified runs with UUIDs ghstack-source-id: 2f85c4a4de6e295213800481d7577a170003d034 Pull-Request: https://github.com/MaterializeInc/materialize/pull/32781

Update (base update) [ghstack-poisoned]

Update [ghstack-poisoned]

2025-06-18

Merge branch 'main' into dov/fix-incremental-compaction

2025-06-17

Id based lookup

2025-06-14

compute,adapter: add peek response stash, for arbitrary-sized peek responses Currently, results of SELECT queries (aka. peeks) are sent back from the cluster to environmentd via the compute protocol and these results are fully materialized (stored in memory) in environmentd before sending them out to the client. This has several implications: - Sending large results "clogs up" the cluster/controller communication - The amount of memory we want to give to environmentd limits the size of results we can return In practice the above make it so we limit the size of results using a `max_result_size` parameter, and some larger customers are chafing against that. Specifically, peek responses are sent inline, in the response stream from `clusterd` to `environmentd`/Controller. As a `ComputeResponse::PeekResponse`. With this change, we can "stash" large peek responses and only send a handle to them back via the response stream. Large peek responses are stored using the new persist API for "free-standing" batches (batches that are never intended for appending to a shard) and read out using the also new `read_batches_consolidated`. One current limitation is that we can only stash results when the SELECT does _NOT_ have an ORDER BY. This is left as a follow up. We introduces a bunch of dyncfgs for controlling when and how the result stash is used: - Enable/disable: for staged rollout. - Peek stash threshold: above what result size we use the stash. This is deliberately separate from the existing `max_result_size` configuration and in practice I think we want a low-ish stash threshold and keep the current `max_result_size`. - Batching while writing: we don't want to block the timely worker thread by writing out the whole result in one go. Instead we have configurable batching and batch size. - Batching while reading: we read consolidated "chunks" out of the batches, when serving the result to the client. Here we also make the batch size configurable. Implements https://github.com/MaterializeInc/database-issues/issues/9180 Co-authored-by: Parker Timmerman <parker.timmerman@materialize.com>

2025-06-13

storage/pg: attempt exact snapshot count only when estimate is low enough We have recently seen cases where the exact count of rows is problematic when the target table contains many rows. The value of this exact count is not worth the resources spent since the result only informs the snapshot progress reporting. This PR changes the logic to always collect the estimated number of rows first and if it's below a threshold (currently one million) attempt an exact count as well. Otherwise we just use the estimate. I retained the flag to compeltely disable exact counting in case the current threshold is problematic. This PR also removes the assertion that the total number of ingested rows match the number of counted rows. This assertion effectively checks that PostgreSQL transactions work, which doesn't seem very useful and due to other reasons cause CI flakes. See MaterializeInc/database-issues#9362 Fixes MaterializeInc/database-issues#9362 Signed-off-by: Petros Angelatos <petrosagg@gmail.com>

2025-06-10

Incremental compaction redux

2025-05-22

WIP

2025-05-21

wire things together, hopefully

add proto serialize and deserialize

add stuff

wip

2025-05-19

[sql_server] handle primary keys from SQL Server (#32251) This PR updates the metadata we fetch for SQL Server tables to include which columns are primary keys and plumb this information into Materialize. ### Motivation While Materialize does not enforce primary keys, we can trust the upstream system and having the Unique constraint in Materialize can improve query planning Fixes https://github.com/MaterializeInc/database-issues/issues/9210 Progress towards https://github.com/MaterializeInc/database-issues/issues/8762 ### Checklist - [ ] This PR has adequate test coverage / QA involvement has been duly considered. ([trigger-ci for additional test/nightly runs](https://trigger-ci.dev.materialize.com/)) - [ ] This PR has an associated up-to-date [design doc](https://github.com/MaterializeInc/materialize/blob/main/doc/developer/design/README.md), is a design doc ([template](https://github.com/MaterializeInc/materialize/blob/main/doc/developer/design/00000000_template.md)), or is sufficiently small to not require a design. <!-- Reference the design in the description. --> - [ ] If this PR evolves [an existing `$T ⇔ Proto$T` mapping](https://github.com/MaterializeInc/materialize/blob/main/doc/developer/command-and-response-binary-encoding.md) (possibly in a backwards-incompatible way), then it is tagged with a `T-proto` label. - [ ] If this PR will require changes to cloud orchestration or tests, there is a companion cloud PR to account for those changes that is tagged with the release-blocker label ([example](https://github.com/MaterializeInc/cloud/pull/5021)). <!-- Ask in #team-cloud on Slack if you need help preparing the cloud PR. --> - [ ] If this PR includes major [user-facing behavior changes](https://github.com/MaterializeInc/materialize/blob/main/doc/developer/guide-changes.md#what-changes-require-a-release-note), I have pinged the relevant PM to schedule a changelog post.

2025-05-16

Merge pull request #32459 from bkirwi/lease-cleanup [persist] Lease cleanup

2025-05-15

cluster: remove old create_sockets implementation

Add pprof profile visualization tool - Introduced a new tool for symbolizing and visualizing pprof profiles using debug symbols from S3. - Updated `requirements.txt` to include `protobuf==5.29.3`.

2025-05-13

WIP checkin

2025-05-12

More serde exchangeable part It's currently possible, but a bit odd, to have an exchangeable part that doesn't have a lease attached. This switches us to using the "exchangeable" part directly, instead of inventing and immediately dropping the lease.

2025-04-24

Store active rollups and GC in state (#32301) Closes https://github.com/MaterializeInc/database-issues/issues/9193 <!-- Describe the contents of the PR briefly but completely. If you write detailed commit messages, it is acceptable to copy/paste them here, or write "see commit messages for details." If there is only one commit in the PR, GitHub will have already added its commit message above. --> ### Motivation <!-- Which of the following best describes the motivation behind this PR? * This PR fixes a recognized bug. [Ensure issue is linked somewhere.] * This PR adds a known-desirable feature. [Ensure issue is linked somewhere.] * This PR fixes a previously unreported bug. [Describe the bug in detail, as if you were filing a bug report.] * This PR adds a feature that has not yet been specified. [Write a brief specification for the feature, including justification for its inclusion in Materialize, as if you were writing the original feature specification.] * This PR refactors existing code. [Describe what was wrong with the existing code, if it is not obvious.] --> ### Tips for reviewer <!-- Leave some tips for your reviewer, like: * The diff is much smaller if viewed with whitespace hidden. * [Some function/module/file] deserves extra attention. * [Some function/module/file] is pure code movement and only needs a skim. Delete this section if no tips. --> ### Checklist - [ ] This PR has adequate test coverage / QA involvement has been duly considered. ([trigger-ci for additional test/nightly runs](https://trigger-ci.dev.materialize.com/)) - [ ] This PR has an associated up-to-date [design doc](https://github.com/MaterializeInc/materialize/blob/main/doc/developer/design/README.md), is a design doc ([template](https://github.com/MaterializeInc/materialize/blob/main/doc/developer/design/00000000_template.md)), or is sufficiently small to not require a design. <!-- Reference the design in the description. --> - [ ] If this PR evolves [an existing `$T ⇔ Proto$T` mapping](https://github.com/MaterializeInc/materialize/blob/main/doc/developer/command-and-response-binary-encoding.md) (possibly in a backwards-incompatible way), then it is tagged with a `T-proto` label. - [ ] If this PR will require changes to cloud orchestration or tests, there is a companion cloud PR to account for those changes that is tagged with the release-blocker label ([example](https://github.com/MaterializeInc/cloud/pull/5021)). <!-- Ask in #team-cloud on Slack if you need help preparing the cloud PR. --> - [ ] If this PR includes major [user-facing behavior changes](https://github.com/MaterializeInc/materialize/blob/main/doc/developer/guide-changes.md#what-changes-require-a-release-note), I have pinged the relevant PM to schedule a changelog post.

2025-04-23

Do the same dance for GC

2025-04-22

Consolidate need_rollup

WIP: optional

better backwards compat

2025-04-21

Store active rollups in state

2025-04-18

storage: flatten oneshot ingestions commands

storage: flatten RunSinks command Signed-off-by: Petros Angelatos <petrosagg@gmail.com>

storage: flatten RunIngestion command Signed-off-by: Petros Angelatos <petrosagg@gmail.com>

storage: flatten AllowCompaction command Signed-off-by: Petros Angelatos <petrosagg@gmail.com>

storage: flatten status updates responses Signed-off-by: Petros Angelatos <petrosagg@gmail.com>

storage: flatten frontier upper responses Signed-off-by: Petros Angelatos <petrosagg@gmail.com>

2025-04-17

[sql_server] add negative tests to ensure we don't allow some data types (#32249) We can't support replication of certain datatypes, e.g. `text`, because they don't provide us with the "before" value when replicating. This used to work but I accidentally regressed the behavior in https://github.com/MaterializeInc/materialize/pull/32181. This PR re-adds the check to ensure all columns we're intending to replicate are supported. Also an mzcompose testdrive case to exercise these unsupported types. ### Motivation Progress towards https://github.com/MaterializeInc/database-issues/issues/8762 ### Checklist - [ ] This PR has adequate test coverage / QA involvement has been duly considered. ([trigger-ci for additional test/nightly runs](https://trigger-ci.dev.materialize.com/)) - [ ] This PR has an associated up-to-date [design doc](https://github.com/MaterializeInc/materialize/blob/main/doc/developer/design/README.md), is a design doc ([template](https://github.com/MaterializeInc/materialize/blob/main/doc/developer/design/00000000_template.md)), or is sufficiently small to not require a design. <!-- Reference the design in the description. --> - [ ] If this PR evolves [an existing `$T ⇔ Proto$T` mapping](https://github.com/MaterializeInc/materialize/blob/main/doc/developer/command-and-response-binary-encoding.md) (possibly in a backwards-incompatible way), then it is tagged with a `T-proto` label. - [ ] If this PR will require changes to cloud orchestration or tests, there is a companion cloud PR to account for those changes that is tagged with the release-blocker label ([example](https://github.com/MaterializeInc/cloud/pull/5021)). <!-- Ask in #team-cloud on Slack if you need help preparing the cloud PR. --> - [ ] If this PR includes major [user-facing behavior changes](https://github.com/MaterializeInc/materialize/blob/main/doc/developer/guide-changes.md#what-changes-require-a-release-note), I have pinged the relevant PM to schedule a changelog post.

2025-04-15

[sql_server] Support `TEXT COLUMNS` and `EXCLUDE COLUMNS` (#32181) This PR updates the SQL Server source to support excluding columns or decoding them as text, specifically it does the following: * Updates `SqlServerColumnDesc` to support representing unknown columns. This allows users to replicate tables with a column type we don't recognize and specify said column in `EXCLUDE COLUMNS`. * Update `SqlServerRowDecoder` to support decoding from any type to text, adds some limited testing. * Updates parsing and purification to include `TEXT COLUMNS` and `EXCLUDE COLUMNS` in both the primary `CREATE SOURCE` statement (for round-tripping) and the generated subsource statements. ### Tips for reviewers I would start with `sql-server-util/src/desc.rs`, that's where the bulk of the changes are. This file contains the types that we use to represent columns and tables from SQL Server. ### Motivation Fixes a `TODO(sql_server1)` and adds a known feature. Progress towards https://github.com/MaterializeInc/database-issues/issues/8762 ### Checklist - [ ] This PR has adequate test coverage / QA involvement has been duly considered. ([trigger-ci for additional test/nightly runs](https://trigger-ci.dev.materialize.com/)) - [ ] This PR has an associated up-to-date [design doc](https://github.com/MaterializeInc/materialize/blob/main/doc/developer/design/README.md), is a design doc ([template](https://github.com/MaterializeInc/materialize/blob/main/doc/developer/design/00000000_template.md)), or is sufficiently small to not require a design. <!-- Reference the design in the description. --> - [ ] If this PR evolves [an existing `$T ⇔ Proto$T` mapping](https://github.com/MaterializeInc/materialize/blob/main/doc/developer/command-and-response-binary-encoding.md) (possibly in a backwards-incompatible way), then it is tagged with a `T-proto` label. - [ ] If this PR will require changes to cloud orchestration or tests, there is a companion cloud PR to account for those changes that is tagged with the release-blocker label ([example](https://github.com/MaterializeInc/cloud/pull/5021)). <!-- Ask in #team-cloud on Slack if you need help preparing the cloud PR. --> - [ ] If this PR includes major [user-facing behavior changes](https://github.com/MaterializeInc/materialize/blob/main/doc/developer/guide-changes.md#what-changes-require-a-release-note), I have pinged the relevant PM to schedule a changelog post.

2025-04-14

Password auth (#32131) This is the bones of self managed password auth. This is missing integration tests and documentation which will come next. You can test this locally like so: ```shell #this starts materialize with auth enabled on the external port but not the internal port $ bin/environmentd --bazel --reset -- --all-features --unsafe-mode --enable-self-hosted-auth #start a mz_system session, turn on sql support and create a user $ psql -U mz_system -h localhost -p 6877 materialize NOTICE: connected to Materialize v0.139.0-dev.0 Org ID: 1bd2c405-c638-44cc-b917-6d05dfb832ac Region: local/az1 User: mz_system Cluster: mz_system Database: materialize Schema: public Session UUID: fe6d2dbe-3a94-430d-82e1-5e79cb14b91f Issue a SQL query to get started. Need help? View documentation: https://materialize.com/s/docs Join our Slack community: https://materialize.com/s/chat psql (14.17 (Homebrew), server 9.5.0) Type "help" for help. materialize=> alter system set enable_self_managed_auth=true; NOTICE: variable "enable_self_managed_auth" was updated for the system, this will have no effect on the current session ALTER SYSTEM materialize=> create role foo with superuser password 'bar'; CREATE ROLE # Now connect over the port with auth enabled $ psql -U foo -h localhost -p 6875 materialize Password for user foo: ``` This begins implementing https://github.com/MaterializeInc/materialize/pull/32005 <!-- Describe the contents of the PR briefly but completely. If you write detailed commit messages, it is acceptable to copy/paste them here, or write "see commit messages for details." If there is only one commit in the PR, GitHub will have already added its commit message above. --> ### Motivation <!-- Which of the following best describes the motivation behind this PR? * This PR fixes a recognized bug. [Ensure issue is linked somewhere.] * This PR adds a known-desirable feature. [Ensure issue is linked somewhere.] * This PR fixes a previously unreported bug. [Describe the bug in detail, as if you were filing a bug report.] * This PR adds a feature that has not yet been specified. [Write a brief specification for the feature, including justification for its inclusion in Materialize, as if you were writing the original feature specification.] * This PR refactors existing code. [Describe what was wrong with the existing code, if it is not obvious.] --> ### Tips for reviewer <!-- Leave some tips for your reviewer, like: * The diff is much smaller if viewed with whitespace hidden. * [Some function/module/file] deserves extra attention. * [Some function/module/file] is pure code movement and only needs a skim. Delete this section if no tips. --> ### Checklist - [ ] This PR has adequate test coverage / QA involvement has been duly considered. ([trigger-ci for additional test/nightly runs](https://trigger-ci.dev.materialize.com/)) - [ ] This PR has an associated up-to-date [design doc](https://github.com/MaterializeInc/materialize/blob/main/doc/developer/design/README.md), is a design doc ([template](https://github.com/MaterializeInc/materialize/blob/main/doc/developer/design/00000000_template.md)), or is sufficiently small to not require a design. <!-- Reference the design in the description. --> - [ ] If this PR evolves [an existing `$T ⇔ Proto$T` mapping](https://github.com/MaterializeInc/materialize/blob/main/doc/developer/command-and-response-binary-encoding.md) (possibly in a backwards-incompatible way), then it is tagged with a `T-proto` label. - [ ] If this PR will require changes to cloud orchestration or tests, there is a companion cloud PR to account for those changes that is tagged with the release-blocker label ([example](https://github.com/MaterializeInc/cloud/pull/5021)). <!-- Ask in #team-cloud on Slack if you need help preparing the cloud PR. --> - [ ] If this PR includes major [user-facing behavior changes](https://github.com/MaterializeInc/materialize/blob/main/doc/developer/guide-changes.md#what-changes-require-a-release-note), I have pinged the relevant PM to schedule a changelog post.