Proto commits in databendlabs/databend

These commits are when the Protocol Buffers files have changed: (only the last 100 relevant commits are shown)

2026-07-08

fix(query): cancel open private task runs on drop (#20109) * fix(query): cancel open private task runs on drop * fix(query): bound dropped task cleanup by task id * fix(query): clear stale task after edges on drop * fix(query): run task drop cleanup outside warehouse filter * fix(query): bound dropped run cleanup by monotonic task id Drop the describe_task recheck and the name-scoped-state guards on the DeleteTask path, restoring unconditional name-scoped cleanup of task afters and the schedule key. Bound cancel_open_task_runs by `task_id <= dropped_id` instead of an exact match: task ids are globally monotonic, so any generation recreated after a drop has a strictly larger id. This sweeps stale open EXECUTING runs (including deletes coalesced away by a later same-name drop) without ever cancelling a run that belongs to a newer, recreated task. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(query): add private task cancel function --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

2026-06-24

feat: meta: add protobuf storage variants (#20038) * feat: meta: add protobuf storage variants # Summary Add protobuf coverage for every `StorageParams` variant so storage metadata can encode all supported backends without falling through to an encode-time incompatibility error. # Details The storage protobuf schema now includes the previously missing external and in-memory storage variants. `StorageParams::None` uses the existing empty protobuf message because it carries no payload. The conversion layer now maps the new protobuf variants in both directions, preserving version checks for payload-bearing messages and keeping the empty none case payload-free. A v178 compatibility test decodes fixed binary protobuf messages for each new storage variant and verifies current round-trip encoding. * fix: meta: skip moka storage protobuf variant # Summary Leave `StorageParams::Moka` out of the new storage protobuf schema while keeping the other newly supported variants covered by v178 compatibility tests. # Details Moka has no current config parser path that creates it as persisted storage metadata, so adding a protobuf variant would create a compatibility promise for leftover storage plumbing. The conversion layer now preserves the unsupported-storage incompatibility path for Moka, while the metadata version text no longer claims coverage for every `StorageParams` variant. * fix: meta: drop removed none storage proto # Summary Remove the stale `StorageParams::None` protobuf coverage so the storage conversion layer matches the current meta storage enum. # Details `StorageParams::None` was removed by the updated base branch, but this PR still added a `StorageConfig.none` field, conversion arms, and a compatibility fixture for it. That made CI fail when `databend-common-proto-conv` tried to compile against the current enum. The v178 compatibility coverage now tracks only the storage variants that still exist in `StorageParams` and were added by this PR. * chore: meta: name storage proto variants # Summary Clarify the v178 metadata changelog entry by listing the exact `StorageParams` variants added to `StorageConfig` protobuf coverage. # Details The previous wording said only that additional variants were added, which hid the exact metadata compatibility surface. Naming the variants makes the version log precise for future compatibility reviews.

2026-06-08

feat(query): support arrow stage file formats (#19953) * feat(query): support arrow stage file formats * fix(query): record arrow copy status and metadata * chore(query): fix arrow reader clippy warnings * fix(query): avoid arrow stage prefix overmatch * fix(query): infer arrow schema past empty files * test(query): cover nested arrow batches * fix(query): load timestamp_tz arrow strings * test(query): avoid arrow nested stringify * fix(query): preserve arrow list child types * test(query): cast arrow timestamp_tz strings in copy * Support Arrow stage unload and empty scans * test(query): fix arrow unload overwrite options

2026-05-27

fix(query): improve private task compatibility (#19918) * fix(query): assign private task ids * fix(query): support private task scripts

2026-05-18

feat(query): support experimental table branch (#19551) * support table branch fix fix fix fix fix * remove HistoryTableBranchMeta * fix * fix review comments * fix review comments * fix * fix review comments * fix * fix review comments * fix * fix * fix * fix test * fix test * resolve conflict * fix review comments

2026-05-12

feat(storage): add role_arn support for OSS connections (#19824)

2026-05-08

feat(fuse): support per-column STATS_TRUNCATE_LEN for string statistics (#19815) * feat(fuse): support per-column STATS_TRUNCATE_LEN for string statistics Add STATS_TRUNCATE_LEN column option to CREATE TABLE DDL, allowing users to specify a custom string truncation length (1-4096) for column-level min/max statistics instead of the default 16 bytes. Changes: - Add STATS_TRUNCATE_LEN keyword to parser and column definition AST - Store field_stats_truncate_len in TableMeta (proto field 39) - Plumb truncate length through WriteSettings, BlockBuilder, and ColumnStatisticsState to gen_columns_statistics - Add trim_string_min/max_with_len helpers for custom-length trimming - Show STATS_TRUNCATE_LEN in SHOW CREATE TABLE output - Add proto-conv v175 test and sqllogictest coverage * z * z * z * z

2026-05-07

feat(query): add billing_usage_daily table function (#19784) * feat(query): add billing usage daily table function * update: tags/details with jsonb, add permission check * refactor args with start/end date (remove month) * updated with review * fix ci check

feat(auth): add key-pair authentication for users (#19786) * feat(auth): add key-pair authentication for users - New auth type: CREATE USER ... IDENTIFIED WITH key_pair BY '<pubkey>' - Key management: ALTER USER ... ADD/REMOVE PUBLIC_KEY - Supported algorithms: RSA (RS256), ECDSA (ES256/ES384), Ed25519 - Multiple keys per user with labels and fingerprints - SHOW PUBLIC KEYS FOR USER returns per-key details - DESC USER shows key count instead of fingerprints - Key-pair login via /v1/session/login returns session tokens - Direct query with Bearer JWT + X-DATABEND-AUTH-METHOD: keypair - Root user blocked from key-pair auth (centralized in UserApiProvider) - PEM round-trip validation ensures stored keys are parseable - Builtin users validated with full PEM parse - MySQL error message correctly omits FlightSQL - JWT requires sub, iat, exp (iss not required) * refactor(proto-conv): extract helper functions for PublicKeyEntry conversion * fix: add version to proto * fix(proto): remove unnecessary ver/min_reader_ver from PublicKeyEntry * chore: update gitignore

2026-04-19

fix(query): make CSV quote_style unload-only (#19720) * feat(query): support CSV QUOTE_MINIMAL unload (#17568) * fix(query): preserve CSV quote-minimal round trips * refactor(query): group CSV quote settings * test(query): update file format show output * fix(query): use CSV QUOTE_STYLE for quote_minimal (#17568) * chore: codefmt * fix(query): make CSV quote_style unload-only (#17568) * test(query): fix csv quote_style detailed output size (#17568) * test(query): fix csv quote_minimal escape output (#17568) * fix(query): use quote_not_null CSV style name (#17568) * fix(query): narrow CSV record delimiter quote check (#17568)

2026-04-10

feat: CSV/TEXT support option trim_space. (#19690) * feat: CSV/TEXT support option trim_space. * refactor: use trim_ascii. * refactor: use trim_ascii.

2026-04-08

feat: CSV/TEXT support encoding. (#19660) * feat: CSV/TEXT support encoding. * rename * use simdutf8. * rename

2026-03-26

feat(query): support experimental table tags for FUSE table snapshots (#19549) * feat: support table tag update update update fix test fix test fix test update update update fix test * fix test * fix test * update update * fix fix * update * fix * fix

2026-03-24

feat(stage): add TEXT file format params (#19588) * feat(stage): add TEXT file format support for empty_field_as null_display and output_header, headers. * chore: not display quote for TextFileFormatParams. * ci: test for TEXT params. * rename tsv to text.

2026-03-20

feat: rename TSV to TEXT. (#19580) - mainly use TEXT in source and docs later. - unload to text, use suffix .txt instead of .tsv. - keep TSV as a alias in SQL for user for compat.

2026-03-13

refactor: remove legacy table branch/tag implementation (#19534) * revert table branch and tag update update fix test update fix test update * fix test * fix test

2026-03-10

feat: support copy into lance dataset. (#19495)

2026-02-25

feat: add sandbox env injection case via cloud control (#19383) * feat: add sandbox env injection case via cloud control * feat: add worker ddl and cloud env mock * chore: codefmt * chore: add worker parse test * feat: support size/auto_suspend.. options for worker * feat: add options/suspend/resume for Alter Worker * fix: Normalize UNSET worker option names before planning * chore: codefmt

2026-02-24

feat: add default_warehouse option to user settings (#19456)

2026-02-07

refactor: replace in-tree meta core crates with external databend-meta v260205.1.0 * refactor: replace in-tree meta core crates with external databend-meta v260205.1.0 The meta core crates (`src/meta/core/`) — service, client, kvapi, raft-store, sled-store, types, runtime-api, and test-harness — are removed from the monorepo and replaced with the published external `databend-meta` crate family at v260205.1.0. This completes the separation of the meta-service into its own repository (databendlabs/databend-meta), reducing build coupling and enabling independent release cycles. All internal crate references are updated from `databend-common-meta-*` path dependencies to the external `databend-meta-*` crates via `[patch.crates-io]` git tag overrides. A new `databend-meta-ver` crate provides user-facing SemVer compatibility constants for `--cmd ver` display, decoupled from the CalVer protocol negotiation versions used internally by the external crate. Changes: - Delete `src/meta/core/` (service, client, kvapi, raft-store, sled-store, types, runtime-api, test-harness, kvapi-test-suite) - Add `databend-meta-ver` crate with `MIN_METASRV_VER_FOR_QUERY` and `MIN_QUERY_VER_FOR_METASRV` for user-facing version display - Remove version argument from `GrpcServer::create` calls (upstream API change) - Upgrade `databend-base` from v0.2.9 to v0.2.10 * feat: add query-meta version compatibility documentation Add `compat_docs/` with a 3-stage pipeline (extract → resolve → query) that builds compatible version range tables between databend-query and databend-meta. Source data is extracted from git tags, CalVer references are resolved to repo tags via external crate mappings, and the final output lists compatible version ranges for every release. Changes: - Add `extract_from_git.py` to extract min-compatible versions from git history - Add `resolve_versions.py` to merge external crate versions and resolve CalVer to SemVer - Add `find_compat.py` to query or enumerate compatible version ranges - Add pre-generated `query_to_meta.txt` and `meta_to_query.txt` lookup tables * chore: remove nextest config for removed databend-meta crate * M src/meta/ver/src/lib.rs * refactor: move compat_docs into src/meta/ver The compatibility documentation now lives alongside the version crate it documents, keeping version-related artifacts co-located under `src/meta/ver/`.

2026-01-30

feat: Sandbox UDF (#19301) * feat: cloud python udf * ci: add cloud udf ci * feat: import presign url for cloud udf imports * chore: codefmt * chore: add session setting for cloud UDF import presign ttl * chore: codefmt * chore: remove CloudScriptUDF reused ScriptUDF so that it can execute regardless of whether cloud is enabled. * chore: Generate specs from queries and migrate Docker builds to the cloud * chore: CloudScriptUDF rename to SandboxUDF * chore: CloudScriptUDF rename to SandboxUDF * chore: fix of review comment * chore: pre pulling base image on test-sandbox-udf.sh * chore: json to proto * chore: fix result test-sandbox-udf.sh * refactor: The ApplyUdfResourceRequest method is generalized, requiring only a python file to deploy sandbox UDFs * chore: fix unit test * chore: fix unit test * chore: Fix sandbox dependency resolution & request header saving * chore: remove unnecessary structures from proto and use tpl for string replacement to improve readability. * chore: udf rename to resource & remove unused function for simple_server.py * chore: improve sandbox imports and tests * chore: codefmt * chore: migrate sandbox udf tests to nox

2026-01-28

feat: support independent schema for table branches (#19269) * update * fix * fix * fix review comments * fix review comments * fix review comments

2026-01-25

refactor: move 7 server-side meta crates to `src/meta/core/` (#19332) * refactor: move 7 server-side meta crates to `src/meta/core/` Move core server-side meta crates to a dedicated directory to better visualize dependencies before eventual migration to a separate repository. Changes: - Move `client`, `raft-store`, `runtime-api`, `service`, `sled-store`, `store`, `types` to `src/meta/core/` - Update workspace member patterns and dependency paths in `Cargo.toml` - Update excluded path in `taplo.toml` * fixup: relative path of TLS cert for tests

2026-01-13

feat(meta): add KvGetMany gRPC API; refactor: split chained awaits (#19244) * feat(meta): add `KvGetMany` gRPC API for batch key lookups Add bidirectional streaming gRPC API for looking up multiple keys. The client streams `KvGetManyRequest` messages (one key per message), and the server responds with a stream of `StreamItem` results. Changes: - Add `KvGetManyRequest` proto message and `KvGetMany` RPC - Implement `MetaLeader::kv_get_many()` for streaming key lookups - Add leader check in `MetaNode::handle_kv_get_many()` - Add `MetaHandle::handle_kv_get_many()` cross-runtime bridge - Implement gRPC service handler with token check and metrics - Add `EstablishedClient::kv_get_many()` client method - Add integration tests for leader, follower, and no-quorum scenarios - Fix: #349 * refactor: split chained await statements for readability Split chained `.await?.try_collect().await?` and `.await?.map_ok(...).try_collect().await?` patterns into separate statements across the codebase for improved readability and debugging. Changes: - Split patterns in `schema_api.rs`, `sequence_api_impl.rs`, `database_util.rs` - Split patterns in `kv_pb_api/mod.rs`, `tag_api.rs` - Split patterns in `compact_all.rs`, `leveled_map_test.rs` - Split patterns in `db_scoped_seq_bounded_read_test.rs`, `sm_v003_test.rs` - Split patterns in `compact_with_db_test.rs`, `store.rs` - Split patterns in `udf_mgr.rs`, `stage.rs` - Split patterns in `kvapi_test_suite.rs` - Split patterns in query service tests: `io.rs`, `commit.rs`, `truncate.rs`, `navigate.rs`, `index_scan.rs`, `index_refresh.rs`

2026-01-12

feat(meta): add `KvList` gRPC API for listing keys by prefix (#19235) * docs: add examples and sample output to metactl subcommand help Improve the CLI help messages for all `metactl` subcommands by adding usage examples and sample output. This helps users understand how to use each command without referring to external documentation. Changes: - Add description, example usage, and sample output for all 16 subcommands - Use `#[command(verbatim_doc_comment)]` to preserve formatting in help text - Include multiple examples where commands have different usage patterns * feat(meta): add `KvList` gRPC API for listing keys by prefix Add streaming `KvList` RPC that lists key-value pairs matching a prefix. Requires leadership - non-leaders return error with leader endpoint in metadata. Changes: - Add `KvList` RPC to `MetaService` proto with `KvListRequest` message - Add `kv_list()` method to `MetaLeader` for listing keys from state machine - Add `handle_kv_list()` to `MetaNode` with leader check and redirect - Add `status_forward_to_leader()` to `GrpcHelper` for error responses - Refactor `GrpcHelper` to share metadata parsing logic - Add `kv_list()` to `EstablishedClient` - Add integration tests for leader, follower redirect, and no-quorum scenarios * refactor(meta-client): add connection TTL to prevent h2 `too_many_internal_resets` error The h2 library has a default limit of 1024 locally-reset streams per connection. When this limit is reached, h2 sends a GoAway frame with ENHANCE_YOUR_CALM error and `too_many_internal_resets` message. This can happen under high concurrency when streams are cancelled or dropped before completion. This fix adds a connection TTL (default 20 seconds) to force periodic reconnection, preventing stream reset accumulation on long-lived connections. Changes: - Add `created_instant` field to `EstablishedClient` for TTL tracking - Add `is_expired()` method to check connection age - Add `connection_ttl` field to `MetaChannelManager` (default: 20s) - Check TTL in `MetaChannelManager::check()` and reject expired connections - Configure server-side `http2_max_pending_accept_reset_streams(4096)` as safety measure

2026-01-09

fix: distinguish quoted and unquoted empty and null string when loading CSV (#19207) * feat: CSV support options QUOTED_EMPTY_FIELD_AS and ALLOW_QUOTED_NULLS * feat: unload CSV support NULL_DISPLAY. * feat: `EMPTY_FIELD_AS=String` try set NULL for non-string columns. * ci: update tpcds logic test. * ci: update logic test. * skip compat test case

2026-01-06

feat: initial support for table branching and tagging (#19035) * branching and tag * table lvt * add test * update * fix review comments * fix * update

2025-12-31

feat: add tag DDL support (CREATE/DROP/SHOW only) (#19109) * feat: add tag DDL support (CREATE/DROP/SHOW only) * fix conversation * add tag <-> object ref * refactor * rename some struct * extract TagError * fix conversation * add reserved TagMeta::drop_on * split TagError, TagMetaError; add some comments * retry set_object_tags and drop concurrent tag error - set_object_tags now perform an optimistic retry with txn_backoff to keep allowed_values consistent with the latest tag meta. - remove the TagMetaConcurrentModification errors and corresponding exception code, instead retry logic processing. * meta: add explicit tag allowed_values enforcement flag - Proto3 `repeated` fields can’t distinguish `None` from `Some(Vec::new())`, so extending TagMeta with a dedicated `enforce_allowed_values` flag lets us tell whether a tag declared `ALLOWED_VALUES` even when the list is empty. - Split the semantics in Rust too: keep `allowed_values: Vec<String>` strictly for ordered values, and use the boolean to record if the constraint is active instead of overloading `Option<Vec<_>>`. - Update CREATE TAG interpreter, tag validation, error text, and system.tags display to respect the new structure, and refresh the proto conversion tests. * optimize - represent tag allowed_values as Option<Vec<String>> without an enforce flag - update tag.proto + conversions to the new AllowedValues message - simplify validation/error paths and keep system.tags display consistent

2025-12-05

feat(query): row access policy support rbac (#19064)

2025-11-27

feat: add column update_on for user functions table (#19018) * feat: add column update_on for user functions table * chore: codefmt * chore: codefmt * chore: codefmt * fix: create_on on bind_alter_udf * chore: codefmt * fix: 05_0013_ddl_alter_udf * chore: codefmt

2025-11-19

feat(query):masking policy support rbac (#18982) * feat(query):masking policy support rbac * add some comment * add some unit test

2025-11-18

feat: impl UDTF Server (#18947) * feat: impl UDTF Server * chore: rebase * chore: fix typo * chore: codefmt * chore: codefmt * fix: udtf single-row expectation

2025-10-29

feat: support DataType `TimestampTz` (#18892) * feat: support DataType `TimestampTimezone` * chore: add `to_timestamp_tz` from timestamp * chore: add interval function for timestamp_tz * chore: add meta test of TimestampTz * chore: fix bug on codex * chore: codefmt * chore: fix e2e test * chore: remove timestamp_tz on old datatype

2025-10-27

feat(query): masking policy USING clause with multi-column support (#18884) * feat(query): masking policy USING clause with multi-column support Enable USING clause to reference multiple table columns instead of only the masked column itself. Move masking policy application from storage read layer to SQL binder for unified handling across SELECT/WHERE/HAVING. Core changes: 1. Multi-column USING support with parameter substitution - Add clone_expr_with_replacement() to walk AST and replace column refs - Map policy parameters to USING columns by position - Support N policy args mapped to N USING columns 2. Move masking from read_plan to type checker (binder layer) - Apply masking in TypeChecker::resolve() for all query paths - Add in_masking_policy flag to prevent infinite recursion 3. Meta protobuf schema fixes (v155) - args: BTreeMap→Vec to preserve parameter order (was losing order) - Only write first USING column to column_mask_policy_columns_ids.key * fix * debug * optimize * try fix

2025-10-24

fix(query): fix row access policy parameter order and case sensitivity (#18889)

2025-10-16

feat: implement irreversible vacuum drop table protection (#18809) * feat: implement irreversible vacuum drop table protection * chore: resolve merge conflicts * fix: resolve merge conflicts * fix: type v153_vacuum_watermakr -> v154_vacuum_watermark

refactor: meta: unify FetchAddU64 into FetchIncreaseU64 (#18847) Rename FetchAddU64 to FetchIncreaseU64 and add `max_value` field to support unified atomic operation semantics. The new operation computes: `after = max(current_value, max_value) + delta`. This unified approach supports three use cases through a single operation: - Pure ADD: max_value=0, delta=N (backward compatible) - Pure MAX: max_value=N, delta=0 (new capability) - Combined: max_value=N, delta=M (ensures minimum then adds) The refactoring maintains backward compatibility through the existing `fetch_add_u64()` helper method which automatically sets max_value=0. Two new helper methods are added: - `fetch_increase_u64(key, max_value, delta)` for combined ops - `fetch_max_u64(key, max_value)` for pure max operations

refactor: simplify mask policy storage structure (#18836) * refactor: simplify masking policy storage structure 1. discard TableMeta::column_mask_policy 2. RowAccessPolicyColumnMap rename to SecurityPolicyColumnMap 3. old version use MaskPolicyTableIdListIdent store policy and table id reference. In pr use MaskPolicyTableIdIdent. * delete mask_policy_table_id.proto * fix flaky test * drop Option

2025-10-15

feat: External UDF support STAGE_LOCATION param (#18833) * feat: External UDF support STAGE_LOCATION param * chore: codefmt * ci: add test for external udf stage location param * chore: fix parse test * chore; fix e2e test * chore: fix e2e test * feat: Added Expr::StageLocation for passing StageLocation as a Function parameter * chore: codefmt * chore: add unit test for TableDataType::StageLocation

2025-10-13

chore: fix some meta typo (#18824)

2025-10-01

feat: meta-service: add `proposed_at` tracking the time when a key is written (#18812) * feat: meta-service: add `proposed_at` tracking the time when a key is written Add `proposed_at_ms` field to `KvMeta` protobuf to track proposal timestamps for key-value operations in the meta service. Introduce `ReduceSeqV` trait with two methods: `erase_proposed_at()` removes timestamps from metadata for test comparisons, and `reduce()` removes default-valued metadata to minimize storage overhead. Implement the trait for common types including `SeqV<T>`, `Option<SeqV<T>>`, collection types, and domain-specific types (`Change`, `pb::Event`, `pb::StreamItem`). Update all meta service tests to use `erase_proposed_at()` when comparing results, allowing tests to ignore variable proposal timestamps while verifying data correctness. Add comprehensive unit tests covering metadata erasure and reduction behavior. * refactor: update tests for KVMeta proposed_at_ms field Fixed compilation errors and test assertions after adding proposed_at_ms to KVMeta structure. Updated tests to handle the new field: - Updated KVMeta::new() calls to include proposed_at_ms parameter - Applied .erase_proposed_at() for SeqV types in assertions - Updated expected values to match database default (Some(0)) - Replaced legacy KVMeta::new_expires_at() calls - Updated JSON serialization test expectations All 94 raft-store tests now passing. * refactor: rename ReduceSeqV to NormalizeMeta with improved docs * M src/meta/service/tests/it/grpc/metasrv_grpc_kv_read_v1.rs * M tests/metactl/want_exported_v004 * M src/meta/client/src/lib.rs * M tests/metactl/subcommands/cmd_export_from_grpc.py

2025-09-30

feat: meta-service: add snapshot keys layout API with depth filtering (#18807) * feat: meta-service: add snapshot keys layout API with depth filtering Implement a new streaming gRPC API `snapshot_keys_layout` that provides hierarchical key distribution information from database snapshots. The API groups keys by directory prefix and returns counts at each level, with optional depth filtering to limit output granularity. Key components: - Streaming algorithm `count_prefix` that processes keys hierarchically and yields prefix counts incrementally as keys are processed - New `keys-layout` subcommand in `metactl` for command-line access Example usage: ```bash # Show all hierarchy levels metactl keys-layout --grpc-api-address 127.0.0.1:9191 # Show only top-level prefixes metactl keys-layout --depth 1 ``` Sample input keys: ``` databases/1/tables/users/columns/id databases/1/tables/users/columns/name databases/1/tables/orders/columns/id databases/2/tables/products/columns/id system/config/max_connections system/stats/query_count ``` API response (no depth limit): ``` databases/1/tables/orders/columns 1 databases/1/tables/orders 1 databases/1/tables/users/columns 2 databases/1/tables/users 2 databases/1/tables 3 databases/1 3 databases/2/tables/products/columns 1 databases/2/tables/products 1 databases/2/tables 1 databases/2 1 databases 4 system/config 1 system/stats 1 system 2 6 ``` API response (depth=1): ``` databases 4 system 2 6 ``` The API enables efficient analysis of key distribution patterns in large datasets without loading all keys into memory, supporting database administration and monitoring use cases. * M src/meta/client/tests/it/grpc_server.rs * M src/meta/service/src/analysis/count_prefix.rs

2025-09-29

feat(query): decouple row access policy argument names from column names (#18799)

2025-09-28

feat(query): RoleInfo support comment (#18788) * feat(query): RoleInfo support comment 1. create role support comment 2. support alter role 3. not support create or replace role ```sql ALTER ROLE [IF EXISTS] <role_name> SET COMMENT = '<string_literal>' | UNSET COMMENT CREATE ROLE [IF NOT EXISTS] <role_name> [COMMENT ='<string_literal>'] ``` * check if_exists in interpreter * fix flaky test

2025-09-26

feat: impl Keyword `AUTOINCREMENT` (#18715) * feat: impl Keyword `AUTO INCREMENT` * chore: codefmt * chore: replace table name to table id on auto increment sequence name * chore: codefmt * feat: support `IDENTITY` & drop auto increment sequence on vacuum * chore: fmt * chore: codefmt * chore: codefmt * chore: codefmt * chore: codefmt * chore: codefmt * chore: codefmt * chore: codefmt * chore: codefmt * chore: resolve review comment * chore: complete the auto increment sequence and create table in the same transaction * chore: codefmt * chore: codefmt * chore: codefmt * chore: codefmt * chore: codefmt * chore: codefmt * chore: add unit test `test_vacuum_dropped_table_clean_sequences` * chore: rebase * refactor: separate AutoIncrementIdent from SequenceIdent * chore: typo * chore: codefmt * chore: codefmt * chore: codefmt * chore: code comment on add column * chore: refactoring `AutoIncrementMeta` and add `AutoIncrementStorageIdent` * chore: codefmt * chore: codefmt * chore: remove auto_increment_display on `v3::TableField` * chore: force clean auto increment sequence on vacuum drop table * chore: codefmt * refactor: Implement AutoIncrementAPI * chore: codefmt * chore: merge main * chore: codefmt like review comment * chore: codefmt * chore: fix `get_auto_increment_next_value` & `next_val` return error * chore: add `auto_increment_api_test_suite` * chore: add comment on metadata.proto AutoIncrementExpr * chore: codefmt * chore: codefmt * chore: Increase the minimum version of meta to 1.2.764 to support `fetch_add` and enable AutoIncrement to work * chore: codefmt * chore: Increase the minimum version of meta to 1.2.768 to support `fetch_add` and enable AutoIncrement to work * chore: modify the column allocation method of auto increment expr and add auto increment check of add column * chore: codefmt

2025-09-24

refactor(query): Optimize Virtual Column Write Performance (#18752) * refactor(query): Optimize Virtual Column Write Performance * fix * fix * add tests * add sample check tests * fix build column fill default values * not generate new variant types

feat(meta-service): add snapshot V004 streaming protocol (#18763) * feat(meta-service): add snapshot V004 streaming protocol Implements KV-entry streaming for snapshot transmission, replacing V003's binary chunk approach in future. The new protocol streams individual key-value entries in protobuf format, enabling transfers and cross-format compatibility for rolling upgrades. Key improvements: - Memory efficient: processes entries as they arrive vs buffering entire snapshot - Cross-format support: enables rolling upgrades between snapshot formats - Fallback compatibility: automatically falls back to V003 for older nodes - Structured data: protobuf streaming vs raw binary chunks * M src/meta/service/src/meta_service/raft_service_impl.rs * M Cargo.lock * M src/meta/types/Cargo.toml

2025-09-18

feat(rbac): procedure object support rbac (#18730) * eat(rbac): procedure object support rbac * 1. replace p_id to procedure_id 2. add new function create_id_value_with_cleanup process ownership kv 3. modify test * no need to assert the seq of ownership key * refactor create_id_value * refactor cleanup_old_fn * fix

2025-09-03

feat: support Check Constraint (#18661) * feat: support Constraint * chore: codefmt * chore: codefmt * chore: codefmt * chore: codefmt * chore: codefmt * chore: codefmt

2025-08-28

feat(query): Supports fixed-length sort fields encoding (#18584)

2025-08-22

feat: implement Scalar UDF (#18575) * feat: implement Scalar UDF * test: add scalar udf parse test on `test_statement`

2025-08-18

feat: implement Tabular SQL UDFs (#18511) * feat: Implement UDTF * feat: udtf supports return types conversion * chore: codefmt * feat: impl UDTF Args replace * chore: codefmt * chore: codefmt * chore: rebase * chore: rebase

feat(query): ALTER TABLE Row Access Policy (#18540)

2025-08-08

feat(query): support create/drop/desc row policy (#18490) * feat(query):support create/drop/desc row policy * fix conversation * Introduce Experimental Feature Gate enable_experimental_row_access_policy default disable * RowAccessPolicyTableIdList -> RowAccessPolicyTableIdIdent __fd_row_access_policy_apply_table_id/tenant/<policy_id>/<table_id> * fix conversation * optimize Error

2025-07-29

feat: impl `WarehouseOptions` for Private Task (#18404) * feat: impl `WarehouseOptions` for Private Task * test: add test-private-task-warehouse.sh * chore: rebase * chore: rebase

feat(rbac): RBAC support Sequence object (#18423) * feat(rbac): RBAC support Sequence object 1. **New Configuration Parameter:** Introduces `enable_experimental_sequence_rbac_check` to toggle RBAC permission verification for sequences. Disabled by default for backward compatibility. 2. **Global Privileges:** Adds `CREATE SEQUENCE` and `ACCESS SEQUENCE` global privileges governing sequence creation and unrestricted usage rights respectively. 3. **Ownership Model:** Implements `OWNERSHIP` semantics, allowing privileged users/roles to perform arbitrary DDL operations on sequences. 4. **Show grants on sequence <sequence_name>.** * fix test * fix ci * fix ci * fix ci * QueryEnv add UserInfo

2025-07-24

feat(query): Add columns to display index sizes in `system.tables` (#18411) * tmp fix feat(query): Add columns to display index sizes in `system.tables` * fix tests

2025-07-23

feat(rbac): RBAC support connection object (#18382) * feat(rbac): RBAC support connection object 1. **New Configuration Parameter:** Introduces `enable_experimental_connection_rbac_check` to toggle RBAC permission verification for connections. Disabled by default for backward compatibility. 2. **Global Privileges:** Adds `CREATE CONNECTION` and `ACCESS CONNECTION` global privileges governing connection creation and unrestricted usage rights respectively. 3. **Ownership Model:** Implements `OWNERSHIP` semantics, allowing privileged users/roles to perform arbitrary DDL operations on connections. 4. **Show grants on connection <connection_name>. * modify test result about err msg * fix conversation * fix test * forbiden grant privileges on connection to USER Because of https://github.com/databendlabs/databend/pull/18400

2025-07-21

feat: impl Task for private (#18311) * feat: impl Task for private * feat: imp when_condition for Task & store TaskRun on system table * feat: use Meta's Watch mechanism to distribute TaskMessage * refactor: Using SQL to simplify DAG * chore: fix meta_store init * chore: log error on spawn * test: add test for private task * chore: add license for private task * chore: fix test_display_license_info * chore: codefmt * chore: add `accept` for Delete & After * chore: add restart test * chore: codefmt * chore: add `system.task_history` for Private Task & use `TaskMgr::accept` replace `TaskMetaHandle::acquire_with_guard` * fix: TableFunctionFactory create fail * ci: rename to private task test * chore: add cron test * feat: add system table: `system.tasks` * chore: fix `update_or_create_task_run` correct on `ScheduleTask` * chore: add Task when test on `test-private-task.sh` * chore: add private task config check on `GlobalServices::init_with` * chore: update Task version on Meta * chore: codefmt * chore: remove `TaskMgr::list_task_fallback` * chore: codefmt

2025-07-20

feat(meta): add VoteV001 RPC with native protobuf types (#18396) * feat(meta): add VoteV001 RPC with native protobuf types - Add VoteRequest and VoteResponse protobuf messages to raft.proto - Implement VoteV001 RPC endpoint alongside existing Vote RPC - Add conversion traits between protobuf and internal types - Implement backward compatibility with fallback to legacy Vote RPC - Update version compatibility matrix for new RPC method * chore: update change log of meta-service server * chore: fix lint * chore: fix lint

2025-07-17

feat(query): Enhanced UDF Functionality with IMMUTABLE Support (#18369) * feat(query): Enhanced UDF Functionality with IMMUTABLE Support * fix * fix tests

The documentation is generated from this commit.

2025-07-04

feat(meta-service): Add `PutSequential` operation for ordered key generation (#18312) Introduce `PutSequential` operation that creates sequential keys using a specified `prefix` and sequence generator. This enables applications to build ordered data structures like queues, which are essential for implementing distributed locks and semaphores. Operation details: - User provides a key `prefix` and sequence generator key - Sequence generator stores JSON-encoded `u64` values for `FetchAddU64` compatibility - Target key format: `<prefix>{next_val(<sequence_key>)}` - Key generation happens atomically during state machine application Use cases: - Distributed queue implementations - Ordered task scheduling - Distributed lock/semaphore mechanisms - Any scenario requiring guaranteed key ordering The operation leverages the existing conflict-free sequence generation infrastructure to ensure high concurrency and consistent ordering across distributed environments.

feat(meta-service): Add millisecond precision for expiration times (#18303) * feat(meta-service): Add millisecond precision for expiration times Extend expiration time support to accept both seconds and milliseconds while maintaining backward compatibility. The meta-service now stores both time formats in the existing `expire_at` field and automatically detects the format based on value magnitude. Detection logic: - Values > 100_000_000_000 are treated as milliseconds - Values ≤ 100_000_000_000 are treated as seconds This threshold corresponds to timestamps after ~1973, ensuring reliable differentiation between the two formats while supporting both legacy second-based expiration times and new millisecond-precision requirements. This is a non-breaking change that maintains full backward compatibility with existing configurations using second-based expiration times. * M Cargo.lock * M src/meta/types/src/cmd/meta_spec.rs * M src/meta/service/tests/it/grpc/metasrv_grpc_kv_read_v1.rs * M src/meta/service/tests/it/grpc/metasrv_grpc_kv_read_v1.rs * M src/meta/service/tests/it/grpc/metasrv_grpc_kv_api_restart_cluster.rs * M src/meta/types/src/proto_display/seqv_display.rs * M src/meta/types/src/proto_ext/seq_v_ext.rs * M src/meta/types/src/proto_ext/watch_ext/event_ext.rs * fix tests

2025-07-02

refactor(meta-service): Drop support for versions before 1.2.547 (#18284) Remove backward compatibility for meta-service versions prior to 1.2.547(exclusive) (released 2024-06-27). Direct upgrades from these older versions are no longer supported. Direct upgrades from versions 1.2.546 and earlier will fail. Users must upgrade through intermediate versions: - Unsupported: `<=1.2.546` → latest - Supported: `<=1.2.546` → `1.2.547` → latest This change simplifies the codebase by removing legacy compatibility code and reduces maintenance overhead for deprecated upgrade paths. And install-snapshot RPC v1(`install_snapshot_v1`) is removed too. Since it sends a V002 format snapshot.

2025-06-30

feat(meta): Add conflict-free sequence generator v1 (#18273) Introduce a new conflict-free sequence generator (storage version 1) alongside the existing version 0 generator. The v1 generator eliminates conflicts during concurrent sequence number generation by using a new `FetchAddU64` operation instead of check-and-set semantics. Key changes: - V1 generator stores sequence values in external standalone keys - Uses atomic FetchAddU64 operation for conflict-free updates - V0 generator remains unchanged for backward compatibility The FetchAddU64 operation performs in-place atomic updates without requiring metadata assertions, eliminating conflicts between concurrent sequence requests and improving throughput for sequence generation workloads. Both storage versions coexist, allowing gradual migration from the conflict-prone v0 implementation to the high-concurrency v1 design. **Non-Breaking**: This Commit just add this function but does not used anywhere.

2025-06-26

feat: Add ZIP compression type (#18257) * feat: Add ZIP compression type * chore: add meta test for Zip

feat(meta-service): add FetchAddU64.match_seq (#18258) If `match_seq: Option<64>` is specified, the record seq number must match the provided value to proceed `FetchAddU64` operation.

refactor: remove unused field step from sequence meta (#18245) * refactor: remove unused field `start` from `SequenceMeta` This is a **Non-Breaking** change. * refactor: nexval call on a sequence should not update the last-updated timestamp Because nextval is not a DDL operation.

2025-06-25

feat(meta-service): add FetchAddU64 command to txn (#18243) `FetchAddU64` fetch the `u64` value of a key and add provided delta to it, in one RPC. The response `FetchAddU64Response` returns the key, the value before update and the value after update.

2025-06-24

feat: parquet and avro add option USE_LOGIC_TYPE. (#18232) * feat: parquet and avro add option USE_LOGIC_TYPE in meta. * refactor: extract build_copy_into_table_input. * add util fn schema_date_time_to_int. * feat: parquet/avro support option USE_LOGIC_TYPE * chore: improve error of session lost. * add parquet tests * test with parquet_fast_read_bytes = 0. * avro tests * fix

2025-06-22

feat(query): support imports and packages in python udf scripts (#18187) * feat(query): support pep723 scripts in python udf scripts * feat(query): support pep723 scripts in python udf scripts * feat(query): support pep723 scripts in python udf scripts * chore(query): fix * update * update * add safe_codes in builder * add safe_codes in builder * fix binder

2025-06-11

feat(meta-service): return current seq-value in txn-put response (#18136) * feat(meta-service): return current seq-value in txn-put response TxnPutResponse now returns both the value before and after the put operation. Each returned value is a `SeqV` struct containing the sequence number and value in bytes. This is a backward-compatible change as protobuf-based clients will ignore the new field. * chore: normalize values before compare for kv-api tests

2025-06-09

refactor(meta-server): remove TxnReply::error (#18112) The `error` field is not used since 1.2.676; and is removed in this commit.

2025-06-03

feat(query): Support Vector data type (#18044) * feat(query): Support Vector data type * native format support vector * fix * remove impl ArgType

2025-05-23

chore(query): support config for storage network (#17983) * chore(query): support config for storage network * chore(query): support config for storage network * chore(query): support config for storage network * chore(query): support config for storage network * chore(query): support config for storage network

2025-05-18

feat(query): UserOption add workload_group (#17913) * feat(query): UserOption add workload_group * add more test

2025-05-13

chore(query): add more test to cover iceberg catalogs (#17905) * chore(query): add more test to cover iceberg catalogs * chore(query): add more catalog type for iceberg catalogs * chore(query): add more catalog type for iceberg catalogs * update * update * update * update * add iceberg logs * add iceberg logs * add catalog for system tables * updatel logs * updatel logs * fix

2025-05-09

feat(meta-service): add initialization complete flag for watch (#17900) When a watch requires initialization flush, the server now sends a flag message indicating when the initialization phase is completed. With this flag, the `cache` module can determine whether data has been fully initialized and is ready to serve, improving reliability of the caching system.

2025-05-08

chore(query): Clean up legacy virtual column codes (#17896) * chore(query): Clean up legacy virtual column codes * fix * fix

2025-04-23

feat(query): support create iceberg table with partition and properties (#17812)

2025-04-17

feat: impl `Create/Drop` Index for Ngram Index (#17789) * feat: impl `Create/Drop `Index for Ngram Index Signed-off-by: Kould <kould2333@gmail.com> * chore: codefmt Signed-off-by: Kould <kould2333@gmail.com> --------- Signed-off-by: Kould <kould2333@gmail.com>

2025-04-03

feat: impl udf server add custom header (#17691) * feat: impl udf server add custom header Signed-off-by: Kould <kould2333@gmail.com> * chore: codefmt Signed-off-by: Kould <kould2333@gmail.com> --------- Signed-off-by: Kould <kould2333@gmail.com>

2025-03-28

feat: support specifying compression when unloading to parquet. (#17664) compression = zstd | snappy default is zstd.

2025-03-13

refactor: extract map-api and watcher to separate crates (#17594) This commit tries to decouple the map-api and watcher from the state machine, so that they can be used in s3-meta-service. - Extract MapApi and related types to crate databend-common-meta-map-api - Move types: SeqValue, Marked, MapKey, MapValue, etc. - Move functions: compacted_get and compacted_range - Move watcher to separate crate databend-common-meta-watcher - Make watcher types independent of state machine implementation - Introduce generic data types for better abstraction

2025-03-12

refactor: add `VirtualDataScheam` to `TableMeta` (#17591) * feat: add `VirtualDataSchema` on `TableMeta` Signed-off-by: Kould <kould2333@gmail.com> * test(meta): add test for `VirtualDataSchema` Signed-off-by: Kould <kould2333@gmail.com> * test(meta): add test for `VirtualDataSchema` Signed-off-by: Kould <kould2333@gmail.com> * chore: remove redundant fields in protobuf Signed-off-by: Kould <kould2333@gmail.com> * chore: add virtual column id bound check Signed-off-by: Kould <kould2333@gmail.com> --------- Signed-off-by: Kould <kould2333@gmail.com>

2025-03-05

feat: support load Avro Files. (#17548)

2025-02-24

feat: object warehouse support rbac (#17262) * feat: object warehouse support rbac * add WAREHOUSE as ident note: only support_forward_warehouse_request will apply warehouse rbac. Now only support SystemResourcesManagement * add new privilege type forward test * add new ident parse func: grant_ident * 1. Normalize the code 2. grant ownership on warehouse object only supported for warehouses managed by the system * fix warehouse uid modify to id * add CreateWarehouse in available_privileges_on_global

2025-01-27

feat(query): Virtual column support alias name (#17365) * feat(query): Virtual column support alias name * fix * fix machete * add tests * fix tests * fix comments * fix

2025-01-22

feat: Add username support for webhdfs (#17350) * feat: Add username support for webhdfs Signed-off-by: Xuanwo <github@xuanwo.io> * Fix test Signed-off-by: Xuanwo <github@xuanwo.io> --------- Signed-off-by: Xuanwo <github@xuanwo.io>

feat: Add disable_list_batch for webhdfs (#17345) * feat: Add disable_list_batch for webhdfs Signed-off-by: Xuanwo <github@xuanwo.io> * Fix test Signed-off-by: Xuanwo <github@xuanwo.io> --------- Signed-off-by: Xuanwo <github@xuanwo.io>

2025-01-11

feat: support vacuum aggregating index (#17231) * mark delete index * vacuum drop index * add ut * fix show table function * add compatibility test * add logic test * move test to ee * simplify marked deleted index * fix test * modify comment * make lint * fix result set * remove unnecessary derived trait * add comment * put tenant at first * fix test

2024-12-28

chore: rename default_cluster_key_id to cluster_key_seq (#17135) * chore: rename default_cluster_key_id to cluster_key_seq * fix

2024-12-25

feat(query): support javascript/python script User Defined Aggregate Function (#17108) * init * update * update * update * update * refine parser * clear * bind aggregate * bind aggregate * update * test * UDFLanguage * update * fix * test * clean * python * test Signed-off-by: coldWater <forsaken628@gmail.com> * fix * init_state * fix * udaf server * remove UdafServer --------- Signed-off-by: coldWater <forsaken628@gmail.com>

2024-12-24

feat: databend-meta watch API provides `initial_flush` field (#17098) * chore: remove unused setting API * feat: databend-meta watch API provides `initial_flush` field If `WatchRequest.initial_flush` is `true`, when the watch stream is established, all the key values in the specified range will be sent through the stream to the watching client. This allows the client to setup a full copy of the key range as a client side cache, and furthur modifications will be received and the client side cache can be updated.

chore(query): Revert "feat(query): support javascript/python script User Defined Aggregate Function" (#17107) Revert "feat(query): support javascript/python script User Defined Aggregate …" This reverts commit 6770592e8dc9f1af11cfdd8bb21c9d945088e61b.

feat(query): support javascript/python script User Defined Aggregate Function (#17054) * init * update * update * update * update * refine parser * clear * bind aggregate * bind aggregate * update * test * UDFLanguage * update * fix * test * clean * python * test Signed-off-by: coldWater <forsaken628@gmail.com> * fix * init_state * fix --------- Signed-off-by: coldWater <forsaken628@gmail.com>

2024-12-20

feat: databend-meta transaction support generic bool-expression and else-if chain (#17064) Since this commit, application is allowed to specify a complex bool expressions as the transaction predicate. For example, the transaction will execute as if running the following pseudo codes: ``` if (a == 1 || b == 2) && (x == 3 || y == 4) { ops1 } else if (x == 2 || y == 1) { ops2 } else if (y == 3 && z == 4) { ops3 } else { ops4 } ``` ```rust let eq = |key: &str, val: &str| TxnCondition::eq_value(sample(key), b(val)); TxnRequest{ operations: vec![ BoolExpression::new( Some(eq("a", 1).or(eq("b", 2)) .and(eq("x", 3).or(eq("y", 4)))), ops1), BoolExpression::new( Some(eq("x", 2).or(eq("y", 1))), ops2), ], condition: vec![eq("y", 3), eq("z", 4)], if_then: ops3, else_then: ops4, } ``` For backward compatibility, both already existing `condition` and the new `operations` will be evaluated: transaction handler evaluate the `operations` first. If there is a met condition, execute and return. Otherwise, it evaluate `condition` and then execute `if_then` branch or `else_then` branch. TxnReply changes: Add field `execution_path` to indicate the executed branch, which is one of: - `"operation:<index>"`, operation at `index` is executed. - `"then"`: `if_then` is executed. - `"else"`: `else_then` is executed. `TxnReply.success` is set to `false` only when `else` is executed.

2024-12-16

feat: add key prefix count assertions to transactions (#17053) Enable transaction conditions based on the number of keys matching a prefix. Example: ```rust let txn = TxnRequest { condition: vec![ TxnCondition::match_keys_with_prefix("key/", Eq, 3) ], //... } ``` This allows transactions to proceed only when a prefix matches an expected number of keys, providing atomic prefix-based cardinality checks. This commit involves a databend-meta server side change: - Add `txn_condition::Target::KeysWithPrefix`. To provide compatibility, any change to the client that uses this feature must update the compatibility doc and upgrade the databend-meta cluster first.

2024-12-14

feat(query): add interval type (#16990) * support Interval DataType :) select to_interval('02:01'), to_interval('1 year 1 day 1 hour'); ┌───────────────────────────────────────────────────────────┐ │ to_interval('02:01') │ to_interval('1 year 1 day 1 hour') │ │ Interval │ Interval │ ├──────────────────────┼────────────────────────────────────┤ │ 2:01:00 │ 1year 1day 1:00:00 │ └───────────────────────────────────────────────────────────┘ * refactor: months_days_ns(pub i32,pub i32,pub i64) -> months_days_micros(pub i128)

2024-12-11

feat(rbac): add GrantObject::Warehouse (#17029) Warehouse permissions are granted to roles, and each warehouse can only be assigned to one role.