Proto commits in databendlabs/databend

These commits are when the Protocol Buffers files have changed: (only the last 100 relevant commits are shown)

2025-05-09

feat(meta-service): add initialization complete flag for watch (#17900) When a watch requires initialization flush, the server now sends a flag message indicating when the initialization phase is completed. With this flag, the `cache` module can determine whether data has been fully initialized and is ready to serve, improving reliability of the caching system.

The documentation is generated from this commit.

2025-05-08

chore(query): Clean up legacy virtual column codes (#17896) * chore(query): Clean up legacy virtual column codes * fix * fix

2025-04-23

feat(query): support create iceberg table with partition and properties (#17812)

2025-04-17

feat: impl `Create/Drop` Index for Ngram Index (#17789) * feat: impl `Create/Drop `Index for Ngram Index Signed-off-by: Kould <kould2333@gmail.com> * chore: codefmt Signed-off-by: Kould <kould2333@gmail.com> --------- Signed-off-by: Kould <kould2333@gmail.com>

2025-04-03

feat: impl udf server add custom header (#17691) * feat: impl udf server add custom header Signed-off-by: Kould <kould2333@gmail.com> * chore: codefmt Signed-off-by: Kould <kould2333@gmail.com> --------- Signed-off-by: Kould <kould2333@gmail.com>

2025-03-28

feat: support specifying compression when unloading to parquet. (#17664) compression = zstd | snappy default is zstd.

2025-03-13

refactor: extract map-api and watcher to separate crates (#17594) This commit tries to decouple the map-api and watcher from the state machine, so that they can be used in s3-meta-service. - Extract MapApi and related types to crate databend-common-meta-map-api - Move types: SeqValue, Marked, MapKey, MapValue, etc. - Move functions: compacted_get and compacted_range - Move watcher to separate crate databend-common-meta-watcher - Make watcher types independent of state machine implementation - Introduce generic data types for better abstraction

2025-03-12

refactor: add `VirtualDataScheam` to `TableMeta` (#17591) * feat: add `VirtualDataSchema` on `TableMeta` Signed-off-by: Kould <kould2333@gmail.com> * test(meta): add test for `VirtualDataSchema` Signed-off-by: Kould <kould2333@gmail.com> * test(meta): add test for `VirtualDataSchema` Signed-off-by: Kould <kould2333@gmail.com> * chore: remove redundant fields in protobuf Signed-off-by: Kould <kould2333@gmail.com> * chore: add virtual column id bound check Signed-off-by: Kould <kould2333@gmail.com> --------- Signed-off-by: Kould <kould2333@gmail.com>

2025-03-05

feat: support load Avro Files. (#17548)

2025-02-24

feat: object warehouse support rbac (#17262) * feat: object warehouse support rbac * add WAREHOUSE as ident note: only support_forward_warehouse_request will apply warehouse rbac. Now only support SystemResourcesManagement * add new privilege type forward test * add new ident parse func: grant_ident * 1. Normalize the code 2. grant ownership on warehouse object only supported for warehouses managed by the system * fix warehouse uid modify to id * add CreateWarehouse in available_privileges_on_global

2025-01-27

feat(query): Virtual column support alias name (#17365) * feat(query): Virtual column support alias name * fix * fix machete * add tests * fix tests * fix comments * fix

2025-01-22

feat: Add username support for webhdfs (#17350) * feat: Add username support for webhdfs Signed-off-by: Xuanwo <github@xuanwo.io> * Fix test Signed-off-by: Xuanwo <github@xuanwo.io> --------- Signed-off-by: Xuanwo <github@xuanwo.io>

feat: Add disable_list_batch for webhdfs (#17345) * feat: Add disable_list_batch for webhdfs Signed-off-by: Xuanwo <github@xuanwo.io> * Fix test Signed-off-by: Xuanwo <github@xuanwo.io> --------- Signed-off-by: Xuanwo <github@xuanwo.io>

2025-01-11

feat: support vacuum aggregating index (#17231) * mark delete index * vacuum drop index * add ut * fix show table function * add compatibility test * add logic test * move test to ee * simplify marked deleted index * fix test * modify comment * make lint * fix result set * remove unnecessary derived trait * add comment * put tenant at first * fix test

2024-12-28

chore: rename default_cluster_key_id to cluster_key_seq (#17135) * chore: rename default_cluster_key_id to cluster_key_seq * fix

2024-12-25

feat(query): support javascript/python script User Defined Aggregate Function (#17108) * init * update * update * update * update * refine parser * clear * bind aggregate * bind aggregate * update * test * UDFLanguage * update * fix * test * clean * python * test Signed-off-by: coldWater <forsaken628@gmail.com> * fix * init_state * fix * udaf server * remove UdafServer --------- Signed-off-by: coldWater <forsaken628@gmail.com>

2024-12-24

feat: databend-meta watch API provides `initial_flush` field (#17098) * chore: remove unused setting API * feat: databend-meta watch API provides `initial_flush` field If `WatchRequest.initial_flush` is `true`, when the watch stream is established, all the key values in the specified range will be sent through the stream to the watching client. This allows the client to setup a full copy of the key range as a client side cache, and furthur modifications will be received and the client side cache can be updated.

chore(query): Revert "feat(query): support javascript/python script User Defined Aggregate Function" (#17107) Revert "feat(query): support javascript/python script User Defined Aggregate …" This reverts commit 6770592e8dc9f1af11cfdd8bb21c9d945088e61b.

feat(query): support javascript/python script User Defined Aggregate Function (#17054) * init * update * update * update * update * refine parser * clear * bind aggregate * bind aggregate * update * test * UDFLanguage * update * fix * test * clean * python * test Signed-off-by: coldWater <forsaken628@gmail.com> * fix * init_state * fix --------- Signed-off-by: coldWater <forsaken628@gmail.com>

2024-12-20

feat: databend-meta transaction support generic bool-expression and else-if chain (#17064) Since this commit, application is allowed to specify a complex bool expressions as the transaction predicate. For example, the transaction will execute as if running the following pseudo codes: ``` if (a == 1 || b == 2) && (x == 3 || y == 4) { ops1 } else if (x == 2 || y == 1) { ops2 } else if (y == 3 && z == 4) { ops3 } else { ops4 } ``` ```rust let eq = |key: &str, val: &str| TxnCondition::eq_value(sample(key), b(val)); TxnRequest{ operations: vec![ BoolExpression::new( Some(eq("a", 1).or(eq("b", 2)) .and(eq("x", 3).or(eq("y", 4)))), ops1), BoolExpression::new( Some(eq("x", 2).or(eq("y", 1))), ops2), ], condition: vec![eq("y", 3), eq("z", 4)], if_then: ops3, else_then: ops4, } ``` For backward compatibility, both already existing `condition` and the new `operations` will be evaluated: transaction handler evaluate the `operations` first. If there is a met condition, execute and return. Otherwise, it evaluate `condition` and then execute `if_then` branch or `else_then` branch. TxnReply changes: Add field `execution_path` to indicate the executed branch, which is one of: - `"operation:<index>"`, operation at `index` is executed. - `"then"`: `if_then` is executed. - `"else"`: `else_then` is executed. `TxnReply.success` is set to `false` only when `else` is executed.

2024-12-16

feat: add key prefix count assertions to transactions (#17053) Enable transaction conditions based on the number of keys matching a prefix. Example: ```rust let txn = TxnRequest { condition: vec![ TxnCondition::match_keys_with_prefix("key/", Eq, 3) ], //... } ``` This allows transactions to proceed only when a prefix matches an expected number of keys, providing atomic prefix-based cardinality checks. This commit involves a databend-meta server side change: - Add `txn_condition::Target::KeysWithPrefix`. To provide compatibility, any change to the client that uses this feature must update the compatibility doc and upgrade the databend-meta cluster first.

2024-12-14

feat(query): add interval type (#16990) * support Interval DataType :) select to_interval('02:01'), to_interval('1 year 1 day 1 hour'); ┌───────────────────────────────────────────────────────────┐ │ to_interval('02:01') │ to_interval('1 year 1 day 1 hour') │ │ Interval │ Interval │ ├──────────────────────┼────────────────────────────────────┤ │ 2:01:00 │ 1year 1day 1:00:00 │ └───────────────────────────────────────────────────────────┘ * refactor: months_days_ns(pub i32,pub i32,pub i64) -> months_days_micros(pub i128)

2024-12-11

feat(rbac): add GrantObject::Warehouse (#17029) Warehouse permissions are granted to roles, and each warehouse can only be assigned to one role.

2024-11-28

feat(query): virtual column allow cast to other type (#16903) * feat(query): virtual column allow cast to other type * add tests * fix typos * fix

2024-11-21

refactor: add new raft-log metrics to "metactl status" response (#16899) These new metrics are added to gRPC API `GetClusterStatus` and corresponding databend-metactl CLI command. Newly added metrics are: ``` $ databend-metactl status # ... RaftLog: - CacheItems: 287 - CacheUsedSize: 4959439 - WALTotalSize: 4378076 - WALOpenChunkSize: 4378058 - WALOffset: 4378076 - WALClosedChunkCount: 1 - WALClosedChunkTotalSize: 18 - WALClosedChunkSizes: - ChunkId(00_000_000_000_000_000_000): 18 ```

2024-11-16

feat: add aws glue as an iceberg connection type (#16824) * added aws glue catalog as option for metastore * linted * removed unused dependencies * ran lint * added test * put toml files in order * added proto test * changed test to use glue as a catalog option * added props for test * got rid of extra block_in_place * linted --------- Co-authored-by: chase.grainger <chase.grainger@agvance.net> Co-authored-by: Ubuntu <ubuntu@ip-172-31-26-21.ec2.internal>

2024-11-13

refactor: align metrics field names with current behavior (#16826) - Rename `db_size` to `raft_log_size`, which is the size of on-disk data of raft-log. - Rename `key_num` to `snapshot_key_count`. This commit updates field names in - the the metrics API, - and the cluster status gRPC API.

2024-10-14

refactor: isolate session_id and session states of diff users. (#16592) * refactor: put session id under user_name when used as a key. * refactor: put session id under user_name when used as a key. * fix

2024-09-19

refactor: add DatabaseMeta.gc_in_progress (#16464) * refactor: add DatabaseMeta.gc_in_progress If `gc_in_progress` is set, no un-drop can be done on this database. Because the related data may already have been deleted. This commit does not use this flag yet, and will be used in next commit. - Part of #16433 * chore: fix up proto tests version: 109->110

feat(query): Procedure Part2 support arguments (#16453) * feat(query): Procedure Part2 support arguments * fix conversation * refactor var_to_ast with parse_expr

2024-09-05

feat(query): Procedure (Part1) (#16348) * feat(query): support procedure 1. support parse * Part2: support interpreter * refactor procedure mgr: use DataId * remove ProcedureIdList, refactor procedure_mgr * fix ut * remove serde::Serialize * fix conflict * add ProcedureIdentity * eaiser ProcedureIdentity en/decode * chore: fix list_dictionaries * fix conflicts

2024-08-22

feat: add endpoint /v1/session/logout and record client session ID in meta. (#16281) * rename TokenManager to ClientSessionManager. * meta add ClientSession. * SessionCtx add field client_session_id. * rename TokenMgr ClientSessionMgr. * add endpoint /session/logout * remove outdated comments. * http query check session id. * http query log client session id. * test logout. * impl<T> MemSized for Option<T> * fix merge up/main * fix test * small refactor * rm secondary_roles in LoginRequest. * fix http error code * change error code of AuthenticateFailure. * fix error code in test. * update tests. * update tests. * fix clippy

2024-08-21

feat(geo): add geography data type (#16286) * add geography type Signed-off-by: coldWater <forsaken628@gmail.com> * fix Signed-off-by: coldWater <forsaken628@gmail.com> * refine Signed-off-by: coldWater <forsaken628@gmail.com> * fix Signed-off-by: coldWater <forsaken628@gmail.com> * move wkb Signed-off-by: coldWater <forsaken628@gmail.com> --------- Signed-off-by: coldWater <forsaken628@gmail.com>

2024-08-14

feat: auth by refresh and session tokens. (#16220) * feat: auth by refresh and session tokens. * fix tests. * remove TokenApi. * polish comments * use Duration const instead of int. * small refactor * /login and /renew return enum. * clean unused code. * more secure nonce. * fix clippy. * remove unused code. * avoid use url path directly. * extract middleware_fn json_response from HttpSessionMiddleware. * fix test. * fix fmt.

2024-08-13

feat(meta): Support create dictionary meta api (#16178) * feat(query): support parse create dictionary stmt * feat(query): support parse create dictionary stmt * feat(query): support parse create dictionary stmt * feat(query): support parse create dictionary stmt * feat(query): support parse create dictionary stmt * feat(meta): support meta store proto * feat(meta): support meta store proto * feat(meta): support meta store proto * feat(meta): support meta store proto * feat(meta): support meta store proto * feat(meta): support meta store proto * feat(meta): support meta store proto * feat(meta): support meta store proto * feat(meta): support meta store proto * feat(meta): support meta store proto * feat(meta): support meta store proto * feat(meta): support meta store proto * feat(meta): support meta store proto * feat(meta): support meta store proto * feat(meta): support meta store proto * feat(meta): support meta store proto * feat(meta): support meta store proto * feat(meta): support meta store proto * feat(meta): support meta store proto * feat(meta): support meta store proto * feat(meta): support meta store proto * feat(meta): support meta store proto * feat(meta): support meta store proto * feat(meta): support meta store proto * feat(meta): support meta store proto * feat(meta): support meta store proto * feat(meta): support meta store proto * feat(meta): support meta store proto * feat(meta): support meta store proto * feat(meta): support meta store proto * feat(meta): support meta store proto * feat(meta): support meta store proto * feat(meta): support meta store proto * feat(meta): support meta store proto * feat(meta): support meta store proto * feat(meta): support meta store proto * feat(meta): support meta store proto * feat(meta): support meta store proto * feat(meta): support meta store proto * feat(meta): support meta store proto * feat(meta): support meta store proto * feat(meta): support meta store proto * feat(meta): support meta store proto * feat(meta): support meta store proto * feat(meta): support meta store proto * feat(meta): support meta store proto * feat(meta): support meta store proto * feat(meta): support meta store proto * feat(meta): support meta store proto * feat(meta): support meta store proto * feat(meta): support meta store proto * feat(meta): support meta store proto * feat(meta): support meta store proto * feat(meta): support meta store proto * feat(meta): support meta store proto * feat(meta): support meta store proto * feat(meta): support meta store proto * feat(meta): support meta store proto * feat(meta): support meta store proto * feat(meta): support meta store proto * feat(meta): support meta store proto * feat(meta): support meta store proto * feat(meta): support meta store proto * feat(meta): support meta store proto * feat(meta): support meta store proto * feat(meta): support meta store proto * feat(meta): support meta store proto * feat(meta): support meta store proto * feat(meta): support meta store proto * feat(meta): support meta store proto * feat(meta): support meta store proto * feat(meta): support meta store proto * feat(meta): support meta store proto * feat(meta): support meta store proto * feat(meta): support meta store proto * feat(meta): support meta store proto * feat(meta): support meta store proto * feat(meta): support meta store proto * feat(meta): support meta store proto * feat(meta): support meta store proto * feat(meta): support meta store proto * feat(meta): support meta store proto * feat(meta): support meta store proto

2024-08-06

feat: transfer leader command for meta-service (#16198) This commit introduce a new admin API(at `admin_api_address`, by default `127.0.0.1:28002`) to force the current Leader to transfer leadership to another node. The url of this command is "/v1/ctrl/trigger_transfer_leader?to=<new_leader_node_id>". Such request returns nothing no matter leadership is successfully transferred or not. If the node receives this request is not the leader it does nothing.

2024-08-03

feat: add share catalog (#16172) * refactor: refector ShareMeta struct * refactor: refector ShareMeta struct * refactor: refector ShareMeta struct * feat: add share catalog * feat: add share catalog * feat: add share catalog * feat: add share catalog * feat: add share catalog

2024-08-02

refactor: refector ShareMeta struct (#16100) * refactor: refector ShareMeta struct * refactor: refector ShareMeta struct * refactor: refector ShareMeta struct * add created_on in ShareDatabaseSpec * refactor ShareTable, add ShareReferenceTable * make clippy happy * make clippy happy * add ShareMetaV2 protocol test * make clippy happy * make clippy happy * merge from main * refactor by comment * refactor by comment

2024-07-13

feat: support must change password option for create user (#16031) * feat: support must change password for new user * fix tests * fix parser * add tests * fix * allow simple select * add comments * fix * fix

2024-07-10

feat: refactor share spec location and format (#15989) * feat: refactor share spec location and format * feat: refactor share spec location and format * feat: remove unused code * add db id into share path * refactor uri path * add proto conv test * add share test cases * refactor share db id * add rename\revoke database test cases * fix test fail

2024-07-09

refactor: upgrade QuotaMgr to using protobuf (#15858) * quota read Signed-off-by: coldWater <forsaken628@gmail.com> * set_quota Signed-off-by: coldWater <forsaken628@gmail.com> * refine Signed-off-by: coldWater <forsaken628@gmail.com> * update Signed-off-by: coldWater <forsaken628@gmail.com> * fix Signed-off-by: coldWater <forsaken628@gmail.com> * fix Signed-off-by: coldWater <forsaken628@gmail.com> * rename Signed-off-by: coldWater <forsaken628@gmail.com> * check_and_upgrade_to_pb Signed-off-by: coldWater <forsaken628@gmail.com> * WRITE_PB Signed-off-by: coldWater <forsaken628@gmail.com> * generic const parameter Signed-off-by: coldWater <forsaken628@gmail.com> * add it Signed-off-by: coldWater <forsaken628@gmail.com> * fix Signed-off-by: coldWater <forsaken628@gmail.com> * fix Signed-off-by: coldWater <forsaken628@gmail.com> --------- Signed-off-by: coldWater <forsaken628@gmail.com> Co-authored-by: Bohu <overred.shuttler@gmail.com>

2024-07-08

feat: parquet add option `missing_field_as`. (#15993) parquet add option `missing_field_as`.

2024-07-05

refactor: Use iceberg-rust to replace icelake (#15951) * Save current work Signed-off-by: Xuanwo <github@xuanwo.io> * Save part of work Signed-off-by: Xuanwo <github@xuanwo.io> * Save work Signed-off-by: Xuanwo <github@xuanwo.io> * Save work Signed-off-by: Xuanwo <github@xuanwo.io> * Save work Signed-off-by: Xuanwo <github@xuanwo.io> * Save current work Signed-off-by: Xuanwo <github@xuanwo.io> * Fix build Signed-off-by: Xuanwo <github@xuanwo.io> * remove openssl Signed-off-by: Xuanwo <github@xuanwo.io> * Fix build Signed-off-by: Xuanwo <github@xuanwo.io> * format cargo Signed-off-by: Xuanwo <github@xuanwo.io> * Fix tests Signed-off-by: Xuanwo <github@xuanwo.io> * Fix test for catalog Signed-off-by: Xuanwo <github@xuanwo.io> * Fix test Signed-off-by: Xuanwo <github@xuanwo.io> * iceberg engine will support later Signed-off-by: Xuanwo <github@xuanwo.io> * Pin upstream instead Signed-off-by: Xuanwo <github@xuanwo.io> * Fix sql Signed-off-by: Xuanwo <github@xuanwo.io> * Format cargo Signed-off-by: Xuanwo <github@xuanwo.io> * Fix create catalog should return error Signed-off-by: Xuanwo <github@xuanwo.io> * Add stateful tests for iceberg-rest Signed-off-by: Xuanwo <github@xuanwo.io> * Fix test Signed-off-by: Xuanwo <github@xuanwo.io> * Fix naming Signed-off-by: Xuanwo <github@xuanwo.io> * fix test Signed-off-by: Xuanwo <github@xuanwo.io> * Fix test Signed-off-by: Xuanwo <github@xuanwo.io> * Fix command Signed-off-by: Xuanwo <github@xuanwo.io> * Fix result Signed-off-by: Xuanwo <github@xuanwo.io> * Add test for proto Signed-off-by: Xuanwo <github@xuanwo.io> * Fix build Signed-off-by: Xuanwo <github@xuanwo.io> --------- Signed-off-by: Xuanwo <github@xuanwo.io>

2024-07-04

feat: orc add option missing_field_as. (#15974) * orc add option `missing_field_as`. * test parser. * fix clippy

2024-07-03

feat: refactor create database from share ddl (#15950) * feat: refactor create database from share ddl * fix compat read bug * fix test fail * feat: refactor create database from share ddl

fix(meta): fix reserved number err (#15955)

2024-07-02

feat: refactor create share endpoint ddl (#15937) * feat: refactor create share endpoint ddl * feat: refactor create share endpoint ddl * feat: refactor create share endpoint ddl --------- Co-authored-by: Bohu <overred.shuttler@gmail.com>

2024-06-27

feat: Persist Meta-Service State Machine on Disk (#15772) This commit introduces a significant change to the meta-service by persisting the state machine on disk rather than loading it entirely into memory. This change aims to reduce memory consumption to a configurable cache size and prevent memory bursts during snapshot creation. **Changes Implemented:** - **Disk-Based State Machine:** - The in-memory state machine has been replaced with a disk-backed implementation using a `rotbl` file (Read-Only on disk Table). - **Efficient Snapshot Building:** - When building a snapshot from the state machine, the output is now written directly to disk, which helps in avoiding double memory consumption. - **Add RPC install_snapshot_v003**: receive a V003 snapshot in binary form. - **New Configurations:** ```rust,ignore /// Whether to check if keys fed to the snapshot are sorted. snapshot_db_debug_check: bool, /// The maximum number of keys allowed in a block within a snapshot database. /// Smaller blocks enable more granular cache control but may increase the index size. snapshot_db_block_keys: u64, /// The total number of blocks to cache. snapshot_db_block_cache_item: u64, /// The total cache size for snapshot blocks, default is 1GB. snapshot_p_db_block_cache_size: u64, ``` **Compatibility:** - On startup, the old V002 snapshot will be upgraded to `V003`. - When receiving a V002 install-snapshot request, it is converted to `V003` after receipt and before installation to the state machine. - Fallback to transport snapshot-v1 if snapshot-v3 is not supported by remote peer. **Detailed Changes:** - **DB Wrapper:** - `DB` acts as a wrapper around `rotbl`, representing an on-disk snapshot and containing meta-service specific information such as metadata and file paths. - **SnapshotStoreV003:** - Provides various snapshot functionalities: - `Self::new_writer` creates a writer for building snapshots from a series of key-value pairs. - `Self::new_loader` creates a loader to load snapshots into the state machine from disk. - `Self::new_receiver` creates a receiver to handle snapshot data from remote peers in binary format. - **Data Export Changes:** - The `DataHeader` is no longer part of the exported snapshot data. - **MapKeyEncode Trait:** - Extracted from `MapKey`, this trait defines the codec behavior for a key used in a `MapApi`. - **Compactor Addition:** - Introduces a `Compactor` responsible for compacting in-memory immutable data or merging immutable data with persisted data. Cherry-picked changes: - refactor: move meta binaries to seperate dir - refactor: improve databend-metactl output

2024-06-21

refactor: Migrate catalog info into table info (#15857) Signed-off-by: Xuanwo <github@xuanwo.io>

2024-06-12

feat: parquet support option null_if (#15776) * feat: parquet support option null_if - meta. * feat: parquet support option null_if - add is_transform. * feat: parquet support option null_if - add processor. * feat: copy from parquet support option null_if.

2024-06-06

feat: add task_history function (#15731) * feat: add task_history function * chore: fix comment

2024-06-04

feat: basic support input file format ORC. (#15720) * feat: support input file format ORC. * fix * fix * fix * fix * fix * fix * fix * fix * fix * fix * fix * fix * fix * fix * fix

2024-05-30

feat(query): Role/UserInfo add field create_on and udpate_on (#15661) * feat(query): Role/UserInfo add field create_on and udpate_on In old version the time will be set to Utc::now(). * fix test & delete useless clone * fix ut err * fmt code

2024-05-23

feat: support millisecond level task scheduling (#15599) * feat: support millisecond level task scheduling * chore: fix unit test

2024-05-21

refactor: Remove `install_snapshot` v0 meta-service API (#15590) **Compatibility Changes:** This commit introduces changes to inter-meta-service compatibility, although compatibility between databend-query and meta-service remains unaffected. **Details:** - The initial snapshot transmission API, `install_snapshot` (v0), has been removed from both the meta server and meta client sides. - For clients, the use of `install_snapshot_v1()` transitions from **OPTIONAL** to **REQUIRED**. **Version Requirements:** - Post this commit, the meta-server requires clients to be at least version `1.2.212`, starting from which the clients began using `install_snapshot_v1()`. - Similarly, clients require the server to be at least version `1.2.212`, which is the version where `install_snapshot_v1()` becomes available.

2024-05-15

feat: add meta key_num metrics (#15536) * feat: add meta key_num metrics * feat: add meta key_num metrics

2024-05-05

feat: Add: stream based snapshot API install_snapshot_v2() to meta-service (#15395) feature: Add: stream based snapshot API install_snapshot_v2() to meta-service

2024-04-22

feat(query): implement ST_ASGEOJSON (#15214) implement ST_MAKEPOLYGON Signed-off-by: Fan Yang <yangfanlinux@gmail.com>

2024-04-19

feat: add sequence meta api (#15247) * feat: add sequence meta api * feat: add sequence meta api * feat: add sequence meta api * feat: add sequence meta api * feat: add sequence meta api * feat: add sequence meta api * feat: add sequence meta api * feat: add sequence meta api * feat: add sequence meta api * feat: add sequence meta api * feat: add sequence meta api * ci * feat: add sequence meta api

2024-04-17

feat(query): add user option: disabled (#15249) * add user option: disabled If you specify DISABLED = TRUE for a user: The user cannot log in again. ```sql -- u1 can not login create user u1 identified by '123' with disabled=true; -- u2 can not login alter user u2 with disabled=true; $ bendsql -uu2 -pabc123; Welcome to BendSQL 0.12.4-unknown. Connecting to localhost:8000 as user u2. Error: APIError: RequestError: Start Query failed with status 401 Unauthorized: {"error":{"code":"401","message":"AuthenticateFailure: user u2 is disabled. Not allowed to login"}} ``` * fix ut error

fix: show locks in current tenant (#15252) * show locks in current tenant * fix review commend * fix review commend

2024-04-16

refactor: replace String with Tenant in xxxPlan (#15240) * refactor: define IndexNameIdent with TIdent * refactor: define RoleIdent with TIdent * refactor: define UdfIdent with TIdent * refactor: use Tenant in CreateShareEndpointReq * refactor: replace String with Tenant in xxxPlan

2024-04-15

refactor: define DatabaseNameIdent with TIdent (#15235)

feat: Add azblob stage support Signed-off-by: Xuanwo <github@xuanwo.io>

2024-04-12

refactor: introduce `UninitTenant` and `TIdentRaw` (#15207) * refactor: introduce `UninitTenant` and `TIdentRaw` `Tenant` is a meta-service key component representing a tenant, with a embedded per-tenant config. `UninitTenant` is a shadow of `TIdent` but without config thus it can not be used as a key. `UninitTenant` can be initialized to build a `Tenant` for use as a meta-service key. Correspondingly, this commit introduce `TIdentRaw` as an uninitialized version of `TIdent`. `TIdentRaw` does not cantain per-tenant config and is only used inside meta-service values(not keys), which could be `serde`. Other changes: Populate usage of `Tennat` to `Catalog` implementation. * chore: fix wrong url in share * chore: fix rebase

2024-04-03

refactor: generate inverted indexs for each blocks (#15150) * refactor: generate inverted indexs for each blocks * fix * remove index refreshed_on * add index info file * fix * add offset numbers in index file * fix * fix * fix * fix * fix tests * refactor index info format * rename snapshot indexes as inverted_indexes * fix unit tests --------- Co-authored-by: Bohu <overred.shuttler@gmail.com>

2024-03-28

refactor: replace `NonEmptyString` with `Tenant` (#15103) * refactor: replace `NonEmptyString` with `Tenant` - Part of #14719

feat(query): Support hook refresh inverted index after insert (#15094) * feat(query): Support hook refresh inverted index after insert * fix * add meta test * remove settings * http test max_wait_time to 1s

2024-03-23

refactor: BackgroundTaskIdent use Tenant (#15080) Remove `impl FromToProto for BackgroundTaskIdent`, it does not need to be encoded to protobuf. - Part of #14719

2024-03-22

refactor: separate key and value: add `BackgroundTaskCreator` (#15063) refactor: add `BackgroundTaskCreator` `BackgroundJobIdent` is a kvapi::Key but not a value. It does not need to be `serde`. Thus in this commit we create another struct `BackgroundTaskCreator` with the same structure as `BackgroundJobIdent` for usage in values. And remove `serde` declaration from `BackgroundJobIdent`. - Part of #14719

2024-03-21

feat: support pagination on task_history (#15047)

2024-03-14

feat: ndjson support option null_if. (#14938)

2024-03-09

feat(query): create drop inverted index (#14859) * feat(query): create drop inverted index * fix typos * fix tests * fix tests * rewrite as table index * index use column ids * check column id exist

2024-03-07

feat: add transaction and mutli-line SQL script support for task (#14865) * feat: add multi-line SQLscripts support for task * feat: try add sql script support for task SQL * chore: fix unit-test

2024-03-06

feat: support notification syntax with rpc, and support task error notification integration (#14845) * feat: support notification rpc and task notification * feat: remove external library protoc dependencies * chore: apply comment rename utils.proto to timestamp.proto --------- Co-authored-by: Bohu <overred.shuttler@gmail.com>

2024-03-04

feat(query): introduce arrow-udf-js (#14799) * feat: support code string block * feat: support udf interpreter create * feat: support udf interpreter create * feat: update * feat: upgrade arrow to version 50 * feat: update * feat: update * feat: update * feat: update * feat: update * feat: update * feat: update * feat: update * feat: update * update * update * update * update * update * update * update * update * update * update * update --------- Co-authored-by: Bohu <overred.shuttler@gmail.com>

2024-03-01

feat: get_client_info() returns the server time in milliseconds (#14805)

2024-02-02

feat(geo): add geometry data type (#14470) add geometry data type

2024-02-01

feat: add column create_on for user functions table (#14540) * feat(meta): add created_on field to UserDefinedFunction Signed-off-by: Chojan Shang <psiace@apache.org> * feat(udf): add created_on field to show user functions Signed-off-by: Chojan Shang <psiace@apache.org> * feat(stateless): add timestamp test for udf Signed-off-by: Chojan Shang <psiace@apache.org> --------- Signed-off-by: Chojan Shang <psiace@apache.org>

2024-01-25

feat: meta-service: customize chunk size when export (#14467) * chore: update databend version for last meta-service API change * feat: meta-service: customize chunk size when export Use `databend-metactl --export --export-chunk-size <n>` to export all meta data with a stream in which each item contains at most `n` lines. Specify smaller value for `--export-chunk-size` if got an `gRPC message body too large error`. `--export-chunk-size` requries v1.2.315 and will be ignored when connecting to a older meta-service. * chore: clean unused dep: tonic * chore: fix missing method for meta-service tests

2024-01-24

feat: allow to config task session parameters. (#14446) * feat: support add session parameters during create task * support to show and alter session parameters * chore: add alter session parameter test

2024-01-22

feat: Remove the need of setting allow_anonymous (#14398) * feat: Remove the need of setting allow_anonymous Signed-off-by: Xuanwo <github@xuanwo.io> * Add test Signed-off-by: Xuanwo <github@xuanwo.io> * Fix test Signed-off-by: Xuanwo <github@xuanwo.io> --------- Signed-off-by: Xuanwo <github@xuanwo.io>

2024-01-21

fix(query): use serde_json maybe cause old query can not access data that generate by new query (#14380) 1. get method will try to deserialize with json, if failed, deserialize with pb 2. can not add or upsert will serialize with pb in this pr. Prevents an issue where old query cannot read data written by new query during a rolling upgrade. 3. add some log Co-authored-by: 张炎泼 <drdr.xp@gmail.com>

2024-01-19

feat: add task_depents and task_dependents_enable function (#14383) * feat: add task functions * feat: add task_dependents and task_dependents_enable function * chore: fix unit test on CI * chore: fix comment and migrate task relevant functions to cloud Dir * chore: fix license check

2024-01-18

refactor: do not use `TxnPutRequest|TxnDeleteRequest.prev_value`, always return previous value (#14371) Meta-service does not check `TxnPutRequest.prev_value` and `TxnDeleteRequest.prev_value` any more, and always assumes it is true. When a `Put` or `Delete` request is executed, the previous value before `Put` will always be responded.

2024-01-16

feat: CSV format add option binary_format and output_header. (#14329) * feat: CSV format add option `binary_format` and `output_header`. * ci: add tests. * fix clippy. * fix tests.

2024-01-13

refactor(query): delete owner in table/db meta (#14307) * delete owner in table/db meta get owner info in meta * add meta test --------- Co-authored-by: Bohu <overred.shuttler@gmail.com>

2024-01-12

feat: Add huggingface support (#14304) * Add huggingface config Signed-off-by: Xuanwo <github@xuanwo.io> * Add meta api Signed-off-by: Xuanwo <github@xuanwo.io> * Add test for huggingfaceconfig Signed-off-by: Xuanwo <github@xuanwo.io> * Allow parse huggingface Signed-off-by: Xuanwo <github@xuanwo.io> * Add create huggingface stage test Signed-off-by: Xuanwo <github@xuanwo.io> * Allow querying from huggingface Signed-off-by: Xuanwo <github@xuanwo.io> * Add commento for fields Signed-off-by: Xuanwo <github@xuanwo.io> --------- Signed-off-by: Xuanwo <github@xuanwo.io>

2024-01-09

feat: CSV add option empty_field_as. (#14269) * feat: CSV add option empty_field_as. * fix. * fix reading header only csv file. * fix date in META_CHANGE_LOG.

refactor: move GrpcHelper to crate common-types (#14277) * chore: minor refactor * refactor: move GrpcHelper to crate common-types `GrpcHelper` will be used by both meta-service and meta-client.

2024-01-08

chore: update doc about compatibility changes (#14260) Since databend-query 1.2.287, it requires at least meta-service 1.2.226 or newer.

2024-01-06

chore(query): fix password policy typos (#14246)

2024-01-04

feat(query): Support check password policy when login (#14124) * feat(query): Support check password policy when login * fix * add logs * add comments

2023-12-27

feat(expr): add Binary type (#14142) * feat(expr): add Binary type * fix * add parser * fix * fix * fix * fix * fix * fix * fix * revert arrow-related things * fix * fix * fix

2023-12-26

feat: support to build DAG on task DDL (#14141) * feat: add task dag sql semantic * feat: support task DAG * chore: fix comments

2023-12-23

fix(query): grant object use table/db id (#14016) * fix * add tablebyid and databasebyid * add func validate_table_access and validate_db_access 1. first to check db_name and table_name, if retrun err, check db_id and table_id. This is compatible with names stored in older versions 2. RewriteKind::ShowTables and ShowColumns need to save item catalog_name 3. information_schema and system.one remove from public role, skip these select in privilege::check and system.tables/databases/columns must store information_schema * add GrantObject::Table/DatabaseById meta proto test * Update src/query/service/src/interpreters/access/privilege_access.rs Co-authored-by: flaneur <me.ssword@gmail.com> * Update src/query/service/src/interpreters/access/privilege_access.rs Co-authored-by: flaneur <me.ssword@gmail.com> * fix_conversation --------- Co-authored-by: flaneur <me.ssword@gmail.com>

2023-12-21

refactor: refactor create agg index (#14085) * refactor create agg index * refactor create agg index * fix sqllogic tests * add some comments

2023-12-20

feat(query): Support create/alter/drop/desc/show password policy (#14012) * feat(query): Support create password policy * fix

2023-12-18

chore: fix license (#14060)

2023-12-16

feat: Introduced `MetaSpec` for setting relative expiration (#14041) This is a compatible change with new feature provided. - This commit introduces a new type, `MetaSpec`, which specifies content of the `KVMeta` to be stored for a key. This new type is particularly utilized in `upsert` requests to enable the setting of relative expiration times. Previously, the `KVMeta` type was used directly for this purpose. However, to avoid impacting existing storage data types, `MetaSpec` has been specifically added for use in `upsert` operations. When applying a raft-log, a `KVMeta` is built from `MetaSpec`. Designed with backward compatibility, `MetaSpec` maintains a serialized format compatible with `KVMeta`, ensuring no disruption to existing functionality. - We introduce two new types `Time` and `Interval` to reprensent serde-able time stamp and time interval. - Tests are added to ensure meta-service works correctly with API with ttl support, but databend-query does not use these API yet.