Proto commits in 4paradigm/OpenMLDB

These commits are when the Protocol Buffers files have changed: (only the last 100 relevant commits are shown)

2024-07-18

feat: merge main to v0.9 (#3961) * feat: sbin use the generated zk conf (#3901) Co-authored-by: lijiangnan <lijiangnan@4paradigm.com> * refactor!: relocate go sdk (#3889) * refactor!: relocate go sdk moving to https://github.com/4paradigm/openmldb-go-sdk * go readme * ci: fix sdk workflow * docs: fix example (#3907) raw SQL request mode example was wrong because execute_mode should be request * fix: make clients use always send auth info (#3906) * fix: make clients use auth by default * fix: let skip auth flag only affect verify * feat: tablets get user table remotely (#3918) * fix: make clients use auth by default * fix: let skip auth flag only affect verify * feat: tablets get user table remotely * fix: use FLAGS_system_table_replica_num for user table * fix: recoverdata support load disk table (#3888) * docs: add map desc in create table (#3912) * ci(#3904): python mac jobs fix (#3905) * fix(#3909): checkout execute_mode in config clause in sql client (#3910) * feat: merge dag sql (#3911) * feat: merge AIOS DAG SQL * feat: mergeDAGSQL * add AIOSUtil * feat: add AIOS merge SQL test case * feat: split margeDAGSQL and validateSQLInRequest * fix: gcformat space and continuous sign (#3921) * fix: gcformat space * fix: gcformat continuous sign use hash * fix: delete incorrect comments * feat: merge 090 features to main (#3929) * Set s3 and aws dependencies ad provided (#3897) * feat: execlude zookeeper for curator (#3899) * Execlude zookeeper when using curator * Fix local build java * Run script to update post release version (#3931) * feat: crud users synchronously (#3928) * fix: make clients use auth by default * fix: let skip auth flag only affect verify * feat: tablets get user table remotely * fix: use FLAGS_system_table_replica_num for user table * feat: consistent user cruds * fix: pass instance of tablet and nameserver into auth lambda to allow locking * feat: best effort try to flush user data to all tablets * fix: lock scope * fix: stop user sync thread safely * fix: default values for user table columns * feat(parser): simple ANSI SQL rewriter (#3934) * feat(parser): simple ANSI SQL rewriter * feat(draft): translate request mode query * feat: request query rewriter * test: tpc rewrite cases * feat(rewrite): enable ansi sql rewriter in `ExecuteSQL` You may explicitly set this feature on via `set session ansi_sql_rewriter = 'true'` TODO: this rewriter feature should be off by default * build(deps-dev): bump urllib3 from 1.26.18 to 1.26.19 in /docs (#3948) Bumps [urllib3](https://github.com/urllib3/urllib3) from 1.26.18 to 1.26.19. - [Release notes](https://github.com/urllib3/urllib3/releases) - [Changelog](https://github.com/urllib3/urllib3/blob/1.26.19/CHANGES.rst) - [Commits](https://github.com/urllib3/urllib3/compare/1.26.18...1.26.19) --- updated-dependencies: - dependency-name: urllib3 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * feat(udf): isin (#3939) * feat(#3916): support @@execute_mode = 'request' (#3924) * feat(udf): array_combine & array_join (#3945) * feat(udf): array_combine * feat(udf): new functions - array_combine - array_join * feat: casting arrays to array<string> for array_combine WIP, string allocation need fix * fix: array_combine with non-string types * feat(array_combine): handle null inputs * fix(array_combine): behavior tweaks - use empty string if delimiter is null - restrict to array_combine(string, array<T> ...) * feat: support batchrequest in ProcessQuery (#3938) * feat: user authz (#3941) * feat: change user table to match mysql * feat: support user authz * fix: cean up created users * build(deps-dev): bump requests from 2.31.0 to 2.32.2 in /docs (#3951) Bumps [requests](https://github.com/psf/requests) from 2.31.0 to 2.32.2. - [Release notes](https://github.com/psf/requests/releases) - [Changelog](https://github.com/psf/requests/blob/main/HISTORY.md) - [Commits](https://github.com/psf/requests/compare/v2.31.0...v2.32.2) --- updated-dependencies: - dependency-name: requests dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * build(deps-dev): bump org.apache.derby:derby (#3949) Bumps org.apache.derby:derby from 10.14.2.0 to 10.17.1.0. --- updated-dependencies: - dependency-name: org.apache.derby:derby dependency-type: direct:development ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * build(deps): bump org.postgresql:postgresql (#3950) Bumps [org.postgresql:postgresql](https://github.com/pgjdbc/pgjdbc) from 42.3.3 to 42.3.9. - [Release notes](https://github.com/pgjdbc/pgjdbc/releases) - [Changelog](https://github.com/pgjdbc/pgjdbc/blob/master/CHANGELOG.md) - [Commits](https://github.com/pgjdbc/pgjdbc/compare/REL42.3.3...REL42.3.9) --- updated-dependencies: - dependency-name: org.postgresql:postgresql dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * feat: iot table (#3944) * feat: iot table * fix * fix * fix delete key entry * fix comment * ut * ut test * fix ut * sleep more for truncate * sleep 16 * tool pytest fix and swig fix * fix * clean * move to base * fix * fix coverage ut * fix --------- Co-authored-by: Huang Wei <huangwei@4paradigm.com> * feat(open-mysql-db): pandas support (#3868) * feat(open-mysql-db): refactor 1. remove unnecessary instance var port 2. fix cause null bug 3. remove unnecessary throws 4. fix ctx.close() sequence bug 5. config sessionTimeout and requestTimeout 6. add docs of SqlEngine * feat(open-mysql-db): refactor * feat(open-mysql-db): revert passsword * feat(open-mysql-db): mock commit and schema table count * feat(open-mysql-db): replace data type text with string * feat(open-mysql-db): remove null --------- Co-authored-by: yangwucheng <yangwucheng@4paradigm.com> * fix: drop aggr tables in drop table (#3908) * fix: drop aggr tables in drop table * fix * fix test * fix * fix --------- Co-authored-by: Huang Wei <huangwei@4paradigm.com> * ci(#3954): fix checkout action on old glibc OS (#3955) * ci(#3954): fix checkout action on old glibc OS * ci: include checkout fix in all workflows * ci: fix python-sdk * test: node-2 to node-3 (#3957) node-3 is not available, moving to node-2 * feat: support locate(substr, str[, pos]) function(#820) (#3943) * fix(scripts): deploy spark correctly (#3958) $SPARK_HOME may be a symbolic link referring to a invalid directory, so we'd try 'rm -f' first * Add changelog for 0.9.1 (#3959) --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: venessa <854003762@qq.com> Co-authored-by: lijiangnan <lijiangnan@4paradigm.com> Co-authored-by: aceforeverd <teapot@aceforeverd.com> Co-authored-by: oh2024 <162292688+oh2024@users.noreply.github.com> Co-authored-by: HuangWei <huangwei@apache.org> Co-authored-by: wyl4pd <164864310+wyl4pd@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Huang Wei <huangwei@4paradigm.com> Co-authored-by: yangwucheng <yangwucheng1985@163.com> Co-authored-by: yangwucheng <yangwucheng@4paradigm.com> Co-authored-by: howd <81472844+howdb@users.noreply.github.com>

The documentation is generated from this commit.

2024-07-01

feat: iot table (#3944) * feat: iot table * fix * fix * fix delete key entry * fix comment * ut * ut test * fix ut * sleep more for truncate * sleep 16 * tool pytest fix and swig fix * fix * clean * move to base * fix * fix coverage ut * fix --------- Co-authored-by: Huang Wei <huangwei@4paradigm.com>

The documentation is generated from this commit.

2024-06-26

feat: user authz (#3941) * feat: change user table to match mysql * feat: support user authz * fix: cean up created users

2024-05-20

feat: crud users synchronously (#3928) * fix: make clients use auth by default * fix: let skip auth flag only affect verify * feat: tablets get user table remotely * fix: use FLAGS_system_table_replica_num for user table * feat: consistent user cruds * fix: pass instance of tablet and nameserver into auth lambda to allow locking * feat: best effort try to flush user data to all tablets * fix: lock scope * fix: stop user sync thread safely * fix: default values for user table columns

2024-04-15

feat: support insert function in offline mode (#3854) * native interface for insert function in offline mode * native entrance of insert plan * java entrance of insert plan * add insert plan * test cases * init test env * fix scala style check * support column with default value * refuse to insert into table with loaded soft copied data * only support default insert mode * update docs * fix test issue * fix test issue * fix test issue * fix test issue * fix test issue * update docs

2024-03-05

feat: map table column type (#3723) * feat: create table with map type columns * feat: insert stmt support for map data type simple insert row codegen support from #3731. * test: fix plan tests

2024-03-01

feat: main to v0.8 (#3785)

2024-02-29

feat(codec): encode/decode map value to codec row (#3713) * feat(engine): array & map data type interface for row level This updated proto defines for `hybridse::type::ColumnDef`, old data type field `ColumnDef::type`, as well as `ColumnDef::is_not_null` consider deprecated, and `ColumnDef::schema` consider the new data type for new codes. Old fields still workable but may going to removed in further release. * feat(codec): encode map type into row * feat: rm ImmutableMap

2024-02-22

feat: add deleteop command (#3702) * feat: add deleteop command * docs: add docs * fix: fix bug * docs: update docs --------- Co-authored-by: 4paradigm <4paradigm@denglong.local>

feat: delete data from all index (#3693)

2024-01-20

feat: put if absent & insert or ignore (#3692) - insert or ignore in c++/java - put if absent for load data - abort imediately in spark data writer - appendentries fix: delete or put - doc - refactor spark connector config setup

2024-01-15

feat: put returns failure if the server memory usage exceeds the specified ratio (#3631)

2023-11-23

feat: main -> v0.8 (#3611)

2023-11-15

feat: support truncate table statement (#3542)

feat: add zk auth (#3581)

2023-10-31

feat: full inspect (#3559) * feat: full inspect * docs: diag and must read

2023-09-19

feat: optimize java sdk (#3445)

2023-09-15

feat: optimize java sdk (#3445)

2023-07-20

fix: unsupport delete with multi index (#3374)

2023-07-18

feat: enhance the delete (#3301)

2023-06-29

feat: deploy with data (#3288)

2023-06-07

feat: deploy with data (#3288)

2023-05-11

feat: online to offline sync tool (#3256)

2023-05-08

feat: support symbolic paths (#3235) * Support symboic paths in offline table info and support load data with symbolic path * Show symbolic paths when desc table * Union data from symbolic paths * Support overwrite mode for soft copy * Fix load ut * Add check for different mode * Reset ut of test load data

2023-03-22

feat: diag tool supports sub commands and hosts & refactor (#3106)

feat: support external udaf function (#2825)

feat: support external udaf function (#2825)

2023-03-15

feat: add cases for taskmanger jobs (#3118) * Support sync job for show batch version api * Add batch job jar in test resource and add test for loading jar * Add ut for import online data and import offline data * Add more ut for taskmanger impl * Assert will check data of dataframe

2023-03-02

feat: diag tool supports sub commands and hosts & refactor (#3106)

2023-02-21

feat: main to v0.7 (#3077)

2023-02-17

feat: cache and show sync job output in taskmanager (#3034)

2022-11-04

feat: collect 0.6.5 features (#2760)

2022-11-03

feat: improve the error handling in sql cluster router (#2749)

2022-10-21

feat: collect 0.6.4 features (#2703)

2022-10-19

feat: delete record interface of specified timestamp (#2341)

feat: support online batch and full table aggregation (#2620) * enable online batch and full table aggregation * enable window cases, online full table and group by aggregation for cluster tests * bugfix: GetValue for RemoteWindowIterator * bugfix: DistributeWindowIterator multiple index * fixes #2633: failed to slidewindow if current_history_buffer_ is empty * fix(#2661): sum(col1) > 10 with group by in cluster online mode * add max_traverse_pk_cnt for DistributeWindowIterator, max_traverse_cnt in fulltable iterator * enable window cases with timestamp 0 * add docs for max_traverse_pk_cnt and max_traverse_cnt

2022-10-17

fix: traverse with same ts (#2637) * fix: traverse with same ts * test: add case

2022-09-27

Merge branch 'main' of github.com:dl239/OpenMLDB into feat/create_from_rtidb

2022-09-19

feat: create/drop database remote (#2488)

2022-08-24

feat: create table from rtidb

2022-07-14

feat: get batch version and find openmldb batch jar from spark home (#2164) * Find openmldb batch jar from spark home * Ignore the version cli case

2022-07-11

feat: support apiserver query, dbapi close and fix (#2091) apiserver query, POST, should set the execute mode in request body. e.g. `curl -X POST http://127.0.0.1:9080/dbs/airflow_example -d'{"sql":"select 1", "mode":"offsync"}'` apiserver get won't return ResultSet now, todo.

2022-07-08

feat: add taskmanager get version api (#2102) * Add new API for taskmanager to get version * Update rule for c to generate version string * Add ut for version util

2022-06-10

feat: support iterating key from remote tablet with `Next` function in `DistributeWindowIterator` (#1901)

2022-06-09

feat: add 0.5.2 features to v0.5 branch (#1936)

2022-06-08

fix: aggregator cleanup (#1919) * fix: aggregator cleanup * unique pre-aggr table name

2022-05-31

feat: support `count_where` op in aggregator (#1821) * feat: support aggregator filter * fix same aggr_col with different filter_col * change to two level aggr buffer

2022-05-26

feat: merge branch main to branch v0.5 (#1871)

2022-05-25

feat: iterate all data with distribute iterator (#1856)

2022-05-06

feat: update external function proto for offline (#1733)

2022-04-27

feat: rm offline_file (#1725)

2022-04-21

feat: support update tablets's global table when execute set command in standalone (#1647)

2022-04-20

feat: add disktable to tablet and nameserver (#1483) * special handling for storage mode * fix gc_interval * set flags_hdd/ssd_root_path default to "" * add GetCount for DiskTable * fix create root path Co-authored-by: Zhang Hao <zhanghao@4paradigm.com> Co-authored-by: litongxin <litongxin@4paradigm.com>

2022-04-17

feat: taskmanager create function (#1623)

2022-04-13

feat: add external c++ udf (#1509)

2022-04-08

feat(monitoring): openmldb prometheus exporter for v0.5.0 (#1584) * build(monitoring): setup openmldb python sdk as dep * feat(monitoring): OpenMLDB exporter collect table status, component status & deploy query response time metrics * refactor: make unit of times in DEPLOY_RESPONSE_TIME second * fix monitoring/CMakeLists

feat: support create table if not exists (#1460) * Support create table if not exists * Add default constructor and refine for pr comments * Add unit test for create table if not exists * Remove ambugulas method in sql cmd test * Remove the useless case in sql cmd test * Avoid register create table node for twice * Use set response ok instead of set response * Remove depcreated api to create table * Add usage of create table api * Fix usage of create table api

2022-04-07

feat: support get output from cmd offline query (#1481) * Support show offline query result from cmd * Combind stdout and stderror for taskmanager get log api * Add todo for execute offline sql api * Refine for code review comments * Remove the new api in sql router

feat: add storage_mode option in create table sql (#1558) * feat: add storage_mode option in create table sql * fix default storage mode * change storage_mode to enum * make StorageMode consistent with common.proto * fix some test

2022-04-04

feat: pre aggregator (#1472) * feat: pre aggregator * fix: make aggregator thread safety * fix: support null value type

2022-04-01

feat: deploy response time collector (#1521) user can enable deploy response time statistics collection through SQL `SET GLOBAL deploy_stats = 'on'` after that, all statistics for a given deployment query, will stored into builtin Table `INFORMATION_SCHEMA.DEPLOY_RESPONSE_TIME`

feat: restore 0.4.4 features (#1561)

2022-03-29

feat: long window plan (#1433) * feat: add long window optimize pass * add BuildAggTableDef in test util * disable merge window project lists in logical plan

2022-03-22

feat: support taskmanager sync rpc (#1486) * feat: support taskmanager sync rpc * add comment * cleanup * cleanup

2022-03-18

feat: create database if not exists (#1470) * upgrade zetasql to v0.2.10 support `IF NOT EXISTS` option for create database * feat(planner): add if not exists option for create database * feat(execute sql): add if_not_exists option for create database

2022-03-17

feat: disk table (#1287) * build: add rocksdb dep in third-party * feat: move back disk-table implementation * disk table test

2022-02-28

feat: add statement class (#1231) Add `HandleSQL` interface to `SQLRouter` class. Java/Python SDK can call this method to execute sql. - If execute success status.IsOK() will be true. - If there are some data returned, it will be packed in the ResultSet. `std::shared_ptr<hybridse::sdk::ResultSet> HandleSQL(const std::string& sql, ::hybridse::sdk::Status* status)`

2022-01-11

feat: add ambiguous unit test and improve ambiguous error message (#779) * feat: add ambigous unit test and improve ambigous error message * feat: add column not exist unit test

2022-01-07

feat: support job log file and fix delete hdfs file (#1045) Add new config to store the log files of jobs, resolve Make the log of spark applications from TaskManager in independent files #1026 Fix the bug when deleting hdfs file, resolve Fail to delete hdfs file and drop table for TaskManager #1044 Update the default log4j file to record Taskmanager log in rolling log file instead of stdout Support new api to get job log Support setting default spark conf in TaskManager, resolve Add default spark parameters in TaskManager for all Spark Jobs #1042

2022-01-05

export offline data in taskmanager server

2021-12-28

feat: add drop offline table api and update drop table in CLI (#966)

2021-12-27

fix: remove relevant code of NearLineTablet (#964)

2021-12-22

feat: support deploy for cluster mode (#921)

2021-12-21

Merge branch 'main' into issue-729

Merge pull request #932 from tobegit3hub/feat/add_taskmanager_client_for_cli feat: add taskmanager client for sdk and jobs commands for cli

Merge remote-tracking branch 'origin/main' into issue-729

2021-12-18

Merge commit 'd2c7ad3bd4f8d37c4bbf3a98e58e12b5565d125a' into feat/eliminate_performance_sensitive # Conflicts: # src/cmd/file_option_parser.h

2021-12-17

feat: openmldb-batch supports load data to online/offline

Merge branch 'main' of github.com:4paradigm/OpenMLDB into feat/add_taskmanager_client_for_cli

feat: support adding index with any ts col (#828)

2021-12-16

feat: Merge commit '16b426bcba18f70e083179f82db51e71e65d1bf6' into feat/eliminate_performance_sensitive

Add default_db for taskmanager jobs api

2021-12-14

Add deep_copy in offline table info

2021-12-09

feat: eliminate performance sensitive 1. eliminate set performance sensitive = true|false in cli 2. remove related API in C++ sdk and java sdk 3. add enable_request_performance_sensitive in engine option, for ddl gen to get a request physical plan performance-insensitive 4. fix yaml case, online batch queries can be executed performance insensitive

2021-12-02

Move offline table info from tablet.proto to name_server.proto

2021-11-30

Move the position of OfflineTableMeta

Add offline table meta in table meta

2021-11-25

Add new apis for taskmanager

2021-11-12

Merge branch 'main' into feat/column-default-value # Resolve Conflicts: # src/sdk/node_adapter.cc # src/sdk/node_adapter.h

2021-11-09

refactor: delete client proto (#659)

2021-11-01

fix: procedure fail to deal with multiple databases (#600)

2021-10-28

feat: add deploy command (#528)

2021-10-22

feat: standalone client supports to update catalog (#538)

2021-10-21

feat: add procedure type (#557)

feat: support column default value

2021-10-18

Merge branch 'main' into feat/openmldb_support_non_performance_sensitive_batch_mode

feat: openmldb support multiple databases (#534)

2021-10-13

Add is_performance_sensitive in tablet request and set false for commandline by default

feat: rm zk dependency in standalone mode (#486) * feat: create db and table in standalone mode

2021-10-08

fix: create index parse and desc (#480) * fix create index