Proto commits in pg-sharding/spqr

These 93 commits are when the Protocol Buffers files have changed:

2025-05-09

AUTO INCREMENT START syntax (#1169) Add syntax for spqr sequences start

The documentation is generated from this commit.

initial

The documentation is generated from this commit.

Syntax for SYNC REFERENCE TABLES (#1176) * syntax for sync * add protos

2025-05-08

add protos

2025-05-06

initial

2025-04-29

Add RETRY MOVE TASK GROUP command (#1149) * Add methods top EntityMgr & qdbCoordinator * Add RETRY MOVE TASK GROUP to gram * Add processing of retryMoveTaskGroup to meta * Add proto rpc for retryoveTaskGroup * FIx displaying of MoveTaskGroup in PsqlInteractor * Fix spell checker * Fix working with empty move task group * Fix feature test * Fix build

2025-04-28

Add proto rpc for retryoveTaskGroup

2025-04-25

Add proto rpc for retryoveTaskGroup

2025-04-16

Add schema storing metadata to distributed relations (#1125) Add schema field to qdb distributed relation; Add schema name to ATTACH RELATION statement Tested with regression tests. balancer.feature checks also advanced. Usage of schema for routing in upcoming commits

Implement many shit

Add schema name to all structs

2025-04-11

Sequences (#1111) Added sequence objects. Columns can be attached to sequence and then will be injected into INSERT queries as nextval --------- Co-authored-by: denchick <denchick@yandex-team.ru>

2025-04-10

drop sequence

2025-04-09

Implement ListSequences and rename ListAllSequences to ListSequences

reset

add sequences field to models and parser

generate next val in etcd

make sequence an attachable object

2025-03-24

Change REDISTRIBUTE KEY RANGE semantics (#1087)

Add Apply field to proto

2025-03-14

Add REDISTRIBUTE CHECK command (#1075)

Add Check to protospecks

2025-02-24

Add spell checker to workflows (#1036) * Add spell checker to workflows * Exclude some files * Add extra dictionaries * can not -> cannot * Remove apt-key from tests.yaml * Remove apt-key elsewhere * non-existent -> nonexistent * Fix Dockerfile * Add more dictionaries * Fix typos & add expect * Fix spelling * Fix build * More spelling fixes * More fixes * Apply proposed changes

2024-11-14

Add 'REDISTRIBUTE KEY RANGE' command (#789) Co-authored-by: Denis Volkov <3149929+Denchick@users.noreply.github.com>

2024-10-31

Refactor protospecs (#790) * Refactor protospecs * Fix feature tests

2024-09-16

Reference table initial infra code change (#769) * Reference table initial infra code change * Add test

2024-07-08

Change yacc and protos for multidim key ranges. (#545) * Change yacc and protos for multidim key ranges.

2024-04-02

Deprecated use of sharding rule proto & service (#518)

refac (#591)

2024-03-27

Add GetKeyRange method to proto & EntityMgr (#567) * Add GetKeyRange to proto * Add GetKeyRange to KeyRangeMgr * Added GetKeyRange method to all of KeyRangeMgr's * Changed id -> ids in GetKeyRange * Implement GetKeyRange in KeyRangeServiceServer's

2024-03-25

Added new spqr-balancer (#552) * Add new balancer * Add Balancer config * Added threshold values to balancer config * WIP * Changed metrics * WIP * WIP * General logic of generateTasks done * More additions * Refactoring * Refactoring + getShardCurrentState * Implement getHostStatus * Update key Ranges info * Implement getKeyRange * Refactoring * WIP on getStatsByKeyRange * Implemented getStatsByKeyRange * Implemented getTasks * Added TaskGroup type * Initial impl of executeTasks * Small refactoring * unificationType -> joinType * Added TasksService to proto-specs * Added TasksService to proto servers * Added internal tasks-related types, implemented Task proto services * Added Task-related methods to QDB * Declared Task-related methods to QDBs * Add Tasks to-from Db methods * Implemented TaskMgr methods * Implemented Task methods in MemQDB * Implemented Task methods in EtcdQDB * Implemented QDB-related methods in balancer * Re-written balancer to use internal Task types * Account for joinType in executeTasks * Add spqr-balancer app * Fixies * Fixed QDBs GetTaskGroup * Some logging alterations * Small interface check * Some renaming & refactoring * More shit * Fixed getting pg_is_in_recovery * Create pg_comment_stats extension on shards * Fix getHostStatus & add logging * Fixes * More small fixes * Fixes transferring from last key range * Fixed transfers of JoinNone type * Small fixes * Add balancer Makefile targets * Fix shard image's setup * Add balancer feature test * Fix working with mixed-case table names * fix boundaries in provider * Added more tests for balancer * Fixed bound selection in balancer * Added CPU test to balancer * Reduced balancer config * Added timeout to balancer config * Changed timeout in balancer configs for feature tests * Lint fix * Fix getting detailed cpu stats * Refactoring in NewBalancer * Ignore unreachable hosts rather then fail * Refactor MaxRelative & getCriterion funcs * Optimized getStatsByKeyRange by adding another index * Fix comment * Simpler-to-read formulas * Fixed fitsOnShard logic * Changed getShardToMoveTo signature to return bool instead of error * Oh no, my cum count... * Fix fitsOnShard logic * Simplify getAdjacentShards * Removed incorrect condition * use TaskGroup.Tasks as stack * Remove redundant comment * Fix moving to non-adjacent shard & more sophisticated test * Fix spqr-balancer description * Add TODO * Fixed balancer test's name * Fixed balancer feature test

2024-03-18

Drop key range upper bound from everywhere (#553) * Drop upper bound from gram * Fixed regress & feature tests * Drop UpperBound from internal & Proto * Fixed tests

2024-03-13

Change key range splitting/unification logic (#549) * Changed proto-specs * Change key range logic from left-right to base-appendage * Add SplitRight field to kr.SplitKeyRange * SplitRight -> SplitLeft * Process SplitLeft in KeyRangeMgr's * Fixed Unite in qdbCoordinator * Fixed feature tests

2024-03-12

Remove balancer (#547)

2024-02-28

Deprecate key range upper bound in grpc (#527) * Deprecate key range upper bound in protos * Fix lint in coordinator.go

Fix spqrdump & add feature tests (#526) * Fix & refactor spqrdump * Added tests & fixed spqrql * Added tests & fixed spqrql * Add ListAllKeyRanges to grpc * Add ListAllKeyRanges to KeyRangeServiceServer's * Process empty hash correctly in spqrql * Fixed DistributionFromProto * Add feature test for dumper & small fixes * Fix lint * Added health checks for routers in feature tests * Fixed ListAllKeyRanges in Adapter * spqr-dumper -> spqrdump * dumper -> spqrdump

2024-02-21

Multidimentional routing basics tmp Multidimentional routing basics

2024-02-16

Simplify & refactor insert on target list + insert on values scan rou… (#504) * Simplify & refactor insert on target list + insert on values scan routing * fix * fix * drop ununed code * fix lint

2024-02-15

Support hash functions in distributions. Add distribution key in inte… (#503) * Support hash functions in distributions. Add distribution key in interfaces * fix * Apply suggestions from code review Co-authored-by: Yury Frolov <57130330+EinKrebs@users.noreply.github.com> * Small fix --------- Co-authored-by: Yury Frolov <57130330+EinKrebs@users.noreply.github.com> Co-authored-by: Yury Frolov <yura200253@gmail.com>

2024-02-13

Forbid default distribution & coordinator fixes (#490) * Changed GetRelationDistribution in MemQDB * Changed unit tests for memQDB * Restored a unit test for memQDB * Changed regression tests execution * Fix router/begin regression test * Require distribution in sharding rule/key range definition * FIXUP_INTO begin test fix * FIXUP_INTO begin test fix * Fixed copy_routing regress test * FIXUP_INTO regression test exec * Fixed router/error regression test * Fixed router/joins regression test * Fixed router/mixed_routing regression test * Fixed router/multishard regression test * Changed & disabled router/routing_hint regression test * Removed absent test from schedule * Fixed router/shard_routing regression test * Fixed router/single_shard_joins regression test * Fixed router/target_session_attrs regression test * Fixed router/transactions regression test * Fixed router/with_tables regression test * Fixed coordinator test execution * Fixed console regression tests * ListDistribution -> ListDistributions * Implemented missing methods in coordinator.go * FIXUP_INTO etcdqdb changes * Added DistributionId to sharding rule proto * Fixed CreateDistribution in coordinator * Fix key range union in qdbCoordinator * Fixed default docker-compose.yaml * Fixed Split/Unite methods in QdbCoordinator * Fixed coordinator feature test * Fixed init.sql in feature tests * FIXUP_INTO some coordinator-related shit * More obvious "*-admin host setup in feature tests" * Fix move feature test * Fix move_recover feature test * Fix other feature tests * Fixed linter * Fixed unit tests * Fixed console regression tests expected output * more refactoring in Makefile * Removed setting default distribution in meta.ProcessDrop * Fixed stress test * Fixes for proto tests

2024-02-07

Added 'ALTER DISTRIBUTION ds_id DETACH RELATION rel_id' statement support (#484) * Add DetachRelation to ast.go * Add DetachRelation support to gram * Added AlterDistributionDetach method to QDB * Implemented AlterDistributionDetach method in MemQDB * Implemented AlterDistributionDetach method in EtcdQDB * Forbid re-attaching relation in EtcdQDB * Forbid re-attaching relation in MemQDB * Added AlterDistributionDetach to DistributionMgr * Added AlterDistributionDetach to PsqlInteractor * Added DetachRelation processing to meta.go * Fix & expanded regression tests * Added AlterDistributionDetach method to DistributionService * Generate artifacts * Implemented other EntityMgr's * Added unit tests to MemQDB * Fixed tests * Fixed lint * Changed AlterDistributionDetach GRPC method's signature * Fix regress tests * Expanded alter distribution regression tests * Check idempotency of relation detaching in unit tests

2024-01-29

Rename dataspace to distribution and refactor relating code (#455) * Add rename dataspace to distribution and refactor related code --------- Co-authored-by: Denis Volkov <3149929+Denchick@users.noreply.github.com>

2024-01-23

Implement GetCoordinator in adapter (#440) * Implement GetCoordinator in adapter * Implement updateCoord * delete todo

2024-01-15

Attach table to dataspace (#389) * Added AttachToDataspace & GetDataspace methods to qdb interface * Added AttachToDataspace & GetDataspace methods to etcd QDB * Added AttachToDataspace & GetDataspace methods to mem QDB * Added getAntRelation method to ProxyRouter * Get dataspaces in routeWithRules * Altered proto specs * Fix DummySessionParamsHandler * Fix unittests * Added AttachStatement to grammatics * Added AttachTable processing to meta.go * Added test for attach statement * Added regress test for table attaching * Fixed routeWithRules * Fixed DataspaceIsDefault * Fixes proxy_routing.go * Fixed unittests * Changed table namespace in qdb

2023-12-15

Better (#365) * Better * Fix * Better * Fix * Better

2023-11-29

Dataspace v3 (#322) * Rebase of some work * Add Delete Dataspace Support * fix * gen * correction * Add validation to add kr and shr * Correction * Fix lint test * refactor set command * refactor set command * Fix * fix regress * fir feature * fix feature * fix feature * fix feature * fix feature * fix feature * fix feature * fix feature * fix feature * change proto * gen * fix feature * fix feature * fix feature * fix feature * Refactor * Refactor * Refactor * Refactor * Refactor * Refactor * feature fix * feature fix * feature fix * change HARD flag * gen * fix regress * fix feture * to gen * gen * refactor * refactor * Add new test to delete dataspace * To gen * gen * implement dataspace methonds in adapter.go * refactor set command * Change error message * fix regress * fix tests --------- Co-authored-by: root <root@spqr-test.ru-central1.internal>

2023-09-11

Run coordinator together with router (#287) Dispatch coordinator goroutines with router binary start. Proxy all modifying requests to Coordinator via grpc. Run admin console in proxy way for several type of queries: All modify requests to router console should be proxied to coordinator via particular gRPC-call. It is very dangerous to allow user to modify router metadata, like key-ranges or sharding rules, etc. This only should by done by coordinator or via coord. API. All show requests, like SHOW CLIENTS etc should be processed locally. In future, we can support installation-wide show requests processing using additional syntax, like SHOW CLIENTS is processed by router but SHOW GLOBAL CLIENTS to be proccesed by coordinator (coord. can process such queries already).

2023-08-09

Coordinator Show clients/pools/backend_connections funcs (#265) * implement pools for coordinator * `show pools` & `show backend connections ` now shows router names * added units for coord pool * added feature for show pools etc.

2023-07-25

Show backend connections and minor refactor (#246) * Show backend connections and minor refactor * Implement router grpc handlers

Show clients on coordinator as well (#243) * show clients on coordinator as well * refactor

2023-01-19

Support add and list shards cmds in coordctl (#130) Example: reshke@oheijweoij:~/spqr$ ./coorctl AddShard --shard-id 10 --shard-hosts 'localhost:1234' ------------------------------------- create shard with id: 10 and hosts: [localhost:1234] ------------------------------------- reshke@oheijweoij:~/spqr$ ./coorctl ListShards ------------------------------------- 1 shards found router 10 serving on host group [localhost:1234] -------------------------------------

Refactor coordinator grpc services. Rename RoutersService to Topolog… (#129) * Refactor coordinator grpc services. Rename RoutersService to TopologyService Also, move generated proto specs out of router to pkg, as this code is sharable accross codebase Simple grpcurl call example reshke@oheijweoij:~$ ./grpcurl -plaintext 'localhost:7003' spqr.RouterService/ListRouters { "routers": [ { "id": "r1", "adress": "localhost:1234" } ] } now can be also done with ctl util reshke@oheijweoij:~/spqr$ ./coorctl AddRouter -e localhost:7003 --router-endpoint localhost:1234 ------------------------------------- create router with id: 19762ac60a45 ------------------------------------- reshke@oheijweoij:~/spqr$ ./coorctl ListRouters ------------------------------------- 1 routers found router 19762ac60a45 serving on address localhost:1234 ------------------------------------- reshke@oheijweoij:~/spqr$

2023-01-12

Add infrasture code and changes to support Hash-based routing (#105) * Add infrasture code and changes to support Hash-based routing Change grpc, sql-console and internal interfaces of sharding rule. * Change create sharding rule syntax to be backward-compatable * Fix sharding rule data model, move table name outside of sharding rule entry Fix regress tests to match new output msg * Fix tests

2022-08-03

Drop unused Query Service protos, since coordinator-router interaction using grpc services, not sql

2022-07-29

Refactor proxy router code, implement missing methods (#60) Do not open router to clients before initialized metadata with coordinator. Add router option, that prevents router from executing client queries before metadata was properly synced with coordinator. Do not change router behavior with no-coordinator setup

Add resolve key range protos (#59)

2022-07-27

Add missing coordinator and router functionality (#52)

Add drop sharding rules statement support, regenerate protos (#51) * Add drop sharding rules statement support, regenerate protos accordingly * Drop tests table before tests, to cleanup from prev runs

2022-07-26

Add sharding rules support in coordinator (#48) Add sharding rules support in coordinator, refactor code and regenerate proto.

2022-05-30

Balancer console (#33) * init console interface for Balancer App * support commands: * show key ranges * split key range * use spqrlog package to log messages

2022-05-20

regenerate proto (#32) * regenerate proto * Implement more basic coordinator - router interaction requests * add test * fix test

2022-05-19

Balancer App (#20) * balancer-db,-installation interaction * balancer-coordinator interaction * fix of moveKeyRange call in balancer * balancer app * fix(router): proxy queries that do not match any shard to the world storage * fix: update proto files and open ports to run coordinator * fix DDL operations * skip world queries * feat(coordinator): unregister routers * fix: implement the ShardingRules service * feat(coordinator): configure a new qrouter using coordinator * feat(coordinator): store ShardingRules and apply the existing rules to the router after registration * feat(balancer): implement shard service * feat(balancer): run balancer, register shards, init the DB, dockerize balancer * fix: list key ranges to use in balancer * fix: balancer db connections and queries to manage actions * fix: locking key ranges * feat: detect key ranges to lock, preparing a shard FDW * feat: transfer data, move key range, delete old key range * feat: unite key ranges * fix: add sharding key to the config, rewrite installation queries by removing hardcoded table values * fix data duplication on move table, make possible the parallel task running * fix: panic on assignment to entry in nil map, remove debug lines * adjust configs * fix config to pass tests * Merge master into shard-service Co-authored-by: Sviatoslav Ermilin <ermilin.c@yandex.ru>

2021-11-22

better protos

add some balancer code

refactoring

2021-11-19

better protos

add some balancer code

2021-11-10

Split key range request in coordinator

2021-11-01

refactor key range & sharding rules services

refactor grpc iface

2021-10-26

apply fmt

Implement a couple of router grpc iface methods

Router grpc iface

2021-10-15

regenerate proto proretly

gen protos for coordinator

stress tests fixes

2021-10-14

fix tests & update proto

2021-10-13

fix protos

update protos

coordinator basics

refactor

Change routing to match byte ranges

2021-10-11

Better

2021-10-08

proto for coordinator

2021-10-06

Move qdb code 2 coordinator

2021-10-05

Regenerate protos

2021-09-14

refactor protos + load mirroring basics

2021-09-09

Key range api basics

2021-09-08

Fix protos

2021-08-22

mv genproto/protos protos/

2021-08-09

refactor

2021-08-05

proto

some code Some ssl fixes protos protos protos