Proto commits in alibaba/fluss

These commits are when the Protocol Buffers files have changed: (only the last 100 relevant commits are shown)

Commit:ed5b624
Author:pisceslj
Committer:GitHub

[client] Implement consumer group Admin APIs (#5267)

Commit:464a8f2
Author:yunhong
Committer:GitHub

[server] Preserve metadata identity across table recreation (#5173)

Commit:96faaf6
Author:Yang Wang
Committer:GitHub

[server] Keep KV recovery and retention on the same snapshot (#5229)

Commit:6b32e23
Author:yunhong
Committer:GitHub

[server] Report TabletServer software versions in describeCluster (#5184)

Commit:faeab32
Author:yangyang
Committer:GitHub

[rust] Add lake source contracts and UnionRead planning (#5158) * [rust] Add lake source contracts and UnionRead planning * [rust] Document lake-only UnionRead boundary coverage * [rust] Clarify the lake planning scope * [rust] Retrigger CI after Trino metadata readiness failure * [rust] Plan KV snapshot baselines and simplify UnionRead tests * [rust] Fix UnionRead task validation and PK baseline fallback * [rust] Refine UnionRead planning contracts and reuse core helpers

Commit:d111c61
Author:pisceslj
Committer:GitHub

[server] Implement consumer group management APIs (#5203) --------- Co-authored-by: Jark Wu <jark@apache.org> Co-authored-by: Codex <noreply@openai.com>

Commit:c56bef9
Author:yangyang
Committer:GitHub

[rust] Add reusable UnionRead source boundaries (#4887) * [rust] Add reusable UnionRead source boundaries * [rust] Simplify UnionRead preparation and reuse core APIs * [rust] Update dependency inventories for fluss-lake * [rust] Refine UnionRead boundary capture and cross-language contracts * Simplify UnionRead boundary capture with structured contexts * Simplify Rust read contexts and defer the Java boundary API * Refine UnionRead boundary imports and shared protocol * Simplify UnionRead tests and refresh dependency inventory * Reject UnionRead tables with shared historical logs

Commit:163719c
Author:Xiaobing Fang
Committer:GitHub

[metrics] Count per-bucket errors in TabletServer response metrics (#4360)

Commit:65261d6
Author:Jark Wu
Committer:Jark Wu

[server] Simplify consumer offset commits on tablet workers Consolidate offsets table initialization and group ownership checks in ConsumerGroupManager. Initialize the table during owner discovery. On commit, validate the local leader using cached table metadata and authorize reads on all source tables before appending on the RPC worker. Concurrent commits can proceed independently while replication is pending. Use the last entry for each bucket in a request. Report metadata size and write errors per bucket, including leadership changes. Encode metadata once as UTF-8, distinguish omitted metadata from an empty string, and reuse row and key encoders per worker thread. Co-Authored-By: Codex <noreply@openai.com> AI-Model: gpt-6 AI-Contributed/Feature: 1210/1503 AI-Contributed/UT: 1419/1961

Commit:7563804
Author:pisceslj
Committer:Jark Wu

[server] Implement consumer offset commits (#4850)

Commit:a61d2e4
Author:Yang Wang
Committer:GitHub

[client][server] Use explicit routing bucket counts (#4984) * [server] Reject incomplete bucket metadata cache hits * [client][server] Publish authoritative routing bucket counts * [rpc] Regenerate Rust metadata protobuf bindings * [client][server] Narrow routing metadata handling * [test] Verify scans after partial metadata recovery Read all three buckets through the recovered client snapshot and check payloads and offsets for both metadata response shapes. Remove the unused partition ID loader. * [test] Adapt routing metadata tests to partition info * [test] Cover legacy routing metadata recovery * [test] Preserve cache validation after batched metadata reads * [client][server] Clarify routing metadata contracts * [test] Clarify routing metadata rejection coverage * [client][server] Simplify metadata constructors Keep a single constructor in PartitionMetadata and Cluster. Update callers to pass the existing default bucket count and epoch metadata explicitly. Co-Authored-By: Codex <noreply@openai.com> AI-Model: gpt-6 AI-Contributed/Feature: 26/26 AI-Contributed/UT: 98/98 --------- Co-authored-by: Jark Wu <jark@apache.org> Co-authored-by: Codex <noreply@openai.com>

Commit:9ae427a
Author:Mahesh Sambaram
Committer:GitHub

[client] Expose standby coordinators via DescribeCluster (#4391) Report the currently live standby coordinators of a high-availability deployment through the existing DescribeCluster RPC instead of the Metadata RPC, so the metadata hot path stays unchanged. - Add `coordinator_standbys` to DescribeClusterResponse; the existing `coordinator_server` field still identifies the active coordinator. - The coordinator leader caches each live coordinator's registered address in CoordinatorContext, populated from the existing /coordinators/ids watcher and a batched read at startup, so describeCluster performs no ZooKeeper calls per request. - Add ClusterDescription#getCoordinatorStandbys() to the client API. A standby is listed while its registration in ZooKeeper exists, which does not guarantee it is ready to take over leadership. Coordinator ids are UUIDs, so standbys use node id 0 like the active coordinator. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

Commit:e79e5bf
Author:yunhong
Committer:GitHub

[client] Unify cluster inspection in DescribeCluster (#4901) Unify cluster nodes and tablet server replica loads in Admin.describeCluster, using RPC API key 1075. Keep getServerNodes on the Metadata discovery path and enforce cluster DESCRIBE authorization for cluster inspection. Document coordinator routing and request volume guidance, and update the protocol bindings and coverage for authorization, missing loads and retries. Classify NotCoordinatorLeaderException as InvalidMetadataException so retryable calls refresh cluster metadata before retrying after a leader change. Cover successful recovery and the one-retry limit. Co-Authored-By: Codex <noreply@openai.com> AI-Model: gpt-6 --------- Co-authored-by: Codex <noreply@openai.com>

Commit:b85258c
Author:Giannis Polyzos
Committer:GitHub

Revert "[Client] Support bounded log scanning (#4545)" (#4876) This reverts commit aa2283f5c2d7749bc61f160c3ca3b731cf141601.

Commit:e1c4f27
Author:Giannis Polyzos
Committer:GitHub

Revert "[Client] Support bounded log scanning (#4545)" This reverts commit aa2283f5c2d7749bc61f160c3ca3b731cf141601.

Commit:aa2283f
Author:yangshangqing
Committer:GitHub

[Client] Support bounded log scanning (#4545) * [Client] Support bounded log scanning * [Client] Improve bounded log scan completion and error handling * [Client] Migrate lake snapshot log reader to bounded scanning * [client] Preserve compatibility for bounded EARLIEST scans * [rust] Remove duplicate proto restored by rebase --------- Co-authored-by: Giannis Polyzos <ipolyzos.se@gmail.com>

Commit:c06677d
Author:Liebing
Committer:GitHub

Change the implementation of Admin#getServerNodes() to support carrying ServerTag (#1410) * [server] Refactor Admin#getServerNodes to support carrying ServerTag * [server] Serve server tags from coordinator cache with fallback * [server] Fix checkstyle empty line in ServerRpcMessageUtils --------- Co-authored-by: Giannis Polyzos <ipolyzos.se@gmail.com>

Commit:7dab023
Author:Muhammet Orazov
Committer:GitHub

[server][rpc] Support DescribeTabletServers API for safe scaling and upgrades (#3743) Add a read-only DescribeTabletServers RPC to the Coordinator, following the pattern of GetClusterHealth (#3399). It returns, per tablet server, the same four counters getClusterHealth() reports cluster-wide, scoped to that server: numReplicas, inSyncReplicas, numLeaderReplicas and activeLeaderReplicas. It also returns leaderlessReplicas: the number of replicas on that server whose bucket has no elected leader. This lets operational tooling (e.g. the Fluss Kubernetes Operator): - gate scale-in on a server being empty (numReplicas == 0), - gate rolling upgrades on a per-server green predicate (inSyncReplicas == numReplicas && activeLeaderReplicas == numLeaderReplicas && leaderlessReplicas == 0), - report per-server tablet load in cluster status. Every reported server is green exactly when getClusterHealth() reports GREEN, so a single response is enough for a rolling-upgrade gate. The counters are computed from in-memory CoordinatorContext state on the coordinator event thread via AccessContextEvent. Tablet servers forward the request to the coordinator. Live and shutting-down servers are always reported, even with zero replicas, and dead servers that still hold assigned replicas remain visible so a scale-in gate cannot treat them as drained. Closes #3570

Commit:37e4462
Author:yuxia Luo
Committer:GitHub

[lake/paimon] Promote existing Paimon log tables to Fluss lake tables (#4619)

Commit:c751131
Author:Junfan Zhang
Committer:GitHub

[rust] remove the duplicate proto definition (#4636) * [rust] remove the duplicate proto definition * [rust] Fail release on uncommitted proto changes --------- Co-authored-by: zhangjunfan <zhangjunfan@qiyi.com> Co-authored-by: Anton Borisov <anton.borisov@fresha.com>

Commit:dcb831c
Author:pisceslj
Committer:GitHub

[rpc] Add consumer offset protocol and error definitions (#4664)

Commit:71f0b99
Author:fhan
Committer:GitHub

[client] Add Admin API to describe buckets (#4029) * [client] Add Admin API to describe buckets * [client] fix test failure in flink2 module * [client] refine code impl according to review comments * [client] Complete describe buckets metadata semantics * [test] Avoid Mockito in ZooKeeper response handling tests

Commit:5e4659b
Author:Liebing
Committer:GitHub

[client] Support specifying rack when adding/removing server tags in Admin API (#2890) * [client] Support specifying rack when adding/removing server tags in Admin API * address yunhong's comments # Conflicts: # fluss-client/src/main/java/org/apache/fluss/client/admin/FlussAdmin.java # fluss-server/src/main/java/org/apache/fluss/server/coordinator/CoordinatorEventProcessor.java * fix rust client * fix format

Commit:e596d20
Author:Kaixuan Duan
Committer:GitHub

[server][client] Support per-partition bucket count for partitioned tables (#3908) Allow changing the default bucket count of a partitioned table for newly created partitions. Existing partitions retain their original bucket counts without requiring data redistribution. - Add a distribution-change API with versioned ALTER_TABLE RPC support. - Persist per-partition bucket counts and atomically backfill legacy partitions when advancing the table's bucket-count epoch. - Route requests using per-partition metadata and reject invalid bucket routing with INVALID_BUCKET_ROUTING, requiring metadata refresh and request reconstruction. - Adapt Java clients, Flink and Spark readers, and lake tiering to handle partitions with different bucket counts. - Propagate bucket-count changes to Paimon while preserving existing partition layouts. - Add rescaling and compatibility tests the new semantics. Closes #3907.

Commit:89a69ce
Author:yuxia Luo
Committer:GitHub

[client][server][paimon] Complete historical partition writes (#4120) Route writes for expired partitions through internal historical targets while preserving original partition metadata across PUT_KV and PRODUCE_LOG. Tier historical KV and log records back to their original Paimon partitions, fail writes to confirmed missing targets, and safely clean fully tiered historical KV overlays with leader-epoch and offset guards.

Commit:32e3668
Author:Yang Guo
Committer:Jark Wu

[server] Restore KV minRetainOffset on follower restart (#4060)

Commit:e01a94e
Author:Yang Guo
Committer:GitHub

[server][rpc] Return ISR and bucket epoch in bucket metadata (#3785)

Commit:9a915d9
Author:yuxia Luo
Committer:GitHub

[server] Support primary-key writes to historical partitions in server (#4001)

Commit:938f8ff
Author:seokjin0414
Committer:Anton Borisov

[rust] Check in generated protobuf code and drop protoc requirement

Commit:37ad1af
Author:Keith Lee
Committer:GitHub

[rust] Wire the scan filter through to the log fetcher (#3971) Adds `TableScan::filter`, which encodes the predicate against the full row type and sends it with the schema id on every fetch request. A response now advances the bucket offset past `filtered_end_offset`, whether the server emptied the fetch or only pruned its tail, and the proto comment claiming those two fields are mutually exclusive is corrected to match the server.

Commit:ca703ac
Author:Yang Guo
Committer:GitHub

[server] Support RemoteLogManifest to handle overlap RemoteLogSegments (#3822)

Commit:9973e6c
Author:yuxia Luo
Committer:GitHub

[lookup] Support historical partition lookup for lake tables (#3630)

Commit:21ac93f
Author:Yang Wang
Committer:GitHub

[server][client] Add cooperative KV backpressure based on RocksDB L0 (#3463)

Commit:5cacbb0
Author:Junfan Zhang
Committer:GitHub

[client] Add log read preference for remote-first fetch (#3606) * [client] Add log read preference for remote-first fetch * rust fix * add e2e * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * fallback to local * fix * not leader gate * wait util all tiered --------- Co-authored-by: zhangjunfan <zhangjunfan@qiyi.com> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Commit:400e132
Author:Anton Borisov
Committer:Anton Borisov

[FIP-40] Adapt imported fluss-rust to the monorepo (proto, CI, docs, release)

Commit:89eafb9
Author:Anton Borisov

[FIP-40] Import fluss-rust under fluss-rust/ with preserved history

Commit:e2a4403
Author:Liurnly
Committer:GitHub

[server] Support adding aggregation columns (#3459)

Commit:49849f7
Author:Pengcheng Huang
Committer:GitHub

[rust] Sync Fluss 1.x proto definitions and ApiKey registry (#628) - Update fluss_api.proto with all 1.x message types (ACLs, KV snapshots, producer offsets, cluster config, rebalance, server tags, etc.) - Add optional fields: rack, remote_data_dir, leader_epoch, agg_mode, etc. - Register 24 new ApiKey variants (1023-1064) in api_key.rs - Update build.rs prost bytes config for new proto fields - Add None defaults in convert.rs and partition.rs - Update pre-existing message wrappers that reference renamed proto fields / ApiKey variants so the crate still builds: * create_partition.rs: ignore_if_exists -> ignore_if_not_exists * get_latest_lake_snapshot.rs: ApiKey::GetLatestLakeSnapshot -> GetLakeSnapshot * list_databases.rs: populate new include_summary field * lookup.rs: PbLookupReqForBucket.key -> keys; new LookupRequest fields Co-authored-by: warmbupt <warmbupt@gmail.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

Commit:96a63d5
Author:Yang Wang
Committer:GitHub

[rpc][server][client] add ListRemoteLogManifests and ListKvSnapshots RPCs for orphan cleanup (#3480)

Commit:ad51d97
Author:yunhong
Committer:GitHub

[server] Add Cluster Health API implementation (#3400)

Commit:e274bd1
Author:Hongshun Wang
Committer:GitHub

[rust] Support API_VERSION in rust client. (#531) * [rust] Support API_VERSION in rust client * remove useless write_version_type and read_version_type which is same as java client does. * Add test about kv_format_v2. * return apiKey.highestSupportedVersion as default. * check whether server type mismatch. * add test about unknown server type

Commit:01c13e8
Author:yunhong
Committer:GitHub

[kv] Support elect standby replicas for primary key table (#2829)

Commit:01c9a22
Author:Anton Borisov
Committer:GitHub

[rust] lookup decoding schema id (#517)

Commit:7b4d711
Author:Anton Borisov
Committer:GitHub

feat: [rust] prefix lookup support (#500) * [rust] prefix lookup support * fix fmt * address comments * address comments * Address comments 2

Commit:c615e27
Author:Giannis Polyzos
Committer:GitHub

[common] Introduce RPC Layer for KV Scan (#2976)

Commit:3d8df9f
Author:Yang Wang
Committer:GitHub

[server] Add server-side filter execution and client-side integration (#2951)

Commit:8957671
Author:Anton Borisov
Committer:GitHub

feat: [TASK-315] Proto + RPC for LimitScan (#472)

Commit:05b6f66
Author:Liebing
Committer:GitHub

[server] Extend metadata models to support per-table/partition remote data directory (#2763)

Commit:14f0f12
Author:Liebing
Committer:GitHub

[server] Support alter database comment and custom properties (#1172)

Commit:ac21483
Author:Giannis Polyzos
Committer:GitHub

[common] Support Protobuf and RPC for Predicates (#2502)

Commit:ce67b9f
Author:Anton Borisov
Committer:GitHub

feat: introduce idempotent writes (#404)

Commit:f7fa971
Author:Junbo Wang
Committer:GitHub

[lake/tiering] Add per-table monitoring metrics for Lake Tiering (#2454)

Commit:1f4ff05
Author:Anton Borisov
Committer:GitHub

feat: support sasl/plain auth (#375)

Commit:d6fd1f1
Author:Jark Wu
Committer:Jark Wu

[kv] Support COUNT(*) pushdown for primary key tables and introduce GetTableStats RPC (#2651)

Commit:3015f6b
Author:Hongshun Wang
Committer:Jark Wu

[client] Thrown exception if ServerType mismatch. (#2606)

Commit:422d422
Author:Jark Wu
Committer:Jark Wu

[kv] Support COUNT(*) pushdown for primary key tables and introduce GetTableStats RPC (#2651)

Commit:fad97df
Author:Hongshun Wang
Committer:GitHub

[client] Thrown exception if ServerType mismatch. (#2606)

Commit:417c220
Author:yuxia Luo
Committer:Jark Wu

[paimon] Fix union read paimon dv table issue (#2326)

Commit:5a4374a
Author:yuxia Luo
Committer:GitHub

[paimon] Fix union read paimon dv table issue (#2326)

Commit:6e7fc26
Author:yunhong
Committer:Jark Wu

[kv] Support kv snapshot lease (#2179)

Commit:0cee383
Author:yunhong
Committer:GitHub

[kv] Support kv snapshot lease (#2179)

Commit:6655bdc
Author:Hongshun Wang
Committer:GitHub

[client] Support list database summaries (#2515)

Commit:bdb44c7
Author:yunhong
Committer:GitHub

[server] Fix rebalance may cause remote file deleted error (#2570) * [server] Fix rebalance may cause remote file deleted error * address jark's comments * improve * change to set offline immediately after set to immigration started --------- Co-authored-by: Jark Wu <jark@apache.org>

Commit:ec5078e
Author:Junbo Wang
Committer:Jark Wu

[kv] Implement lookup with insertIfNotExists on tablet server (#2485)

Commit:2671af6
Author:Yang Wang
Committer:GitHub

[kv] Return log end offset in UpsertResult and DeleteResult for exactly-once semantics (#2530)

Commit:1e84897
Author:Yang Wang
Committer:GitHub

[kv] Add OVERWRITE MergeMode support for undo recovery in aggregation tables (#2492)

Commit:81e3f3a
Author:Yang Wang
Committer:GitHub

[kv] Add Producer Offset Snapshot for Exactly-Once semantics (#2434)

Commit:62db319
Author:yuxia Luo
Committer:GitHub

[lake] Tiering service support commit by time (#2185)

Commit:d695cb8
Author:Keith Lee
Committer:GitHub

feat: Implement Admin APIs for partitioning (#208)

Commit:eda19e2
Author:Anton Borisov
Committer:GitHub

feat: introduce priority queue for downloading remote segments (#187)

Commit:f519ad9
Author:yuxia Luo
Committer:GitHub

feat: introduce KvWriteBatch and PutKvRequest (#176)

Commit:93048ed
Author:Andrea Bozzo
Committer:GitHub

feat: introduce lookup support for primary key tables (#159)

Commit:7addc72
Author:yunhong
Committer:GitHub

[server] Support generate and execute reblance plan (#1452)

Commit:a316c3d
Author:yuxia Luo
Committer:GitHub

[lake] Record a file path storing log offsets in lake snapshot property (#2223)

Commit:6868157
Author:yunhong
Committer:GitHub

[server] Introduce new rebalance API (#1380) Co-authored-by: Jark Wu <jark@apache.org>

Commit:e585d1f
Author:yuxia Luo
Committer:GitHub

chore: Improve error (#77)

Commit:746f948
Author:AlexZhao
Committer:GitHub

feat: support s3 as remote segment (#93) --------- Co-authored-by: luoyuxia <luoyuxia@alumni.sjtu.edu.cn>

Commit:f26312f
Author:yuxia Luo
Committer:GitHub

[lake] Use remote file to store lake snapshot offsets to reduce the size of zookeeper node (#2037)

Commit:72b10bb
Author:yunhong
Committer:GitHub

[common] Remove TableInfo from Cluster (#2068)

Commit:07721fe
Author:Jark Wu
Committer:Jark Wu

[server][client] Improve the design and performance of ADD COLUMN operation 1. Make lookup requests async. 2. Introduce ServerProjectionCache to share ProjectionInfo in TabletServer level. 3. Revert FlinkAsFlussRow changes, and introduce PaddingRow for the padding null columns for schema changes. 4. Make PutKv zero-copy to persist kv records. 5. Rename SchemaMetadataManager to ServerSchemaCache and use TableId instead of TablePath to track schemas which is safer. 6. Revert source related changes to still include the projectedFieldIndexes parameter.

Commit:ea2d61c
Author:Hongshun Wang
Committer:Jark Wu

[server][client] Support Schema Evolution (ADD COLUMN LAST) in Fluss and connector (#2010)

Commit:a2cc54d
Author:naivedogger
Committer:GitHub

[feat] Create Python bindings for table writing and reading (#9) --------- Co-authored-by: luoyuxia <luoyuxia@alumni.sjtu.edu.cn>

Commit:8c4004c
Author:Hongshun Wang
Committer:GitHub

[FIP-12] Support dynamic cluster config to enable lakehouse (#1567)

The documentation is generated from this commit.

Commit:e7d92ce
Author:Yang Guo
Committer:GitHub

[server] Rename RPC AlterTablePropertiesRequest to AlterTableRequest for future extensibility (#1766)

Commit:4345663
Author:Jark Wu
Committer:Jark Wu

[server] Use unified batch-concurrent implementation for ZooKeeper operations and remove redundant code This commit replaces all single-operation async calls with a unified batch-concurrent implementation. This eliminates the need to maintain three separate code paths (batch concurrent, single async, and sync), significantly reducing code duplication in ZooKeeperClient. Going forward, all sync ZooKeeper operations will be replaced by the batch-concurrent approach, which improves coordinator performance through better I/O utilization and reduced latency. Additionally, the `cacheOnly` flag has been removed from the current metadata request. Support for cache-only semantics will be reintroduced in a future `MetadataRequestV2` with a cleaner design.

Commit:32d50b8
Author:ocean.wy
Committer:Jark Wu

[server] server process metadata request in async mode (#1463)

Commit:68ce225
Author:Jark Wu
Committer:GitHub

[server] Improve the implementation of ALTER TABLE (#1760)

Commit:9df7b72
Author:Yang Guo
Committer:GitHub

[server] Support alter table properties (#1625)

Commit:79d9dbb
Author:yunhong
Committer:GitHub

[server] TabletServer support controlled shutdown (#1159)

Commit:ece2fe3
Author:Liebing
Committer:GitHub

[lake] Lake tiering service report the max timestamp of fluss record (#1700)

Commit:e6d877b
Author:Yang Wang
Committer:GitHub

[protogen] Fix infinite recursion in RecordsFieldFinder (#1282)

Commit:69b0ead
Author:naivedogger
Committer:GitHub

[feat] Add more functions to the Rust client Admin (#5)

Commit:f32db76
Author:Hongshun Wang
Committer:Hongshun Wang

[FIP-12] Add dynamic config and enable lakehouse dynamically.

Commit:f83b32c
Author:yuxia Luo
Committer:GitHub

[feat] Support basic write & read (#2)

Commit:bb0e12b
Author:Jark Wu
Committer:Jark Wu

[infra] Update project package name to org.apache.fluss Co-authored-by: benjobs <benjobs@apache.org>

Commit:db12388
Author:Jark Wu
Committer:Jark Wu

[infra] Update project package name to org.apache.fluss Co-authored-by: benjobs <benjobs@apache.org>

Commit:420194a
Author:Jark Wu
Committer:GitHub

[client] RemoteLogDownloader supports download log files in parallel and in priority (#1579)

Commit:dfe2176
Author:luoyuxia
Committer:Hongshun Wang

support enable lake dynamically

Commit:f82d103
Author:Hongshun Wang
Committer:Hongshun Wang

[FIP-12] add dynamic config to enable lakehouse.

Commit:d2dc98f
Author:yuxia Luo
Committer:GitHub

[lake] Store partition name in lake snapshot property (#1485)