Proto commits in pixelsdb/pixels

These commits are when the Protocol Buffers files have changed: (only the last 100 relevant commits are shown)

2026-06-15

[Issue #1327] implement retina recovery protocol (#1328)

The documentation is generated from this commit.

2026-04-06

[Issue #1288] implement file rewriting to enable garbage collection at the storage layer and fix some visibility bugs (#1310)

2026-02-27

[Issue #1293] implement visibility checkpoint cache on worker node (#1295)

2026-01-17

[Issue #1275] partition a single point index into multiple cf (#1276)

2025-12-24

[Issue #1257] support multiple WriteBuffers (#1258) Using multiple write buffers in one retina instance to improve data writing concurrency.

2025-12-13

[Issue #1245] fix watermark push (#1246)

2025-12-11

[Issue #1163] implement lease protocol for transaction (#1235) Transaction lease is only enabled for non-readonly transactions. The default lease expire time is 300 seconds, which means a write transaction will be rollbacked if it does not extend its lease by calling TransService::extendTransLease/extendTransLeaseBatch in time.

[Issue #1236] revise checkpoint-related RPC interfaces (#1239) In this PR, the query engine is responsible for checking and marking a long-running query, and sends RPC requests to transaction server and retina to push watermarks and checkpoint (offload) visibility structures.

2025-12-07

[Issue #1229] implement visibility checkpoint for long-running queries to unblock GC (#1232)

2025-12-06

[Issue #1133] add PixelsPollingService for polling cdc record (#1234)

2025-12-01

[Issue #1211] rename PixelsWriterBuffer to PixelsWriteBuffer (#1228)

2025-11-25

[Issue #1215] implement NodeService, Retina Node Heartbeat, and Consistent Hashing (#1216) Co-authored-by: Haoqiong Bian <bianhaoqiong@gmail.com>

2025-11-19

[Issue #1202] implemented periodic garbage collection of visibility (#1203)

2025-10-16

[Issue #1151] add retina update api and fix memory leak in processUpdateRequest (#1155)

2025-10-08

[Issue #1116] remove unused import in sink.proto (#1118)

2025-10-07

[Issue #1108] add support for querying Metadata by tableId in MetadataService (#1109)

2025-10-06

[Issue #1106] remove unnecessary proto messages to speed up serde (#1107)

2025-10-04

[Issue #1099] Increase DEFAULT_AI_STEP and simplify trans commit protocol (#1103) Increase DEFAULT_AI_STEP from 1000 to 10000 and remove timestamp from the trans commit and trans commit batch rpc methods.

2025-09-30

[Issue #1076] implement batch transaction commit functionality (#1077)

2025-09-29

[Issue #1055] handle Multiple Transactions in a Single Retina Update RPC Request (#1056)

2025-08-26

[Issue #983] modify index in retina service (#984)

[Issue #944] remove duplicate RowLocation definition in proto (#945)

[Issue #932] add file type to identify empty files (#936)

[Issue #930] add corresponding writer buffer when create new table (#931)

[Issue #926] implement retina insert related interfaces (#927)

[Issue #901] implement the writerBuffer (#919) Element addition is currently implemented; however, element deletion functionality has to wait for the index to be implemented.

[Issue #862] update retina interfaces (#863)

[Issue #860] fix visibility merge error (#861)

[Issue #999] change the empty file type to temporary (#1001)

[Issue #998] exclude temporary files when compacting (#1000) The creation of empty files: Retina's writer buffer needs to write data from memory to a file using PixelsWriter. When the writer is created, empty files are generated. The information about these files is stored in advance in the metadata and marked as EMPTY. After the file completes writing, it is marked as REGULAR. To avoid reading these temporary files during compaction, non-regular files must be excluded. These empty files may have written partial data, resulting in a non-zero length, so they cannot be directly identified by file length and instead require reading the metadata.

[Issue #964] remove unused dependency in sink.proto (#995)

[Issue #986] bidirectional streaming rpc for update records (#991)

[Issue #978] reconstruct retina and adjust interface (#979)

2025-08-13

[Issue #999] change the empty file type to temporary (#1001)

[Issue #998] exclude temporary files when compacting (#1000) The creation of empty files: Retina's writer buffer needs to write data from memory to a file using PixelsWriter. When the writer is created, empty files are generated. The information about these files is stored in advance in the metadata and marked as EMPTY. After the file completes writing, it is marked as REGULAR. To avoid reading these temporary files during compaction, non-regular files must be excluded. These empty files may have written partial data, resulting in a non-zero length, so they cannot be directly identified by file length and instead require reading the metadata.

2025-08-11

[Issue #964] remove unused dependency in sink.proto (#995)

[Issue #986] bidirectional streaming rpc for update records (#991)

2025-08-09

[Issue #983] modify index in retina service (#984)

2025-07-25

[Issue #978] reconstruct retina and adjust interface (#979)

[Issue #944] remove duplicate RowLocation definition in proto (#945)

[Issue #926] implement retina insert related interfaces (#927)

[Issue #932] add file type to identify empty files (#936)

[Issue #901] implement the writerBuffer (#919) Element addition is currently implemented; however, element deletion functionality has to wait for the index to be implemented.

[Issue #930] add corresponding writer buffer when create new table (#931)

[Issue #860] fix visibility merge error (#861)

[Issue #862] update retina interfaces (#863)

2025-07-23

[Issue #916] revise index service interface and add purge index (#981)

2025-07-22

[Issue #978] reconstruct retina and adjust interface (#979)

[Issue #916] implement open/close/remove/flush index in index service (#976)

2025-07-21

[Issue #916] revise the index framework and the main index implementation (#975) 1. Implement main index backed by SQLite, add a main index buffer. 2. Redesign the gRPC protocol of the cache service, add update index entry, open, close, and remove methods. 3. Change the MainIndexManager to a MainIndexFactory and a MainIndexProvider. 4. Revise the interfaces of single-point index, get scheme and unique from metadata. 5. Revise SinglePointIndexFactory, support managing index instances of different index IDs. 6. Remove some redundant data structure definitions.

2025-07-15

[Issue #916] update index service interface, index docs, and the rockset Jni code structure. (#969) We also revise DaemonMain for the index server startup.

2025-07-11

[Issue #944] remove duplicate RowLocation definition in proto (#945)

[Issue #932] add file type to identify empty files (#936)

[Issue #930] add corresponding writer buffer when create new table (#931)

[Issue #926] implement retina insert related interfaces (#927)

[Issue #901] implement the writerBuffer (#919) Element addition is currently implemented; however, element deletion functionality has to wait for the index to be implemented.

[Issue #862] update retina interfaces (#863)

[Issue #860] fix visibility merge error (#861)

2025-07-09

[Issue #939] update the retina-dev branch (#940) Co-authored-by: Haoqiong Bian <bianhaoqiong@gmail.com> Co-authored-by: whzruc <72006683+whzruc@users.noreply.github.com> Co-authored-by: vladaliii <2773158174@qq.com> Co-authored-by: slzo <85170982+slzo@users.noreply.github.com> Co-authored-by: Ubuntu <ubuntu@ip-172-31-31-111.us-east-2.compute.internal> Co-authored-by: AgentGuo <57658408+AgentGuo@users.noreply.github.com> Co-authored-by: Rolland1944 <145736819+Rolland1944@users.noreply.github.com> Co-authored-by: Ubuntu <ubuntu@ip-172-31-2-226.cn-north-1.compute.internal> Co-authored-by: Hua Siyuan <64294509+huasiy@users.noreply.github.com>

2025-07-08

[Issue #930] add corresponding writer buffer when create new table (#931)

2025-07-02

[Issue #926] implement retina insert related interfaces (#927)

2025-06-30

[Issue #923] simplify dependencies of pixels-invoker-spike and remove the duplicated apiserver.proto (#925)

2025-06-27

[Issue #901] implement the writerBuffer (#919) Element addition is currently implemented; however, element deletion functionality has to wait for the index to be implemented.

2025-06-26

[Issue #916] Redesign the main index implementation to support primary/secondary index and allocateRowIdBatch (#922)

[Issue #916] cleanup the poms for pixels-index and pixels-spike (#920)

2025-06-22

[Issue #916] update index service rpc interface (#917) Finish the first two tasks in Issue #916.

[Issue #913] rename secondary index to single point index and implement the related metadata RPCs (#915)

2025-06-17

[Issue #900] fix putRowId and deleteRowId interfaces of MainIndex and SecondaryIndex (#910) 1. Add putRowId and deleteRowId interfaces to insert and delete a single index in the MainIndex. 2. Put the MainIndex insert inside the SecondaryIndex insert to form an atomic operation. 3. Remove the sorting code from the MainIndex insert. 4. Change the return value of putRowId from boolean to rowid.

2025-06-15

[Issue #869] support pipline execution (#868) Currently, pipeline execution is only supported on vHive. Pipeline execution on pixels-spike still has network issues when pixels-work-spike tries to connect to the query coordinator in EC2.

2025-06-10

[Issue #906] update the retina-dev branch (#909) Co-authored-by: Haoqiong Bian <bianhaoqiong@gmail.com> Co-authored-by: whzruc <72006683+whzruc@users.noreply.github.com> Co-authored-by: vladaliii <2773158174@qq.com> Co-authored-by: slzo <85170982+slzo@users.noreply.github.com> Co-authored-by: Ubuntu <ubuntu@ip-172-31-31-111.us-east-2.compute.internal> Co-authored-by: AgentGuo <57658408+AgentGuo@users.noreply.github.com> Co-authored-by: Li Zinuo <antio2@qq.com> Co-authored-by: Rolland1944 <145736819+Rolland1944@users.noreply.github.com> Co-authored-by: Ubuntu <ubuntu@ip-172-31-2-226.cn-north-1.compute.internal>

2025-06-09

feat/use one state atomic variable to ensure consistency

[Issue #741] add visibility structure (#822) pixels-daemon: create retina rpc server. pixels-common: acts as a client and sends rpc requests to the retina server. pixels-retina: build and manage various data structures.

[Issue #741] add visibility structure for real-time CRUD (#847) Co-authored-by: slzo <85170982+slzo@users.noreply.github.com> Co-authored-by: Haoqiong Bian <bianhaoqiong@gmail.com>

[Issue #860] fix visibility merge error (#861)

[Issue #862] update retina interfaces (#863)

feat/create schema from column types

fix/use bytes type to represent column data instead

2025-06-07

[Issue #904] update Sink proto to align with Retina proto changes (#905)

2025-06-06

[Issue #897] support begin a batch of transactions in transaction service (#898) This allows the client to begin a batch of transactions with a single RPC request. --------- Co-authored-by: antio2 <antio2@qq.com>

2025-05-27

[Issue #893] Update sink and retina proto files (#894)

2025-05-20

[Issue #838] support index services backed by rocksdb and rockset (#885)

2025-05-16

[Issue #880] implement spike worker and invoker (#883)

2025-03-30

only one worker will pass schema to down stream

one worker write to one port of down stream worker

2025-03-17

[Issue #862] update retina interfaces (#863)

2025-03-14

[Issue #860] fix visibility merge error (#861)

2025-03-13

[Issue #741] add visibility structure for real-time CRUD (#847) Co-authored-by: slzo <85170982+slzo@users.noreply.github.com> Co-authored-by: Haoqiong Bian <bianhaoqiong@gmail.com>

[Issue #741] add visibility structure (#822) pixels-daemon: create retina rpc server. pixels-common: acts as a client and sends rpc requests to the retina server. pixels-retina: build and manage various data structures.

2025-03-06

[Issue #838] support lookup unique and non-unique indexes. (#850) Secondary indexes can be unique or non-unique.

2025-02-19

[Issue #838] implement the framework for secondary indexes (#844)

2025-02-07

[Issue #838] implement metadata service for secondary index (#839) Also add min/max row ids into file metadata objects and fix some minor problems in metadata.

2025-02-05

[Issue #658] revise range index definition (#837) Remove the index structure from the metadata table RANGE_INDEXES.

2025-01-03

[Issue #745] add ML-based auto-scaling policy (#818)

2024-12-02

[Issue #787] revise hidden column implementation and fix cf reading error (#788) 1. move row group level hidden column statistic into RowGroupStatistics. 2. revise the variable names related to the hidden column and transaction timestamp. 3. fix a buy at line 445 of PixelsRecordReaderImpl, using (this.timestamp != -1) to check whether hidden column should be read is incorrect.

2024-10-31

[Issue #82] clean old retina implementation (#766) We will use a new design for real-time updates.

2024-10-30

[Issue #737] implement hidden timestamp column (#754) Adding a hidden column into the pixels file and filtering out invisible tuples based on timestamp ordering. This is compatible with the previous file format without such a hidden column.

2024-10-25

[pixels-retine] clean proto files (#765)

2024-10-08

[Issue #645] add scaling server and auto-scaling policies (#747) Add a grpc service to collect the scaling metrics and a background thread to trigger auto-scaling according to scaling policies.

2024-09-17

[Issue #708] remove the file id generated by etcd (#709) The file ID is now generated in the metadata. The metadata schema should be updated using the SQL scripts provided in PR #709.

2024-09-08

[Issue #700] getting layout while getting table (#702) This is needed by `show create table` in the clients.