Proto commits in polarsignals/frostdb

These 27 commits are when the Protocol Buffers files have changed:

2024-05-17

db: ignore writes below new block transaction on recovery (#864) * db: ignore writes below new block transaction on recovery Previously, the recovery code was ignoring writes below the persist transaction of a table block. However, this could lead to dropped writes since the active block is swapped atomically on rotation *before* the old block is persisted. Writes in between these two events would be written to the new table block, but ignored on recovery given the recovery code assumed they were in the old table block. * db: fix WAL off-by-one truncation error Previously snapshots were performed using a write txn and the WAL was truncated so that that txn would be the first txn in the truncated WAL. However, snapshots were changed to use a read txn so if a write at txn k was included in the snapshot, a truncated WAL would still contain this write as the first write in the WAL, resulting in duplicate data after recovery.

The documentation is generated from this commit.

2024-04-04

Add proto CI runs (#807) * Add proto CI runs Mostly copied from Parca. * Add comments to proto messages where buf lint complains * buf generate

Add FrostDB gRPC Query service (#784) Add FrostDB gRPC Query service to query with proto

2024-01-10

Add support for uint64 columns (#618) * Add support for uint64 columns * Add query support for Uint64 The commit adds support for Uint64 in binaryscalarexpr.go and various places. Most of these involve additional case statements to handle inputs of Uint64 type.

2024-01-07

buf format -w (#673) * buf format -w I was working on protobuf files and notices my vscode was formatting the files differently. We already use buf for generating, it is reasonable to use buf for formatting as well. Strictly for consistency. * add workflow for buf format

2024-01-02

Add support for int32 columns (#647) Signed-off-by: Nicolas Takashi <nicolas.tcs@hotmail.com>

2023-12-04

use reserved keyword for deprecated txnmetadata proto (#612) * use reserved keyword for deprecated txnmetadata proto * lint

Deprecate txn metadta (#609) * proto: deprecate: txn metadata * proto: removed txn metadata from snapshots This feature is not longer used, and it didn't ever quite work with snapshots as intended.

2023-10-06

dynparquet: add unique primary index option (#553) This commit adds the option to specify that the primary index should be unique. This results in dropping duplicate rows during compaction. Tables with this option might experience slower compactions.

2023-09-14

Pre hash columns (#524) * proto/schema: add prehash column option * Prehash columns based on schema

2023-08-08

*: allow attaching user-defined metadata to txns (#483) * *: add user-defined TxnMetadata to wal and snapshot protos * *: add WithUserDefinedTxnMetadatProvider option This allows users to associate txns with user-defined metadata. These are persisted in WAL records and snapshots. * *: store txn metadata in db high watermark * *: expose high watermark * *: fix TestDBRecover flake

2023-05-02

Repeated schema (#435) * proto: repeated field in storage layout * support repeated storage layouts

2023-03-10

381 wal persist table config (#383) * proto: TableConfig primitive * support new table config def

2023-03-09

frostdb: miscellaneous snapshot improvements (#374) * frostdb: snapshot quality of life improvements This commit: - Adds Info/Debug log messages to snapshots - Adds an atomic that ensures only one snapshot is taken at a time. - Moves the async snapshot code into a method. - Writes WAL record before taking the snapshot. - Adds metrics. * frostdb: add active TableBlock information to snapshot This is necessary since the block size and last snapshot size was not set. The txns are also serialized (min/prev) since some stuff will probably break if not. * design: add snapshot design document

2023-03-08

frostdb: enable snapshots (#371) * frostdb: fix txpool race on shutdown Close would close a channel that is written to from another goroutine, which is a race condition since a write could panic. This commit changes the txpool to use a context with a cancel function to shut down instead. * frostdb: enable snapshots This commit adds the option to enable snapshotting a database whenever a block exceeds a given size or is rotated out. A snapshot is taken of the db state and stored in the snapshots/ folder. The use of the WAL is not modified yet, even though it could be truncated. This will be done in a future change once snapshots have been running for a while. Note that old snapshots are not cleaned up yet, which gives us more txns to reload at in case of issues. In the future, a cleanup policy will be implemented.

2023-03-06

*: add snapshots (#357) This commit adds the ability to write a database snapshot at a given tx to disk and load a snapshot into a database. This commit does not yet add a public-facing API for snapshots. The motivation for this change is to improve use cases where users enable the WAL. Snapshots aim to solve two problems: 1) WAL replay can be very slow given that each insert needs to be replayed to get to a "final" state. Snapshots can be loaded into a database in O(1) time with respect to the number of inserts performed against the database instead of the O(n) inserts required during a WAL replay. 2) With multiple tables, the WAL can only be truncated when all tables have been persisted/rotated out and only at the minimum persisted tx across all tables. If the user has a small table that doesn't change often and is therefore not persisted/rotated out, the WAL can never be truncated. Snapshots allow us to persist these small tables and truncate the WAL at the snapshot txn.

2023-02-06

Arrow wal (#320) * proto: add arrow field to wal entry * log arrow records to wal * wal: support arrow record writes * replay arrow wal entries

2022-12-19

wal: fix wal.proto backwards incompatibility (#289) The new schema changes overrode the proto field, which caused the old schema message to be incorrectly unmarshalled. This commit adds the field back in as a deprecated oneof so that we can handle WALs generated before the change.

2022-12-08

Schema v2 nested (#273) * proto: added v1alphav2 schema to support nested schemas WAL: changed the schema to a generic pb.Any * wal: support pb.Any

2022-12-07

collapse v2 schema

2022-12-06

allow groups to be nullable/repeated

v1alpha2: nested schema

2022-12-05

proto: add repeated field to storage layout

2022-11-29

Support boolean types (#256) * support boolean types * logictest: boolean logic test

2022-07-31

Add support for Delta Byte Array and Delta Length Byte Array encoding (#144)

2022-07-25

Add parquet-reencode tool to switch encodings and compressions (#136) This tool is useful in order to try and identify whether a different encoding or compression might be useful for a certain column. It allows for a quick cycle to experiment without having to change the ingesting application and wait for data to accumulate.

2022-07-21

Add write-ahead log