Proto commits in projectnessie/nessie

These 59 commits are when the Protocol Buffers files have changed:

2024-09-05

Add `referenced` attribute to persisted `Obj`s (#9401) This is a prerequisite to eventually be able to delete any stale objects in the backing database. The `referenced` attribute contains the timestamp when the object was last "referenced" (aka: attempted to be written). It is ... * set when an object is first persisted via a `storeObj()` * updated in the database, when an object was not persisted via `storeObj()` * set/updated via `upsertObj()` * updated via `updateConditional()` Let's assume that there is a mechanism to identify the IDs of all referenced objects (it would be very similar to what the export functionality does). The algorithm to purge unreferenced objects must never delete an object that is referenced at any point of time, and must consider the case that an object that was unreferenced when a purge-unreferenced-objects routine started, but became referenced while it is running. An approach could work as follows: 1. Memoize the current timestamp (minus some wall-clock drift adjustment). 2. Identify the IDs of all referenced objects. We could leverage a bloom filter, if the set of IDs is big. 3. Then scan all objects in the repository. Objects can be purged, if ... * the ID is not in the set (or bloom filter) generated in step 2 ... * _AND_ have a `referenced` timestamp less than the memoized timestamp. Any deletion in the backing database would follow the meaning of this pseudo SQL: `DELETE FROM objs WHERE obj_id = :objId AND referenced < :memoizedTimestamp`. Noting, that the `referenced` attribute is rather incorrect when retrieved from the objects cache (aka: during normal operations), which is not a problem, because that `referenced` attribute is irrelevant for production accesses. There are two edge cases / race conditions: * (for some backends): A `storeObj()` operation detected that the object already exists - then the purge routine deletes that object - and then the `storeObj()` tries to upddate the `referenced` attribute. The result is the loss of that object. This race condition can only occur, if the object existed but was not referenced. * While the referenced objects are being identified, create a new named reference (branch / tag) pointing to commit(s) that would be identified as unreferenced and being later purged.

The documentation is generated from this commit.

2024-07-10

Export related objects (#9034) Export and import objects related to the repository, the references, the commits and contents. The IDs of the related objects to be exported are retrieved via `TransferRelatedObjects` implementations, introduced by #9033. Those are exported and imported using the ability to (de)serialize "any" custom object, introduced by #9032. This change also bumps the export version of the "contents export" from 1 to 3 to handle the related objects, including the necessary export changes. --------- Co-authored-by: Dmitri Bourlatchkov <dmitri.bourlatchkov@gmail.com> Co-authored-by: Alexandre Dutra <adutra@users.noreply.github.com>

2024-05-22

UDF type: additional changes (#8560) Follow up of #8478, safe to change the type, because 8478 hasn't been released.

2024-05-08

Update `UDF` + `IcebergView`content types (#8478) * `UDF`: add metadata-location and version-ID * `IcebergView`: make dialect + schema optional The changes to the `UDF` type are non-controversial, becasue `UDF` isn't used anywhere yet. The changes to `IcebergView` allow clients to omit the dialect + schema fields _later_, i.e. after a few Nessie/Iceberg versions. "Later" due to backwards compatibility reasons, as long as Nessie/Iceberg versions are around that require these attributes.

2024-01-12

Persist: Conditional deletes + updates (#7932)

2024-01-04

Custom obj types should not persist class name (#7915) Custom object types do not need to and should not persist the fully qualified _implementation_ class for the custom object type, because it can be inferred from the type. This saves a bunch of storage but more importantly allows later refactorings of custom obj types. Since custom object types are not used yet, this change is safe. Not changing the column names for CQL + SQL in case users have already migrated their tables.

2023-12-04

Ensure that content IDs are unique in a Nessie repository (#7757) Nessie content IDs are random IDs, but we do not guarantee that those are actually really unique. This change adds a new object type to ensure that a generated ID is unique by leveraging existing functionality of the `Persist` framework that already provides "`INSERT IF NOT EXIST`" guarantees. New content IDs from this change on are now verified. This change does not include functionality to automatically add already existing content-IDs. IMHO it is probably okay for now given the practically non-existing probability of content-ID conflicts.

Remove database adapter code (#6890) Fixes #6701

Persist/custom objects: allow compression (#7795) Adds compression capability to custom object types. Uses snappy for serialized data larger than 8192 bytes. Snappy has a relatively good compression ratio, outperforms other compression algorithms during compression and is comparably quick during decompression. Also shortens the field names for DynamoDB + MongoDB.

2023-12-01

Extensible object types (#7771)

2023-10-13

Record previous HEADs of named references (#7607) Persisted `Reference`s now keep a history of previous HEADs, limited in size and age, defaults to up to 20 elementa with a maximum age of 300 seconds. Exposes a new Nessie-API V2 endpoint to retrieve the status of a reference. The returned information contains a consistency status of the HEAD commit, the recorded recent HEADs and, if requested, a combined status for the recent commits in the commit log. This change also ensures that a "broken" reference can be re-assigned to another existing commit, and that it can be deleted.

2023-06-22

Enhance references with created-timestamp + extended-info-obj-pointer (#7088) The intent of this change is to provide the base to fix an annoying race where users can failures for reference creations or deletions due to a potential race with concurrent reference queries, which trigger reference recovery, which can legitly finish a half-reference-creation/deletion, in these cases the original reference creation/deletion. This race is legit and intended, the errors should not happen though, and can be prevented by having some form of "token" - in this case the reference creation timestamp, which this change duplicates in the `refs` table. Since this change already changes all the references persistence, it also adds a new `extendedInfoObj` attribute to references. The intent here is to remove the (not yet) ambiguous pointer to "commt pointers" (extended tags via `TagObj`, not exposed as a functionality yet). The goal here is to only store the HEAD commit ID in `Reference.pointer()` and if a reference has more information (for example that `TagObj`), store the pointer to that one in `extendedInfoObj`. This also saves one additional load for `CommitObjReference` implementations (like `TagObj`) and to remove that interface.

2023-05-09

Add new content type UDF (#6761)

2023-03-28

Remove old content-attachments approach (#6422) Cleanup in preparation for metadata-in-Nessie. The old "content attachment" approach, only implemented by the old storage model, was never really used. This is basically the "whole spiel" to remove the content-attachment (de)serialization code from StoreWorker, where mostly just some parameters are removed, ending in a massive removal of code (and tests).

Expand iceberg-views version-ID to 64 bit (#6421) Should be safe to do this now, since Iceberg Views are not "in production" yet.

2023-03-23

Use relocated protobuf runtime (#6355) Generated protobuf code must be run with the protobuf runtime using the exact same version as the generator used. This is currently a soft constraint, but it is likely that protobuf will enforce that in future protobuf versions. Using a different version of the protobuf runtime can (and does) lead to issues at runtime, see [this issue for an example](https://github.com/protocolbuffers/protobuf/issues/11986). This change adds a new Nessie artifact `nessie-protobuf-relocated` which contains a relocated protobuf runtime that is used by all other Nessie artifacts. Also: * Move `*.proto` files into the directories corresponding to the Java package names. * Remove the `ProtobufHelperPlugin`, as the underlying issue (not exposing the generated java source folders in IDEs) has been fixed. * Do not apply checkstyle to Gradle projects that just generate protobuf code (the `ProtobufHelperPlugin` did that before). Side note: CEL-Java heavily depends on protobuf itself via its API and not just as a runtime/compile-only dependency. As long as no generated protobuf code is being used, it _should_ be okay to use a different protobuf version at runtime, but there is and will be no guarantee.

2023-03-07

New Nessie storage model (#5641) Implementation of the new Nessie storage model, driven by some inefficiencies in the current ("database adapter") storage model. * Storage space efficient index structures, with layered / striped indexes, supporting N * 100,000 keys * Flexible named references * Flexible store-keys to allow multiple "key universes" and "key variants" (latter: per "content key") * Simpler code

2022-12-05

Export: store exporting Nessie version and show during import (#5636)

2022-09-07

Nessie repository export + import (#5015) Nessie export/import functionality. Contains: * Ability to dump all commits and named references to either a zip file or a directory. * Ability to load commits and named references from either a zip file or directory. * Creates the loaded commits and named references via `DatabaseAdapter`. Loaded commits neither have the list of parent commits nor an aggregated key list. * Additional functionality to "optimize" all commits, means to populate the parent commits and to generate aggregated key lists and update the commits accordingly. (This is a separate class outside of the import functionality. * New `export` + `import` commands for `nessie-quarkus-cli` Notes: * `Content`s and `CommitMeta` are exported using the public format, i.e. no global state et al. * Imports write commits using their current way (i.e. global-state's implicitly converted to on-reference-state). However, for content that was persisted using global state, there will multiple on-reference-states referring to the same Iceberg table-metadata. * No database internal information, beside the commit-sequence-number and technical commit-create-timestamp, are exported. Fixes #4012

2022-08-18

Re-add `payload` to `StoreWorker` to identify the content-type (#4943) Previously we did not really enforce that the `payload` (a byte value corresponding to a particular `Content.Type`) was correctly persisted and as a workaround completely removed the handling of the `payload` value. This is rather inefficient, because the `StoreWorker` implementation has to deserialize the whole persisted value to identify the content type. Beside the above inefficiency having the right `payload` value is also a prerequisite to make content-types completely pluggable.

2022-08-11

Minor open-addressing key-list hashing changes (#4890) This commit makes a cluster of intertwined changes related to storage of keylists using open-addressing hashing: * Fixes lookups of keys which hash to the highest-indexed segment of an open-addressed key-list, but which physically reside in another segment due to collisions (overwhelmingly likely to reside in the embedded segment). Also adds a unit test and integration test exercising this case. The unit test is TestFetchValuesUsingOpenAddressing#segmentWrapAroundWithPresentKey, which fails on FetchValuesUsingOpenAddressing from main (expect to fix a method rename when mixing revs though: getKeyListSegmentCount becomes getKeyListBucketCount in this commit). * Attempts to make the word "segment" consistently refer to a collection of contiguous hashtable cells, corresponding to either the embedded keylist or a non-embedded keylist entity * Attempts to make the word "bucket" consistently refer to a single cell in the virtual keylist hashtable spread across all of the segments. This extends to persist.proto, where I renamed a field, but did not alter any tags, so it should remain wire-compatible. * Made FetchValuesUsingOpenAddressing#segmentForKey(bucket, round) stop clamping segment indices using the open addressing bucket index mask. Instead, it sums the segment index and round, then applies the remainder operator using the total segment count (embedded + not) as the divisor. * Added tests exercising some pathological edge cases. Although these elicited some failures against main, i.e. by checking out FetchValuesUsingOpenAddressing from main, the importance is debatable. Getting into these states generally requires major misconfiguration. One case involving segment index overrun exercised by the new segmentWrapAround test method might have come up, but the rest seem low-probability.

2022-07-13

Iceberg metadata in Nessie (#3907) Store `IcebergTable.metadata` and `IcebergView.metadata` as content-attachments in Nessie. Supports v1 + v2 table metadata.

2022-06-28

Content attachments in Nessie (#3293) Allow binary values related to content-ids via database-adapter. This is a prerequisite to have Icebert table and view metadata in Nessie. The content attachment keys is a composite of: * content-id * object-type (some string, e.g. `snapshot` or `schema`) * object-id (some identifier, e.g. a hash over the content value or some other identifier) Attachment values: * persisted with "format" _and_ "compression" attributes. * field "object ID", which represents e.g. the snapshot ID * "version" field for optimistic locking + CAS operations The new functionality via `DatabaseAdapter` allows: * unconditional bulk writes and reads, used to allow bulk writes + reads of for example the Iceberg snapshot + partition spec + schema for an Iceberg table. * conditional writes (think: "insert" + "update if") to store per-content-id information about a table - for Iceberg this would be for example the "last sequence number" or "last column id". * listing all attachment keys for a content id, which may return keys for non-existent attachments (compromise for non-transactional databases).

Hashed key-lists using open-addrressing (#4536) Refactors key-lists handling in `CommitLogEntry`. The `KeyListEntry`s in the key-list(s) in `CommitLogEntry` are currently organized by "least recently updated keys first", which degrades read performance with many keys per reference. This PR re-organizes the keys using an open-addressing hash-map, with the content's `Key` as the open-addressing hash-map's key. The whole open-addressing hash-map is split into multiple segments: the first segment is the "embedded" key-list inside `CommitLogEntry`, with additional segments as `KeyListEntity`s. This allows effective point-queries, both to only fetch the required `KeyListEntry` and within the `KeyListEntry`.

2022-06-07

Store and retrieve merge-parent (#4333) Merge operations store the merged commit ID as an additional parent. This change introduces a new field `additionalParents`, which is a list of commit-IDs. This list is usually empty, except for squashing merge commits, that are not single-commit fast-forward merges. Transplant operations do _not_ store an "additional parent". Merge operations that are configured to keep separate commits do _not_ store an "additional parent" as well, because this is effectively the same as transplanting (cherry-picking). This change _prepares_ the OpenAPI type `LogEntry` with a new attribute `additionalParents`. The attribute will _not_ be populated though to not break older clients. Otherwise, older clients will fail to deserialize commit log entries. This change however does return the first additional parent as a property in `CommitMeta`, with the property key `_merge_parent`, so clients that know about the meaning of that property, can handle it.

2022-05-25

Eliminiate global-pointer contention (non-tx database adapters) (#4234) * Moves named-references from global-pointer to separate entities (one "row" per named reference), concurrent updates to the same named ref can content, but concurrent updates to the different named refs do not (minus ref-log, see below). * Move ref-log HEAD from global-pointer to striped (mutliple) ref-log HEADs, using the name of the reference. This reduces the probability for contention to 1/16th (by default). Number of stripes can be increased (requires a restart of all nessie instances, no rolling upgrade). Both changes are backwards compatible: * Existing named references are migrated when Nessie finds existing named referenes in the global pointer. * Existing ref-log does not need to be migrated, ref-log reads just "include" the ref-log HEAD in global-pointer as an additional stripe. During a Nessie commit there is a (neglectible) chance that the commit is written and the named-reference correctly updated, but the corresponding ref-log-entry not written. This can happen when the Nessie server crashes or is stopped right a successful CAS of the named-reference HEAD but before the ref-log HEAD CAS. This is a trade-off. Upgrade procedure: 1. stop all existing Nessie instances 2. bump versions 3. restart.

2022-05-10

Unify identical enums in `persist.proto` (#4201) The enums `RefPointer.Type` and `RefLogEntry.RefType` are identical and can be unified.

2022-04-14

Support adding/updating/removing Namespace properties (#3931) This adds a new API endpoint to the existing `NamespaceApi` that supports adding/updating/removing Namespace properties. It also changes the existing `createNamespace(..)` API to allow creating a Namespace with properties. It also adds upgrade tests to make sure the changes are backward compatible

2022-04-12

Remove global state for Iceberg tables + views (#3866) Before this change, the location to the Iceberg table/view metadata was maintained globally, so all commits/references/branches/tags shared the same metadata. It was originally introduced to ensure that there can be no duplicate IDs for column-ID, schema-ID, et al. On the flip side, the global metadata "leaked" all table snapshots and schemas to all branches and even experimental branches could (potentially) exhaust all possible column IDs for a table. So this change moves the Iceberg metadata-pointer back to "per commit".

2022-04-11

Let StoreWorker pull global-state (#3908) `StoreWorker` implementations previously got an `Optional` containing the value of a content's global state. This change changes the behavior to pull the global state value. Also marks an attribute for delta-lake-tables as optional to properly handle the case coded in `TableCommitMetaStoreWorker`. `PersistVersionStore` checks the content-IDs of the new and expected values in a `Put` operation. Migrates to AssertJ for tests in `servers/store`. Prerequisite for #3866

Do not write empty global-log-entries (#3909) The current non-transactional database-adapters write empty global-log entries for changes that do not affect global state. Since it is not great to write empty global-log-entries, this change optimizes the non-transactional database-adapter implementation to no longer write "empty" global-log entries. The change there is to split the global-id, which was used to both point to the head of the global-log and to serve as the reference value for the global-pointer CAS operation. The head of the global log is now maintained as a separate field in global-pointer so that the global-id can still be used as the reference value for the CAS operation. Prerequisite for #3866

Rename internal packages, avoid package name shared across artifacts (#3906) Just package renames - no functional changes. (Re)using the same package name in different artifacts is not just only bad practice, it can likely cause issues later down the road, e.g. with JPMS, but also Quarkus warns about that.

2022-03-17

Fix Namespace encoding and only use elements for the canonical representation of a Namespace (#3632) Our namespace encoding was wrong because if a single element would contain a ".", then the elements within a Namespace weren't corrently translated and stored.

2022-03-16

Support Namespaces (#3572) * Support Namespaces This introduces a new `NamespaceApi` that allows creating/retrieving/deleting a single Namespace and also retrieving all Namespaces with a given prefix. A Namespace is now a subtype of `Content` to keep changes to a minimum at the storage level. Given that users currently don't explicitly create a Namespace when adding a new table 'a.b.c.table', the `NamespaceApi` is designed in a way where it first needs to evaluate: * whether an explicitly created Namespace exists: An explicitly created Namespace is created via `NamespaceApi#createNamespace(NamespaceParams)` * whether an implicitly created Namespace exists: An implicitly created namespace generally occurs when adding a table 'a.b.c.table' where the 'a.b.c' part represents the namespace without This feature also adds 3 new Exceptions: `NessieNamespaceAlreadyExistsException` / `NessieNamespaceNotEmptyException` / `NessieNamespaceNotFoundException`

2022-03-11

Record commit-id in key-lists (#3592) * Record commit-id in key-lists Fetching content values can become expensive as fetching the value from a `Put` opartion may require reading a lot of commit-log-entries, the more commit-log-entries the older the content. This change adds the commit-id of the last `Put` operation to the already existing entries in `KeyList`, which allows `DatabaseAdapter.fetchValues` to quickly "navigate" to the commits that contain the most recent `Put` operation that contains the content value for the requested content key. Time/resource complexity for existing keys is reduced from "number of commit log entries from HEAD to last Put operation" to at max two bulk-fetch operations against the commit-log. Updates & renames the type `KeyWithType` to `KeyListEntry`, adds the commit-id to `KeyListEntry`. This is not a breaking change and requires no data migration. Upgrade tests are included. Although not tested, and therefore not guaranteed, this change should also be safe to use during a rolling-upgrade. Fixes #3591 * update javadoc * Dynamically calculate commits-per-version

2022-03-08

Remove unused 'type' + "type-extractor" functions (#3556) The `type` field stored in the global-log was never really used and also always `0`. This change removes the `type` field from the immutable-types, renames it in the serialized representation, and removes the "type extractor" arguments from the respective functions in `DatabaseAdapter`. Note: the `type` field for "on reference" contents / content keys is correct and used.

2022-03-03

Allow IntelliJ to property generate proto sources (#3512) Splits out the .proto files and related code generation to separate modules without dependencies to not-yet-built artifacts. IntelliJ's "Generate Sources and Update Folders" action won't work, if the module for which sources should be generated (think: `process-sources` Maven goal) has not-yet available dependencies.

2022-03-01

Shave off two database reads reads per Nessie operation (#3463) To write a new global log entry, Nessie reads the parent hashes from the current global log entry. This can be avoided by storing the parent hashes in the `GlobalStatePointer` as well (nothing to do for transactional databases). To write a new ref log entry, Nessie reads the parent hashes from the current ref log entry. This can be avoided by storing the parent hashes in the `GlobalStatePointer` as well (transactional database: add the parent hashes to `TABLE_REF_LOG_HEAD`). The change can easily be implemented to be backwards compatible: * If the list or parent hashes is not present, read the parent hashes like the current implementation does. * If the "head" of the parent hashes is not equal to the expected "head", a concurrently running Nessie server updated the `GlobalStatePointer` (or `TABLE_REF_LOG_HEAD`) -> fall back to the old behavior and read the parent hashes like the current implementation does. Fixes #3448

2022-01-31

Replace SqlView with IcebergView (#3183) We don't really use SqlView and Iceberg View support is around the corner, thus we're switching to supporting Iceberg Views.

2022-01-21

Track Nessie database repo state/version (#3028) With the current work on implementing the ref-log it becomes kinda obvious that repository migration on the database level need to be handled. The idea in this change is to have a description about the repository that provides these attributes: * `repo_version` an integer version describing the repository, but not the database layout * `properties` as a generic key-value bag describing the repository configuration Behavior: * if no repository-description exists, assume version `0` and an empty properties Version `0` would be what we have now (including the ref-log). Fixes #3009

2022-01-20

Ensure stable order of references in GlobalStatePointer (#3111) As pointed out in the review of #3028, Java's `Map` does not ensure a stable order of elements. This was definitely a problem in that PR. Nessie's `GlobalStatePointer` also uses a `Map` for the named references. Although the design of the code ensures that the order of elements in `GlobalStatePointer.namedReferences` is stable, it feels safer to refactor that property to a list. The "order" of the list is: most recently updated named references appear at the beginning of that list. This change brings more advantages beside the "defined order or elements": * Most recently updated references appear before "stale" references. * Allows to "spill out" stale references to a separate table, if that becomes necessary, i.e. when there are too many references, the GSP becomes big and databases suffer from that size. * (Probably neglectible:) a bit less (de)serialization effort as the map on top of the protobuf list is gone.

2022-01-11

Reflog support in Nessie (#2950) Reflog backend entity support to get the complete details of what happened at the backend (similar to git reflog). Reflog support for both non-transactional db and transactional db. write support (protobuf serialized entries) for all operations (commit, create/delete/assign ref, merge, transplant). Reflog entry contents can be seen here: https://github.com/ajantha-bhat/nessie/blob/reflog/versioned/persist/serialize/src/main/proto/persist.proto#L81-L106 Added two tables for transactional store, One for storing the data and one for storing the reflog head. https://github.com/ajantha-bhat/nessie/blob/reflog/versioned/persist/tx/src/main/java/org/projectnessie/versioned/persist/tx/SqlStatements.java#L128-L164 read support with new "reflogs" REST resource and swagger API doc with pagination support startHash , endHash support Authz support

2021-11-29

Track total number of commits (#2767) * Track total number of commits Allows retrieval of the total number of commits on a reference. Via REST: in the `ReferenceMetadata`, if `fetchAdditionalInfo` is true. Via VersionStore/DatabaseAdapter: if `retrieveCommitMetaForHead` is true. The total number of commits is tracked as a field in every commit. The first commit since "beginning of time" starts at `1`, following commits "increment" this number. Since the commit history is immutable, the number of commits since "beginning of time" is immutable as well. * review * only add info if requested

2021-11-12

Move table-metadata-pointer back to global state, use snapshot,schema,spec,sort-order ids as on-ref state (#2626) This change basically reverts #2313. It moves the Iceberg able-metadata-pointer back to Nessie's global state and snapshot id plus other relevant ids (schema-id, partition-spec-id, sort-order-id) to Nessie's on-reference state.

2021-11-08

Rename ContentsKey to ContentKey and Contents to Content (#2585)

2021-10-19

Fix global and on-reference state (#2313) * Also track schemaId, specId, sortOrderId for Iceberg's on-reference-state The current way to track only the Iceberg snapshot-ID as the on-reference-state via `IcebergTable` is insufficient. There are several issues with it: 1. Schema changes in different branches are not properly tracked. 2. Iceberg schema-ID, partitionSpec-ID and sortOrder-ID are not properly tracked 3. The snapshot-log in `TableMetadata` is not maintained Although Iceberg's `Snapshot` class has a field `schemaId`, it is not enough to track schema changes on branches, because a schema change does _not_ produce a new snapshot in Iceberg. This means, that the snapshot on the Nessie reference still contains the schema-ID for that snapshot, but not the effective schema-ID after the DDL. For example: ```sql CREATE TABLE foo (col STRING); -- table is created in Nessie, snapshotId == -1, tableMetadata.schemaId == 0 INSERT INTO foo ('hello'); -- snapshot created, snapshotId == 42, tableMetadata.currentSchemaId == 0, snapshot.schemaId == 0 ALTER TABLE foo ADD COLUMN other STRING; -- no new snapshot, snapshotId == 42, tableMetadata.currentSchemaId == 1, snapshot.schemaId == 0 INSERT INTO foo ('bar', 'baz'); -- above statement fails, because tableMetadata.currentSchemaId is set to the snapshot's schemaId ``` Related issue is that `TableMetadata.snapshotLog` is validated to contain `TableMetadata.currentSnapshotId` as the last entry. This becomes a problem when the above example is "enhanced" w/ working on multiple branches: ```sql CREATE TABLE foo (col STRING); -- schemaId == 0 INSERT INTO foo ('hello'); -- snapshotId == 1 CREATE BRANCH branch_1; CREATE BRANCH branch_2; USE BRANCH branch_1; -- snapshotId == 1 ALTER TABLE foo ADD COLUMN added_1 STRING; -- schemaId == 1 INSERT INTO foo VALUES ('a', 'b'); -- snapshotId == 2 USE BRANCH branch_1; -- snapshotId == 1 ALTER TABLE foo ADD COLUMN added_2 STRING; -- fails in TableMetadata.removeSnapshotLogEntries(), because snapshotId == 1, but last one in TableMetadata.snapshotLog == 2 INSERT INTO foo VALUES ('c', 'd'); ``` This PR changes Nessie's "on-reference-state" to track the table-metadata pointer (in Nessie commits, as it was initially) but adds relevant information about Iceberg's ID-generators (last-column-ID, etc) into Nessie's "global-state" for Iceberg tables. Fixes #2312

2021-09-24

TVS2: remove old version-store implementations (#2112) * remove old `VersionStore` implementations for jgit, inmemory, dynamo, tiered * remove old Nessie GC implementation * remove temporary backwards-compatibility code * remove old Nessie GC functionality in `VersionStore`

2021-09-17

Prepare code for global-states (#1923) * Global-states: code changes (non-functional changes) * Global-State: Python changes * Global-State: Python changes (Cassettes + docs)

2021-08-30

TVS: do not persist Operation.Unchanged (#1861)

2021-08-20

Remove remainders of Hive (#1796) * Remove remainders of Hive * fix spotless

2021-08-17

TVS2: Serialization for database-adapters (#1765) Some `DatabaseAdapter` implementations, for example RocksDB, store certain entities in their serialized form. Introduces a new artifact `nessie-versioned-persist-serialize` that contains protobuf based serialization. Objects that are exposed via the `DatabaseAdapter` interface are still based on "immutables" and not protobuf.

2021-04-15

add an id to all contents types (#1023)

2021-03-10

GC support for iceberg (#803) * Better modularity in GC * remove StoreWorker and associated Asset Key stuff moving to gc * separate the gc into two stages: find referenced state of values and convert to unreferenced asset keys minor clean-up gc algo extended for Iceberg first round of refactoring first round of code review and rebased to new base package name build fix add another unit test for the asset key reader move gc packages around iceberg as its own package fix build Add timestamp, hashcode to objects and perform bloom filter on hashcode rather than byte[] address code review and clean-up switch to Spark Action for expire as thats the common way to expire snapshots * post split update to iceberg * code review comments * code review updates * fix dependency * run local dynamo before nessie * fix dynamo exec

2021-03-04

New module for TableCommitMetaStoreWorker (#895) Create a new `nessie-server-store` module for placing TableCommitMetaStoreWorker and related classes so that the StoreWorker implementation can be shared with other server implementation.

2021-02-12

Rename com.dremio.nessie package to org.projectnessie (#810) To better reflect the name of the project, rename classes from `com.dremio.nessie` to `org.projectnessie` (same number of characters).

2020-10-01

Spark delta client (#237) * Delta Lake Spark 2/3 * support maintaining multiple branches in Delta

2020-09-19

Update Rest API (#147) * Update rest api * Fix for in memory version store and first full test working. * building * more building * more * remove extra xml * Fix last unit tests. * Name cleanups * Checkstyle and more names cleanup * make test resilient to changed ordering * More cleanups and removing duplication for iceberg tests. * More cleanups * Exception cleanups and new rest api interfaces (impl/rework coming) * Pull new api through and fix ambiguous urls. * Fixes and cleanup

2020-09-07

Remove branch controller and refactor JGitVersionStore (#95) * Remove branch controller and refactor JGitVersionStore * remove entire (now unused) jgit module * refactor and improve JGitVersionStore including simpler commit model * increase test coverage * minor formatting and test changes * remove metadata from protobuf utils * improve coverage * rebase and address comments from #88

2020-06-24

DX-22538: Refactor to git-like endpoints * rename to nessie * use jgit * refactor endpoint and catalg Change-Id: I53dc2be8aca8591f1fd44f726a3fd8794aef30e1