Proto commits in man-group/ArcticDB

These commits are when the Protocol Buffers files have changed: (only the last 100 relevant commits are shown)

2026-07-07

Enhancement 9929831600: Arrow timezone support

2026-07-06

Enhancement 9929831600: Arrow timezone support

Arrow norm meta RFC

2026-07-02

WIP tidy up the implementation

Persist timezones in norm metadata

Add NaN and NaT counter to column stats Adds a null counter to column statistics, counting NaN/NaT values, and uses it during query pruning: - Count nulls for null-only column-stats segments. - Prune only-null column-stats slices for == / isin, but keep wholly-null row-slices for != / isnotin instead of pruning them. - C++ and Python regression tests, including Arrow coverage.

2026-06-30

Add NaN and NaT counter to column stats Adds NaN/NaT/null counting to min/max column stats, counted inline with minmax computation, and fixes column stats pruning silently dropping NaN/NaT/null rows. Includes tests covering null and nan counts across sparse floats and multiple segments.

2026-06-26

Add NaN and NaT counter to column stats Adds NaN/NaT/null counting to min/max column stats, counted inline with minmax computation, and fixes column stats pruning silently dropping NaN/NaT/null rows. Includes tests covering null and nan counts across sparse floats and multiple segments.

2026-05-29

change nat_count to null_count

2026-05-28

add nan and nat count to min and max stats

2026-05-05

Polars write support and change index_column to bool (#3037) #### Reference Issues/PRs Monday ref: 18362798253 #### What does this implement or fix? Two main changes: #### 1. `index_column` change from `str` to `bool` Changes the `index_column` argument from `str` to `bool` meaning we now only accept the first column to be the timeseries index. This is NOT a breaking change because it only affects api strictly labeled as experimental. This is done for two reaons: - Simplifies drastically denormalization logic. We don't have to worry about where to put the index column after schema changes. - It is future proof. We can re-enable the custom index column position in the future without being a braking change. The `index_column` arg could just start accepting strings alongside bools. We would not be able to change our minds in the opisite direction. Change involves: - Modification of experimental protobuf. This is ok as arrow writes are experimental - Removal of all column reordering logic - Updates tests and docs #### 2. Accept `polars.DataFrame`s to write like methods if arrow input is enabled Make `ArrowTableNormalizer` work with `polars.DataFrame`s. This allows all write methods to work with polars input. Currently this works by converting `polars.DataFrame` to `pyarrow.Table` and works the same as pyarrow tables going forward. In the future this will be better done via the `PyCapsule` polars interface, when sparrow fully supports it. Change invloves: - Normalizer accepting polars dataframes - update with date_range truncation of polars dataframes via pyarrow (was benchmarked faster than polars because pyarrow can work with views) - Parametrizing a subset of tests to work with polars (done in a separate commit for easier review) #### Note to reviewers The polars test parametrization is done in a separate commit for easier review. #### Checklist <details> <summary> Checklist for code changes... </summary> - [ ] Have you updated the relevant docstrings, documentation and copyright notice? - [ ] Is this contribution tested against [all ArcticDB's features](../docs/mkdocs/docs/technical/contributing.md)? - [ ] Do all exceptions introduced raise appropriate [error messages](https://docs.arcticdb.io/error_messages/)? - [ ] Are API changes highlighted in the PR description? - [ ] Is the PR labelled as enhancement or bug so it appears in autogenerated release notes? </details> <!-- Thanks for contributing a Pull Request to ArcticDB! Please ensure you have taken a look at: - ArcticDB's Code of Conduct: https://github.com/man-group/ArcticDB/blob/master/CODE_OF_CONDUCT.md - ArcticDB's Contribution Licensing: https://github.com/man-group/ArcticDB/blob/master/docs/mkdocs/docs/technical/contributing.md#contribution-licensing --> --------- Co-authored-by: Ivo <ivo.dilov@man.com>

The documentation is generated from this commit.

2026-04-17

Encode column stats metadata in the segment header rather than column names, and support multi-index (#2990) Monday: 11292562756 11292649800 We believe that no one has created column stats yet as there is no benefit to users. #2958 will start to support using them at read time. We want to make sure that we get the column stats format right on disk before we announce column stats to users and they start to serialize them. This PR is making changes to how we save column stats on disk. It also allows us to create column stats over multi-indexed dataframes, which was not possible before. Global search in Man to check no one ever used `create_column_stats`: https://chat-man.slack.com/archives/CKQBVA96D/p1774986019195379 Column stats information is currently written in `KeyType::COLUMN_STATS` with column names like, `v1.0_min(col_three)`. This is not ideal because at read time we need to parse these string column names to understand what each statistic means. This PR adds a structure in `descriptors.proto`: ``` enum ColumnStatsType { // Older clients reading a new enum value written by a newer client will decode to this first // element, so make sure they don't mis-interpret it as a statistic they understand. // https://protobuf.dev/best-practices/dos-donts/#unspecified-enum COLUMN_STATS_UNKNOWN = 0; // The version numbers here refer to the format of a given statistic. For example we might start // off saving string min and max truncated to 8 bytes in a uint64_t, and later change to saving it truncated // to a different length. That would necessitate a COLUMN_STATS_MIN_V2 so that old readers do not // misinterpret the new statistics format. COLUMN_STATS_MIN_V1 = 1; COLUMN_STATS_MAX_V1 = 2; } message StatEntry { uint32 stats_seg_offset = 1; // offset in to fields in the KeyType::COLUMN_STATS's StreamDescriptor ColumnStatsType type = 2; } message StatEntryList { repeated StatEntry entries = 1; } // Stored in the user defined metadata for KeyType::COLUMN_STATS message ColumnStatsHeader { // This version number refers to the format of this header structure. // Increment the version number if the meaning of the fields changes, to help older clients // avoid mis-interpreting data written by newer clients. uint32 version = 1; // key = data_col_offset (offset in to TimeseriesDescriptor#fields_ in the Index key) map<uint32, StatEntryList> stats_by_column = 2; // end of fields in version 1 } ``` that we save in `KeyType::COLUMN_STATS` header, in the user defined metadata field. This lets us understand the contents of column stats keys without needing to parse a particular string format. We keep meaningful strings as the names of the column stats segment columns for debugging. ### Compat Testing [Manual testing of what happens if this wheel sees unknown stats](https://github.com/man-group/ArcticDB/blob/aseaton-stats-compat-testing/compat_testing_notes.md) --------- Co-authored-by: Alex Seaton <alex.seaton@man.com>

2026-04-01

Changes for manual compat testing

New encoding for column stats, using protos to encode column stats information in the segment header's user metadata field

2026-03-18

Move COLUMN_STATS metadata into segment header Store version info and stat-to-column mappings in a protobuf ColumnStatsHeader packed into the segment metadata, rather than encoding them in column names. Column names simplify from "v1.0_MIN(col)" to "MIN(col)". No backwards compatibility needed as there are no existing COLUMN_STATS keys in use.

2026-01-14

Update copyright notices to 2026 (#2848) #### What does this implement or fix? Update all years in copyright notices to 2026. Done mechanically by running: `find . -type f -exec sed -i 's/Copyright 2023 Man Group Operations Limited/Copyright 2026 Man Group Operations Limited/' {} \;` for 2023/4/5 in `cpp/arcticdb`, `cpp/proto` and `python` directories.

2025-10-07

Enhancement 8987170104: arrow write numeric dense data (#2663) #### Reference Issues/PRs [8987170104](https://man312219.monday.com/boards/7852509418/pulses/8987170104) #### What does this implement or fix? Allows writing of `pyarrow.Table` objects provided they contain only dense numeric data. This is by no means the final API and so must currently be opted into by calling `_set_allow_arrow_input()` on `NativeVersionStore`. Everything is zero copy as far as possible, with the unavoidable exception of bool columns, as our internal representation uses 1 byte per value, whereas Arrow uses 1 bit per value. Performance is pretty much identical to Pandas in comparable use cases, with the exception of bool columns for the reason stated above, and wide, shallow dataframes, which are a bit faster to write with Arrow (presumably because of normalization stuff happening in the Python layer with Pandas that we can skip with Arrow). #### Any other comments? In addition to not supporting strings or sparse data, there are still a lot of improvements to be made: - `update` with a `date_range` overlapping the provided table index not currently supported - Converting from Arrow's packed bitset bool column representation to our internal format is not parallelised - Non-nanosecond timestamp columns can be written, but are returned as nanosecond timestamp columns - Non-UTC timestamp columns also not handled correctly - The index column (if specified) is only guaranteed to be at the correct index when reading all data, not when selecting specific columns or performing processing - Only `pyarrow.Table` is supported right now, not lower level primitives such as `RecordBatch/Array/ChunkedArray` - On read, processing pipeline operations on data written as Arrow probably won't work - `get_info/get_description` output will not be accurate for data written as Arrow - No testing of appending/updating Arrow data with Pandas data and vice-versa, so probably doesn't work - Index column must always be specified for timeseries data, even when it could be inferred in `append/update` calls

2025-10-06

Enhancement 8987170104: Write Arrow Tables consisting only of dense numeric data

2025-10-03

Address Ivo's comments

Enhancement 8987170104: Write Arrow Tables consisting only of dense numeric data

2025-10-02

Enhancement 8987170104: Write Arrow Tables consisting only of dense numeric data

2025-09-29

Enhancement 8987170104: Write Arrow Tables consisting only of dense numeric data

2025-09-24

Enhancement 8987170104: Write Arrow Tables consisting only of dense numeric data

2025-09-23

Enhancement 8987170104: Write Arrow Tables consisting only of dense numeric data

2025-09-05

Specifiying index column wip

2025-08-08

Storage lock increase wait time and add artificial slow writes (#2497) #### Reference Issues/PRs <!--Example: Fixes #1234. See also #3456.--> #### What does this implement or fix? In continuation of https://github.com/man-group/ArcticDB/pull/2359 which was reverted. This is the same but without the check for slow writes, as that was causing the lock to never be taken on inherently slow storages. This is the diff with the reverted PR https://github.com/man-group/ArcticDB/compare/symbol_list_slow_writes_tests_before_revert..symbol_list_slow_writes_tests_after_revert Changelog is similar to #2359: * Storage lock wait increased to 1000ms * ~Locking gives up on slow writes~ * StorageFailureSimulator extended to simulate slow writes * Stress tests (some are removed from #2359) * Refactoring the storage lock share common code #### Any other comments? #### Checklist <details> <summary> Checklist for code changes... </summary> - [ ] Have you updated the relevant docstrings, documentation and copyright notice? - [ ] Is this contribution tested against [all ArcticDB's features](../docs/mkdocs/docs/technical/contributing.md)? - [ ] Do all exceptions introduced raise appropriate [error messages](https://docs.arcticdb.io/error_messages/)? - [ ] Are API changes highlighted in the PR description? - [ ] Is the PR labelled as enhancement or bug so it appears in autogenerated release notes? </details> <!-- Thanks for contributing a Pull Request to ArcticDB! Please ensure you have taken a look at: - ArcticDB's Code of Conduct: https://github.com/man-group/ArcticDB/blob/master/CODE_OF_CONDUCT.md - ArcticDB's Contribution Licensing: https://github.com/man-group/ArcticDB/blob/master/docs/mkdocs/docs/technical/contributing.md#contribution-licensing -->

2025-08-07

Symbol list and storage lock improvements (#2359) #### Reference Issues/PRs <!--Example: Fixes #1234. See also #3456.--> #### What does this implement or fix? * Storage lock wait increased to 1000ms * Locking gives up on slow writes * StorageFailureSimulator extended to simulate slow writes * Stress tests #### Any other comments? https://github.com/man-group/ArcticDB/actions/runs/16205330703?pr=2359 proof of all tests passing (except macOS ones which seem problematic) #### Checklist <details> <summary> Checklist for code changes... </summary> - [ ] Have you updated the relevant docstrings, documentation and copyright notice? - [ ] Is this contribution tested against [all ArcticDB's features](../docs/mkdocs/docs/technical/contributing.md)? - [ ] Do all exceptions introduced raise appropriate [error messages](https://docs.arcticdb.io/error_messages/)? - [ ] Are API changes highlighted in the PR description? - [ ] Is the PR labelled as enhancement or bug so it appears in autogenerated release notes? </details> <!-- Thanks for contributing a Pull Request to ArcticDB! Please ensure you have taken a look at: - ArcticDB's Code of Conduct: https://github.com/man-group/ArcticDB/blob/master/CODE_OF_CONDUCT.md - ArcticDB's Contribution Licensing: https://github.com/man-group/ArcticDB/blob/master/docs/mkdocs/docs/technical/contributing.md#contribution-licensing -->

Merge branch 'master' of github.com:man-group/ArcticDB into symbol_list_slow_writes_tests

Merge branch 'master' of github.com:man-group/ArcticDB into symbol_list_slow_writes_tests_after_revert

2025-08-06

Arrow output normalization (#2488) Makes sure arrow output is consistent with pandas normalization metadata. Change includes: - properly constructing timestamp type in C++ layer - Introduction of ArrowTableNormalizer - Handles index column timezones - Handles index column renames - Constructs `pyarrow.Schema.pandas_metadata` which `pyarrow` can then use to reconstruct the pandas dataframe. - Arrow normalization tests - Simplifies testing logic for existing tests by allowing to reuse `assert_frame_equal` - Pyarrow requires [extra installation of timezone data](https://arrow.apache.org/docs/cpp/build_system.html#download-timezone-database) on Windows. Adds github action step to download it. #### Reference Issues/PRs Monday ref: 8987164697 #### What does this implement or fix? #### Any other comments? #### Checklist <details> <summary> Checklist for code changes... </summary> - [ ] Have you updated the relevant docstrings, documentation and copyright notice? - [ ] Is this contribution tested against [all ArcticDB's features](../docs/mkdocs/docs/technical/contributing.md)? - [ ] Do all exceptions introduced raise appropriate [error messages](https://docs.arcticdb.io/error_messages/)? - [ ] Are API changes highlighted in the PR description? - [ ] Is the PR labelled as enhancement or bug so it appears in autogenerated release notes? </details> <!-- Thanks for contributing a Pull Request to ArcticDB! Please ensure you have taken a look at: - ArcticDB's Code of Conduct: https://github.com/man-group/ArcticDB/blob/master/CODE_OF_CONDUCT.md - ArcticDB's Contribution Licensing: https://github.com/man-group/ArcticDB/blob/master/docs/mkdocs/docs/technical/contributing.md#contribution-licensing -->

2025-08-01

Arrow output normalization Makes sure arrow output is consistent with pandas normalization metadata. Change includes: - properly constructing timestamp type in C++ layer - Introduction of ArrowTableNormalizer - Handles index column timezones - Handles index column renames - Constructs `pyarrow.Schema.pandas_metadata` which `pyarrow` can then use to reconstruct the pandas dataframe. - Arrow normalization tests - Simplifies testing logic for existing tests by allowing to reuse `assert_frame_equal`

2025-07-25

Arrow output normalization Makes sure arrow output is consistent with pandas normalization metadata. Change includes: - properly constructing timestamp type in C++ layer - Introduction of ArrowTableNormalizer - Handles index column timezones - Handles index column renames - Constructs `pyarrow.Schema.pandas_metadata` which `pyarrow` can then use to reconstruct the pandas dataframe. - Arrow normalization tests - Simplifies testing logic for existing tests by allowing to reuse `assert_frame_equal`

2025-07-23

Arrow output normalization Makes sure arrow output is consistent with pandas normalization metadata. Change includes: - properly constructing timestamp type in C++ layer - Introduction of ArrowTableNormalizer - Handles index column timezones - Handles index column renames - Constructs `pyarrow.Schema.pandas_metadata` which `pyarrow` can then use to reconstruct the pandas dataframe. - Arrow normalization tests - Simplifies testing logic for existing tests by allowing to reuse `assert_frame_equal`

Read arrow frontend

2025-07-15

Revert "Symbol list and storage lock improvements (#2359)" (#2467) This reverts commit 7a5f09b75adfd8ff4d34f8c6501db7d5bd25e35b (merge of #2359) due to issues seen in real S3 storage tests.

Revert "Symbol list and storage lock improvements (#2359)" This reverts commit 7a5f09b75adfd8ff4d34f8c6501db7d5bd25e35b.

2025-07-11

Read arrow frontend

Symbol list and storage lock improvements (#2359) #### Reference Issues/PRs <!--Example: Fixes #1234. See also #3456.--> #### What does this implement or fix? * Storage lock wait increased to 1000ms * Locking gives up on slow writes * StorageFailureSimulator extended to simulate slow writes * Stress tests #### Any other comments? https://github.com/man-group/ArcticDB/actions/runs/16205330703?pr=2359 proof of all tests passing (except macOS ones which seem problematic) #### Checklist <details> <summary> Checklist for code changes... </summary> - [ ] Have you updated the relevant docstrings, documentation and copyright notice? - [ ] Is this contribution tested against [all ArcticDB's features](../docs/mkdocs/docs/technical/contributing.md)? - [ ] Do all exceptions introduced raise appropriate [error messages](https://docs.arcticdb.io/error_messages/)? - [ ] Are API changes highlighted in the PR description? - [ ] Is the PR labelled as enhancement or bug so it appears in autogenerated release notes? </details> <!-- Thanks for contributing a Pull Request to ArcticDB! Please ensure you have taken a look at: - ArcticDB's Code of Conduct: https://github.com/man-group/ArcticDB/blob/master/CODE_OF_CONDUCT.md - ArcticDB's Contribution Licensing: https://github.com/man-group/ArcticDB/blob/master/docs/mkdocs/docs/technical/contributing.md#contribution-licensing -->

2025-06-03

Temp, dont merge

2025-06-02

Add some lightweight encodings

2025-05-22

Add some lightweight encodings

2025-05-12

Temp, dont merge

Add some lightweight encodings

2025-05-09

Temp, dont merge

2025-05-01

Add some lightweight encodings

2025-03-26

Add some lightweight encodings

2025-02-20

GCP Support over the S3 XML API (#2176) This adds support for GCP over the S3-compatible XML API. GCP support did not work out of the box because Google does not support any of the S3 "batch" APIs. We used batch delete for all our deletion operations, which broke. Instead I use the unary delete operation here, using the AWS SDK's async API to reduce the performance impact of sending single object deletes. Users connect to GCP like: ``` ac = Arctic("gcpxml://storage.googleapis.com:aseaton_bucket?access=...&secret=...") ac = Arctic("gcpxmls://storage.googleapis.com:aseaton_bucket?access=...&secret=...") # Kept aws_auth naming as this will use ~/.aws/credentials ac = Arctic("gcpxml://storage.googleapis.com:aseaton_bucket?aws_auth=true&aws_profile=blah") ac = Arctic("gcpxml://storage.googleapis.com:aseaton_bucket?aws_auth=true&prefix=my_prefix") ``` We have a URL of `gcpxml` so that if we do a full GCP implementation later, users can get it with `gcp://`. I've added a GCP storage proto object so that users on the `gcpxml` API will later be able to move to a `gcp://` API without updating any library configs. The proto is tiny, just storing the library's prefix, which is the only information we need to serialize. This is implemented as a small sublass of our existing S3 implementation, with its deletion functionality overridden. The Python library adapter creates the `NativeVersionStore` with a `GCPXMLSettings` in memory object (in the `NativeVariantStorage`) which makes us create a `GCPXMLStorage` C++ adapter rather than the normal `S3Storage`. The interesting part of this PR is the threading model in `do_remove_no_batching_impl`. This is called from an IO executor as part of the `RemoveTask` so it is not safe (due to the risk of deadlocks) to submit any work to the IO executor from it. Instead: - The storage client `s3_client_impl.cpp` submits delete operations with the S3 API. The S3 SDK has its own event loop to handle these. When they complete, they resolve a `folly::Promise`, and the client returns a `folly::Future` out of that promise. - Our storage adapter (`detail-inl.hpp`) collects these futures, on an inline executor. This is fine because there is no work for this executor to do, other than waiting to be notified that promises have been resolved by the AWS SDK. For testing, I created a `moto` based test fixture that returns errors when batch delete operations are invoked. Also tested manually against a real GCP backend. There's some duplication between the `gcpxml` and the `s3` Python adapters, but since hopefully the `gcpxml` adapter will be temporary until a full GCP implementation, I didn't see the value in refactoring it away. Monday: 8450684276

2025-02-13

WIP GCP XML support - dedicated proto type for upgrade path to JSON API later

2025-01-28

Adaptive encoding

Add some lightweight encodings

2025-01-20

Refactor aggregator set data, add statistics

2024-11-05

Bugfix/1841/maintain empty series names (#1983) #### Reference Issues/PRs Fixes #1841 #### What does this implement or fix? Before this change, if a `Series` had an empty-string as a name, this would be roundtripped as a `None`. This introduces a `has_name` bool to the normalization metadata protobuf, as a backwards-compatible way of effectively making the `name` field optional. The behaviour (which has been verified) can be summarised as follows: ``` Writer version | Series name | Protobuf name field | Protobuf has_name field | Series name read by <=5.0.0 | Series name read by this branch ---------------|-------------|---------------------|-------------------------|-----------------------------|-------------------------------- <=5.0.0 | "hello" | "hello" | Not present | "hello" | "hello" <=5.0.0 | "" | "" | Not present | None | None <=5.0.0 | None | "" | Not present | None | None This branch | "hello" | "hello" | True | "hello" | "hello" This branch | "" | "" | True | None | "" This branch | None | "" | False | None | None ```

Bugfix/1841/maintain empty series names (#1983) #### Reference Issues/PRs Fixes #1841 #### What does this implement or fix? Before this change, if a `Series` had an empty-string as a name, this would be roundtripped as a `None`. This introduces a `has_name` bool to the normalization metadata protobuf, as a backwards-compatible way of effectively making the `name` field optional. The behaviour (which has been verified) can be summarised as follows: ``` Writer version | Series name | Protobuf name field | Protobuf has_name field | Series name read by <=5.0.0 | Series name read by this branch ---------------|-------------|---------------------|-------------------------|-----------------------------|-------------------------------- <=5.0.0 | "hello" | "hello" | Not present | "hello" | "hello" <=5.0.0 | "" | "" | Not present | None | None <=5.0.0 | None | "" | Not present | None | None This branch | "hello" | "hello" | True | "hello" | "hello" This branch | "" | "" | True | None | "" This branch | None | "" | False | None | None ```

2024-10-31

snapshot

Address PR comments

non-protobuf new s3 settings snapshot

-Introduce aws_auth to proto (_RBAC_ is not replaced yet) -Add STS auth method to codebase (But no test yet)

2024-10-28

Revert "Bugfix 1841: Correctly roundtrip None and empty string pd.Series names (#1878)" (#1953) #### Reference Issues/PRs This reverts commit 2c8d74b579290f595bf1f39b6a489e4ccb9616b2. The fix was broken, new clients incorrectly read Series with non-empty strings as names written with older clients as `None`

Revert "Bugfix 1841: Correctly roundtrip None and empty string pd.Series names (#1878)" (#1953) #### Reference Issues/PRs This reverts commit d6bdd48f0f3e4c971588b0680b4089409646eaa9. The fix was broken, new clients incorrectly read Series with non-empty strings as names written with older clients as `None`

2024-10-16

-Introduce aws_auth to proto (_RBAC_ is not replaced yet) -Add STS auth method to codebase (But no test yet)

2024-10-15

-Introduce aws_auth to proto (_RBAC_ is not replaced yet) -Add STS auth method to codebase (But no test yet) Add auto test More detail readiness check Fast test quicker test Format fix print more log Shorten the test Always print Change test More specific test More test Fix test Remove pytest mark Complete branch test Quick test fix xdist test Try fix python selection Fix test Update venv Update fix More echo Test Test More echo Default to py310 Update Update python ver print py version 310 default Remove version

2024-10-11

Bugfix 1841: Correctly roundtrip None and empty string pd.Series names (#1878) #### Reference Issues/PRs Fixes #1841

Bugfix 1841: Correctly roundtrip None and empty string pd.Series names (#1878) #### Reference Issues/PRs Fixes #1841

2024-09-10

Support reading from prefixes that include a dot

Support reading from prefixes that include a dot

2024-08-12

Remove rocksdb

Remove rocksdb

2024-08-09

Fix Segment use-after-move when replicating to NFS (#1756) ## Motivation (this section copied from previous (closed) attempt - https://github.com/man-group/ArcticDB/pull/1746) The motivation for the change is to allow `arcticdb-enterprise` to copy blocks to NFS storages without a use-after-move. I explained this in https://github.com/man-group/arcticdb-enterprise/pull/139 but to have an open record: CopyCompressedInterStoreTask has: ``` // Don't bother copying the key segment pair when writing to the final target if (it == std::prev(target_stores_.end())) { (*it)->write_compressed_sync(std::move(key_segment_pair)); } else { auto key_segment_pair_copy = key_segment_pair; (*it)->write_compressed_sync(std::move(key_segment_pair_copy)); } ``` KeySegmentPair has a shared_ptr to a KeySegmentPair, which we can think of here as just a `Segment`. Therefore the old `key_segment_pair_copy` is shallow, the underlying Segment is the same. But the segment eventually gets passed as an rvalue reference further down the stack. In `do_write_impl` we call `put_object` which calls `serialize_header`. This modifies the segment in place and passes that buffer to the AWS SDK. In the `NfsBackedStorage` we have: ``` void NfsBackedStorage::do_write(Composite<KeySegmentPair>&& kvs) { auto enc = kvs.transform([] (auto&& key_seg) { return KeySegmentPair{encode_object_id(key_seg.variant_key()), std::move(key_seg.segment())}; }); s3::detail::do_write_impl(std::move(enc), root_folder_, bucket_name_, *s3_client_, NfsBucketizer{}); } ``` where the segment gets moved from. Subsequent attempts to use the segment (eg copying on to the next store) then fail. https://github.com/man-group/arcticdb-enterprise/pull/139 fixed this issue by cloning the segment, but this approach avoids the (expensive) clone. ## Logical Change Copy the `KeySegmentPair`'s pointer to the `Segment` in `nfs_backed_storage.cpp` rather than moving from the segment. ## Refactor and Testing ### Copy Task Move the CopyCompressedInterStoreTask down to ArcticDB from arcticdb-enterprise. Add a test for it on NFS storage. I've verified that the tests in this commit fail without the refactor in the HEAD~1 commit. The only changes to `CopyCompressedInterstoreTask` from enterprise are: - Pass the `KeySegmentPair` by value in to `write_compressed{_sync}`. The `KeySegmentPair` is cheap to copy (especially considering we are about to copy an object across storages, likely with a network hop). - We have adopted the new `set_key` API of `KeySegmentPair`: ``` if (key_to_write_.has_value()) { key_segment_pair.set_key(*key_to_write_); } ``` - We have namespaced the `ProcessingResult` struct in to the task ### KeySegmentPair - Replace methods returning mutable lvalue references to keys with a `set_key` method. - Remove the `release_segment` method as it dangerously leaves the `KeySegmentPair` pointing at a `Segment` object that has been moved from, and it is not actually necessary. ## Follow up work The non-const `Segment& KeySegmentPair#segment()` API is still dangerous and error prone. I have a follow up change to remove it, but that API change affects very many files and will be best raised separately so that it doesn't block this fix for replication. A draft PR showing a proposal for that change is here - https://github.com/man-group/ArcticDB/pull/1757 .

Fix Segment use-after-move when replicating to NFS (#1756) ## Motivation (this section copied from previous (closed) attempt - https://github.com/man-group/ArcticDB/pull/1746) The motivation for the change is to allow `arcticdb-enterprise` to copy blocks to NFS storages without a use-after-move. I explained this in https://github.com/man-group/arcticdb-enterprise/pull/139 but to have an open record: CopyCompressedInterStoreTask has: ``` // Don't bother copying the key segment pair when writing to the final target if (it == std::prev(target_stores_.end())) { (*it)->write_compressed_sync(std::move(key_segment_pair)); } else { auto key_segment_pair_copy = key_segment_pair; (*it)->write_compressed_sync(std::move(key_segment_pair_copy)); } ``` KeySegmentPair has a shared_ptr to a KeySegmentPair, which we can think of here as just a `Segment`. Therefore the old `key_segment_pair_copy` is shallow, the underlying Segment is the same. But the segment eventually gets passed as an rvalue reference further down the stack. In `do_write_impl` we call `put_object` which calls `serialize_header`. This modifies the segment in place and passes that buffer to the AWS SDK. In the `NfsBackedStorage` we have: ``` void NfsBackedStorage::do_write(Composite<KeySegmentPair>&& kvs) { auto enc = kvs.transform([] (auto&& key_seg) { return KeySegmentPair{encode_object_id(key_seg.variant_key()), std::move(key_seg.segment())}; }); s3::detail::do_write_impl(std::move(enc), root_folder_, bucket_name_, *s3_client_, NfsBucketizer{}); } ``` where the segment gets moved from. Subsequent attempts to use the segment (eg copying on to the next store) then fail. https://github.com/man-group/arcticdb-enterprise/pull/139 fixed this issue by cloning the segment, but this approach avoids the (expensive) clone. ## Logical Change Copy the `KeySegmentPair`'s pointer to the `Segment` in `nfs_backed_storage.cpp` rather than moving from the segment. ## Refactor and Testing ### Copy Task Move the CopyCompressedInterStoreTask down to ArcticDB from arcticdb-enterprise. Add a test for it on NFS storage. I've verified that the tests in this commit fail without the refactor in the HEAD~1 commit. The only changes to `CopyCompressedInterstoreTask` from enterprise are: - Pass the `KeySegmentPair` by value in to `write_compressed{_sync}`. The `KeySegmentPair` is cheap to copy (especially considering we are about to copy an object across storages, likely with a network hop). - We have adopted the new `set_key` API of `KeySegmentPair`: ``` if (key_to_write_.has_value()) { key_segment_pair.set_key(*key_to_write_); } ``` - We have namespaced the `ProcessingResult` struct in to the task ### KeySegmentPair - Replace methods returning mutable lvalue references to keys with a `set_key` method. - Remove the `release_segment` method as it dangerously leaves the `KeySegmentPair` pointing at a `Segment` object that has been moved from, and it is not actually necessary. ## Follow up work The non-const `Segment& KeySegmentPair#segment()` API is still dangerous and error prone. I have a follow up change to remove it, but that API change affects very many files and will be best raised separately so that it doesn't block this fix for replication. A draft PR showing a proposal for that change is here - https://github.com/man-group/ArcticDB/pull/1757 .

2024-08-05

Move CopyCompressedInterStoreTask down to ArcticDB for testability

Move CopyCompressedInterStoreTask down to ArcticDB

2024-08-01

Read index API (#1568) #### Reference Issues/PRs <!--Example: Fixes #1234. See also #3456.--> Resolve #1150 #### What does this implement or fix? Read only the index column of a DataFrame stored in ArcticDB. Implemented only the V2 Library API as part of the `read` call. * Passing `columns=None` will require all columns in the DF * Passing `columns=[]` will return a DF containing only the index columns #### Any other comments? #### Checklist <details> <summary> Checklist for code changes... </summary> - [ ] Have you updated the relevant docstrings, documentation and copyright notice? - [ ] Is this contribution tested against [all ArcticDB's features](../docs/mkdocs/docs/technical/contributing.md)? - [ ] Do all exceptions introduced raise appropriate [error messages](https://docs.arcticdb.io/error_messages/)? - [ ] Are API changes highlighted in the PR description? - [ ] Is the PR labelled as enhancement or bug so it appears in autogenerated release notes? </details> <!-- Thanks for contributing a Pull Request to ArcticDB! Please ensure you have taken a look at: - ArcticDB's Code of Conduct: https://github.com/man-group/ArcticDB/blob/master/CODE_OF_CONDUCT.md - ArcticDB's Contribution Licensing: https://github.com/man-group/ArcticDB/blob/master/docs/mkdocs/docs/technical/contributing.md#contribution-licensing --> --------- Co-authored-by: Vasil Pashov <vasil.pashov@man.com>

Read index API (#1568) #### Reference Issues/PRs <!--Example: Fixes #1234. See also #3456.--> Resolve #1150 #### What does this implement or fix? Read only the index column of a DataFrame stored in ArcticDB. Implemented only the V2 Library API as part of the `read` call. * Passing `columns=None` will require all columns in the DF * Passing `columns=[]` will return a DF containing only the index columns #### Any other comments? #### Checklist <details> <summary> Checklist for code changes... </summary> - [ ] Have you updated the relevant docstrings, documentation and copyright notice? - [ ] Is this contribution tested against [all ArcticDB's features](../docs/mkdocs/docs/technical/contributing.md)? - [ ] Do all exceptions introduced raise appropriate [error messages](https://docs.arcticdb.io/error_messages/)? - [ ] Are API changes highlighted in the PR description? - [ ] Is the PR labelled as enhancement or bug so it appears in autogenerated release notes? </details> <!-- Thanks for contributing a Pull Request to ArcticDB! Please ensure you have taken a look at: - ArcticDB's Code of Conduct: https://github.com/man-group/ArcticDB/blob/master/CODE_OF_CONDUCT.md - ArcticDB's Contribution Licensing: https://github.com/man-group/ArcticDB/blob/master/docs/mkdocs/docs/technical/contributing.md#contribution-licensing --> --------- Co-authored-by: Vasil Pashov <vasil.pashov@man.com>

2024-06-25

Add binary encoding format

Add binary encoding format

2024-06-05

Add binary encoding format

Comment bucketize_dynamic in the proto file

2024-05-29

-Set ca path and directory automatically Revoke removing assert Remove useless ca cert path in non ssl enabled testing environment Address PR comment Better test Address PR comments Update docs/mkdocs/docs/api/arctic_uri.md Co-authored-by: Alex Seaton <alexbseaton@gmail.com>

-Set ca path and directory automatically Revoke removing assert Remove useless ca cert path in non ssl enabled testing environment Address PR comment Better test Address PR comments Update docs/mkdocs/docs/api/arctic_uri.md Co-authored-by: Alex Seaton <alexbseaton@gmail.com>

2024-04-25

Add empty index type and feature flag for it (#1524) #### Reference Issues/PRs <!--Example: Fixes #1234. See also #3456.--> #### What does this implement or fix? #### Any other comments? #### Checklist <details> <summary> Checklist for code changes... </summary> - [ ] Have you updated the relevant docstrings, documentation and copyright notice? - [ ] Is this contribution tested against [all ArcticDB's features](../docs/mkdocs/docs/technical/contributing.md)? - [ ] Do all exceptions introduced raise appropriate [error messages](https://docs.arcticdb.io/error_messages/)? - [ ] Are API changes highlighted in the PR description? - [ ] Is the PR labelled as enhancement or bug so it appears in autogenerated release notes? </details> <!-- Thanks for contributing a Pull Request to ArcticDB! Please ensure you have taken a look at: - ArcticDB's Code of Conduct: https://github.com/man-group/ArcticDB/blob/master/CODE_OF_CONDUCT.md - ArcticDB's Contribution Licensing: https://github.com/man-group/ArcticDB/blob/master/docs/mkdocs/docs/technical/contributing.md#contribution-licensing --> --------- Co-authored-by: Vasil Pashov <vasil.pashov@man.com>

Add empty index type and feature flag for it (#1524) #### Reference Issues/PRs <!--Example: Fixes #1234. See also #3456.--> #### What does this implement or fix? #### Any other comments? #### Checklist <details> <summary> Checklist for code changes... </summary> - [ ] Have you updated the relevant docstrings, documentation and copyright notice? - [ ] Is this contribution tested against [all ArcticDB's features](../docs/mkdocs/docs/technical/contributing.md)? - [ ] Do all exceptions introduced raise appropriate [error messages](https://docs.arcticdb.io/error_messages/)? - [ ] Are API changes highlighted in the PR description? - [ ] Is the PR labelled as enhancement or bug so it appears in autogenerated release notes? </details> <!-- Thanks for contributing a Pull Request to ArcticDB! Please ensure you have taken a look at: - ArcticDB's Code of Conduct: https://github.com/man-group/ArcticDB/blob/master/CODE_OF_CONDUCT.md - ArcticDB's Contribution Licensing: https://github.com/man-group/ArcticDB/blob/master/docs/mkdocs/docs/technical/contributing.md#contribution-licensing --> --------- Co-authored-by: Vasil Pashov <vasil.pashov@man.com>

2024-04-12

Add S3 ca path support

Add S3 ca path support

2024-04-02

Implement empty index for 0-rowed columns (#1429) #### Reference Issues/PRs Closes: #1428 #### What does this implement or fix? Create an empty-index type. This required change in the Python and in the C++ layer. * In the C++ layer an new index type was added. (IndexDescriptor::EMPTY). It does not allocate a filed in the storage (similar to how row range index does not allocate a field). The checks for index compatibility are relaxed, the empty index is compatible with all other index types and it gets overridden the first time a non-empty index is written (either through update or append). On write we check if the dataframe contains 0 rows and if so it gets assigned an empty index. * The logic in the python layer is dodgy and needs discussion. In the current state the normalization metadata and the index descriptor are stored separately. There is one proto message describing both DateTime index and Ranged Index. The current change made it so that in case of 0 rows the python layer passes RowRange index to the C++ layer which checks if there are any rows in the DF. If there are rows Row range index is used, otherwise empty index is used. Note the `is_not_range_index` proto field. IMO it needs some refactoring in further PRs. It's used in the python layer to check if the first column is index or not. #### Any other comments? Merge this after: https://github.com/man-group/ArcticDB/pull/1436 #### Checklist <details> <summary> Checklist for code changes... </summary> - [ ] Have you updated the relevant docstrings, documentation and copyright notice? - [ ] Is this contribution tested against [all ArcticDB's features](../docs/mkdocs/docs/technical/contributing.md)? - [ ] Do all exceptions introduced raise appropriate [error messages](https://docs.arcticdb.io/error_messages/)? - [ ] Are API changes highlighted in the PR description? - [ ] Is the PR labelled as enhancement or bug so it appears in autogenerated release notes? </details> <!-- Thanks for contributing a Pull Request to ArcticDB! Please ensure you have taken a look at: - ArcticDB's Code of Conduct: https://github.com/man-group/ArcticDB/blob/master/CODE_OF_CONDUCT.md - ArcticDB's Contribution Licensing: https://github.com/man-group/ArcticDB/blob/master/docs/mkdocs/docs/technical/contributing.md#contribution-licensing --> --------- Co-authored-by: Vasil Pashov <vasil.pashov@man.com>

2024-03-21

Feature flag empty (#1440) Feature flag off the empty type behaviour by default, allow it to be re-enabled at the library level --------- Co-authored-by: Nick Clarke <nclarke@live.co.uk>

Feature flag empty (#1440) Feature flag off the empty type behaviour by default, allow it to be re-enabled at the library level --------- Co-authored-by: Nick Clarke <nclarke@live.co.uk>

2024-03-14

LMDB exception normalization with mock client (#1414) This is the final PR for exception normalization. Previous PRs are #1411, #1360, #1344, #1304, #1297 and #1285 #### Reference Issues/PRs Previously #1285 only normalized `KeyNotFoundException` and `DuplicateKeyException` correctly. As mentioned in [this comment](https://github.com/man-group/ArcticDB/pull/1285#discussion_r1474179896), we need to normalize other lmdb specific errors too. This PR does that. A mock client is also created to simulate the lmdb errors which aren't easily produce-able with real lmdb but can occur. The ErrorCode list has been updated in `error_code.hpp`. Previously, all the storage error codes were just sequential. This required that each time a new error was added for a specific storage, all the other error codes needed to be changed as we want to assign sequential error codes to the same storage. We now leave 10 error codes for each storage which allows us to easily add new error codes for a storage without having to change all the others. `::lmdb::map_full_error` is now normalized. When this error occurs, lmdb throws `LMDBMapFullException` which is child of `StorageException` #### What does this implement or fix? #### Any other comments? #### Checklist <details> <summary> Checklist for code changes... </summary> - [ ] Have you updated the relevant docstrings, documentation and copyright notice? - [ ] Is this contribution tested against [all ArcticDB's features](../docs/mkdocs/docs/technical/contributing.md)? - [ ] Do all exceptions introduced raise appropriate [error messages](https://docs.arcticdb.io/error_messages/)? - [ ] Are API changes highlighted in the PR description? - [ ] Is the PR labelled as enhancement or bug so it appears in autogenerated release notes? </details> <!-- Thanks for contributing a Pull Request to ArcticDB! Please ensure you have taken a look at: - ArcticDB's Code of Conduct: https://github.com/man-group/ArcticDB/blob/master/CODE_OF_CONDUCT.md - ArcticDB's Contribution Licensing: https://github.com/man-group/ArcticDB/blob/master/docs/mkdocs/docs/technical/contributing.md#contribution-licensing -->

LMDB exception normalization with mock client (#1414) This is the final PR for exception normalization. Previous PRs are #1411, #1360, #1344, #1304, #1297 and #1285 #### Reference Issues/PRs Previously #1285 only normalized `KeyNotFoundException` and `DuplicateKeyException` correctly. As mentioned in [this comment](https://github.com/man-group/ArcticDB/pull/1285#discussion_r1474179896), we need to normalize other lmdb specific errors too. This PR does that. A mock client is also created to simulate the lmdb errors which aren't easily produce-able with real lmdb but can occur. The ErrorCode list has been updated in `error_code.hpp`. Previously, all the storage error codes were just sequential. This required that each time a new error was added for a specific storage, all the other error codes needed to be changed as we want to assign sequential error codes to the same storage. We now leave 10 error codes for each storage which allows us to easily add new error codes for a storage without having to change all the others. `::lmdb::map_full_error` is now normalized. When this error occurs, lmdb throws `LMDBMapFullException` which is child of `StorageException` #### What does this implement or fix? #### Any other comments? #### Checklist <details> <summary> Checklist for code changes... </summary> - [ ] Have you updated the relevant docstrings, documentation and copyright notice? - [ ] Is this contribution tested against [all ArcticDB's features](../docs/mkdocs/docs/technical/contributing.md)? - [ ] Do all exceptions introduced raise appropriate [error messages](https://docs.arcticdb.io/error_messages/)? - [ ] Are API changes highlighted in the PR description? - [ ] Is the PR labelled as enhancement or bug so it appears in autogenerated release notes? </details> <!-- Thanks for contributing a Pull Request to ArcticDB! Please ensure you have taken a look at: - ArcticDB's Code of Conduct: https://github.com/man-group/ArcticDB/blob/master/CODE_OF_CONDUCT.md - ArcticDB's Contribution Licensing: https://github.com/man-group/ArcticDB/blob/master/docs/mkdocs/docs/technical/contributing.md#contribution-licensing -->

2024-03-08

#447 Add a `MockMongoClient` which can simulate mongo failures (#1395) Similar to #1331. The mock client will then be used to test exception normalization. Wrap up common things like `StorageOperation` and `StorageFailure` into `storage_mock_client.hpp`

#447 Add a `MockMongoClient` which can simulate mongo failures (#1395) Similar to #1331. The mock client will then be used to test exception normalization. Wrap up common things like `StorageOperation` and `StorageFailure` into `storage_mock_client.hpp`

2024-03-05

Metadata cache work

2024-03-04

Refactor to allow multiple segments in the same physical object

Refactor to allow multiple segments in the same physical object

2024-02-20

Bugfix: Empty type (#1227) #### Reference Issues/PRs Closes #1107 #### What does this implement or fix? Fixes how empty type interacts with other types. In general we should be able to: * Append other types to columns which were initially empty * Append empty columns to columns of any other type * Update values with empty type preserving the type of the column ### Changes: * Each type handler now has a function to report the byte size of its elements. * Each type handler now has a function to default initialize some memory * Empty type handler now backfills the "empty" elements. * integer types -> 0 (Not perfect but in future we're planning to add default value argument) * float types -> NaN * string types -> None * bool -> False * Nullable boolean -> None * Date -> NaT * The function which does default initialization up to now was used only in dynamic schema. Now the empty handler calls it as well to do the backfill. The function first checks the PoD types and if any of them matches uses the basic default initialization, otherwise it checks if there is a type handler and if so uses its default_initialize functionality * Refactor how updating works. `Column::truncate` is used instead of copying segment rows one by one. This should improve the performance of update. * Add a new python fixture `lmdb_version_store_static_and_dynamic` to cover all combinations {V1, V2} ecoding x {Static, Dynamic} schema * Empty typed columns are now reported as dense columns. They don't have a sparse map and both physical and logical rows are left uninitialized (value `-1`) **DISCUSS** - [x] Should we add an option to support updating non-empty stuff with empty (Conclusion reached in a slack thread: yes, we should allow to update with None as long as the type of the output is the same as the type of the column.) - [x] What should be the output for the following example (a column of none vs a column of 0) (Conclusion reached in a slack thread. The result should be [0,0], i.e. the output should have the same type as the type of the column.) ```python lib.write("sym", pd.DataFrame({"col": [1,2,3]})) lib.append("sym", pd.DataFrame({"col": [None, None]})) lib.read("sym", row_range=[3:5]).data ``` - [ ] Do we need hypothesis testing random appends of empty and non-empty stuff to the same column? **Dev TODO** - [x] Verify the following throws ```python lib.write("sym", pd.DataFrame({"col": [None, None]})) lib.append("sym", pd.DataFrame({"col": [1, 2, 3]})) lib.append("sym", pd.DataFrame({"col": ["some", "string"]})) ``` - [x] Appending to empty for dynamic schema - [x] Fix appending empty to other types with static schema - [x] Fix appending empty to other types with dynamic schema - [x] Create a single function to handle backfilling of data and use it both in the empty handler and in reduce_and_fix - [x] Change the name of PYBOOL type - [x] Fix update e.g. ```python lmdb_version_store_v2.write('test', pd.DataFrame([None, None], index=pd.date_range(periods=2, end=dt.now(), freq='1T'))) lmdb_version_store_v2.update('test', pd.DataFrame([None, None], index=pd.date_range(periods=2, end=dt.now(), freq='1T'))) ``` - [ ] Add tests for starting with empty column list e.g. `pd.DataFrame({"col": []})`. Potentially mark it as xfail and fix with a later PR. - [ ] Add tests for update when the update range is not entirely contained in the dataframe range index #### Any other comments? #### Checklist <details> <summary> Checklist for code changes... </summary> - [ ] Have you updated the relevant docstrings, documentation and copyright notice? - [ ] Is this contribution tested against [all ArcticDB's features](../docs/mkdocs/docs/technical/contributing.md)? - [ ] Do all exceptions introduced raise appropriate [error messages](https://docs.arcticdb.io/error_messages/)? - [ ] Are API changes highlighted in the PR description? - [ ] Is the PR labelled as enhancement or bug so it appears in autogenerated release notes? </details> <!-- Thanks for contributing a Pull Request to ArcticDB! Please ensure you have taken a look at: - ArcticDB's Code of Conduct: https://github.com/man-group/ArcticDB/blob/master/CODE_OF_CONDUCT.md - ArcticDB's Contribution Licensing: https://github.com/man-group/ArcticDB/blob/master/docs/mkdocs/docs/technical/contributing.md#contribution-licensing --> --------- Co-authored-by: Vasil Pashov <vasil.pashov@man.com>

Bugfix: Empty type (#1227) #### Reference Issues/PRs Closes #1107 #### What does this implement or fix? Fixes how empty type interacts with other types. In general we should be able to: * Append other types to columns which were initially empty * Append empty columns to columns of any other type * Update values with empty type preserving the type of the column ### Changes: * Each type handler now has a function to report the byte size of its elements. * Each type handler now has a function to default initialize some memory * Empty type handler now backfills the "empty" elements. * integer types -> 0 (Not perfect but in future we're planning to add default value argument) * float types -> NaN * string types -> None * bool -> False * Nullable boolean -> None * Date -> NaT * The function which does default initialization up to now was used only in dynamic schema. Now the empty handler calls it as well to do the backfill. The function first checks the PoD types and if any of them matches uses the basic default initialization, otherwise it checks if there is a type handler and if so uses its default_initialize functionality * Refactor how updating works. `Column::truncate` is used instead of copying segment rows one by one. This should improve the performance of update. * Add a new python fixture `lmdb_version_store_static_and_dynamic` to cover all combinations {V1, V2} ecoding x {Static, Dynamic} schema * Empty typed columns are now reported as dense columns. They don't have a sparse map and both physical and logical rows are left uninitialized (value `-1`) **DISCUSS** - [x] Should we add an option to support updating non-empty stuff with empty (Conclusion reached in a slack thread: yes, we should allow to update with None as long as the type of the output is the same as the type of the column.) - [x] What should be the output for the following example (a column of none vs a column of 0) (Conclusion reached in a slack thread. The result should be [0,0], i.e. the output should have the same type as the type of the column.) ```python lib.write("sym", pd.DataFrame({"col": [1,2,3]})) lib.append("sym", pd.DataFrame({"col": [None, None]})) lib.read("sym", row_range=[3:5]).data ``` - [ ] Do we need hypothesis testing random appends of empty and non-empty stuff to the same column? **Dev TODO** - [x] Verify the following throws ```python lib.write("sym", pd.DataFrame({"col": [None, None]})) lib.append("sym", pd.DataFrame({"col": [1, 2, 3]})) lib.append("sym", pd.DataFrame({"col": ["some", "string"]})) ``` - [x] Appending to empty for dynamic schema - [x] Fix appending empty to other types with static schema - [x] Fix appending empty to other types with dynamic schema - [x] Create a single function to handle backfilling of data and use it both in the empty handler and in reduce_and_fix - [x] Change the name of PYBOOL type - [x] Fix update e.g. ```python lmdb_version_store_v2.write('test', pd.DataFrame([None, None], index=pd.date_range(periods=2, end=dt.now(), freq='1T'))) lmdb_version_store_v2.update('test', pd.DataFrame([None, None], index=pd.date_range(periods=2, end=dt.now(), freq='1T'))) ``` - [ ] Add tests for starting with empty column list e.g. `pd.DataFrame({"col": []})`. Potentially mark it as xfail and fix with a later PR. - [ ] Add tests for update when the update range is not entirely contained in the dataframe range index #### Any other comments? #### Checklist <details> <summary> Checklist for code changes... </summary> - [ ] Have you updated the relevant docstrings, documentation and copyright notice? - [ ] Is this contribution tested against [all ArcticDB's features](../docs/mkdocs/docs/technical/contributing.md)? - [ ] Do all exceptions introduced raise appropriate [error messages](https://docs.arcticdb.io/error_messages/)? - [ ] Are API changes highlighted in the PR description? - [ ] Is the PR labelled as enhancement or bug so it appears in autogenerated release notes? </details> <!-- Thanks for contributing a Pull Request to ArcticDB! Please ensure you have taken a look at: - ArcticDB's Code of Conduct: https://github.com/man-group/ArcticDB/blob/master/CODE_OF_CONDUCT.md - ArcticDB's Contribution Licensing: https://github.com/man-group/ArcticDB/blob/master/docs/mkdocs/docs/technical/contributing.md#contribution-licensing --> --------- Co-authored-by: Vasil Pashov <vasil.pashov@man.com>

2024-02-16

Add a `MockAzureClient` which can simulate azure failures (#1331) This is similar to #1281. - Adds a config option to use the `MockAzureClient` instead of the `RealAzureClient` - If a `MockAzureClient` is created, one can simulate failures by passing the appropriate symbol name - Creates tests to ensure that `MockAzureClient` works properly for `read`, `write`, `update`, `delete` and `list` functions. To do: Test exceptions thrown by `MockAzureClient`. This will be done in the next PR where we normalize Azure exception. #### Checklist <details> <summary> Checklist for code changes... </summary> - [ ] Have you updated the relevant docstrings, documentation and copyright notice? - [ ] Is this contribution tested against [all ArcticDB's features](../docs/mkdocs/docs/technical/contributing.md)? - [ ] Do all exceptions introduced raise appropriate [error messages](https://docs.arcticdb.io/error_messages/)? - [ ] Are API changes highlighted in the PR description? - [ ] Is the PR labelled as enhancement or bug so it appears in autogenerated release notes? </details>

Add a `MockAzureClient` which can simulate azure failures (#1331) This is similar to #1281. - Adds a config option to use the `MockAzureClient` instead of the `RealAzureClient` - If a `MockAzureClient` is created, one can simulate failures by passing the appropriate symbol name - Creates tests to ensure that `MockAzureClient` works properly for `read`, `write`, `update`, `delete` and `list` functions. To do: Test exceptions thrown by `MockAzureClient`. This will be done in the next PR where we normalize Azure exception. #### Checklist <details> <summary> Checklist for code changes... </summary> - [ ] Have you updated the relevant docstrings, documentation and copyright notice? - [ ] Is this contribution tested against [all ArcticDB's features](../docs/mkdocs/docs/technical/contributing.md)? - [ ] Do all exceptions introduced raise appropriate [error messages](https://docs.arcticdb.io/error_messages/)? - [ ] Are API changes highlighted in the PR description? - [ ] Is the PR labelled as enhancement or bug so it appears in autogenerated release notes? </details>

2024-02-04

Saving stuff before build server decom

2024-02-02

Adds a MockS3Client which can simulate s3 failures - Adds a config option to use the MockS3Client instead of the RealS3Client - If a MockS3Client is created, one can simulate failures by passing the appropriate symbol name - Adds tests which run various s3 storage failure scenarios - The tests expose two issues with s3 which will be fixed in a follow up commit

Adds a MockS3Client which can simulate s3 failures - Adds a config option to use the MockS3Client instead of the RealS3Client - If a MockS3Client is created, one can simulate failures by passing the appropriate symbol name - Adds tests which run various s3 storage failure scenarios - The tests expose two issues with s3 which will be fixed in a follow up commit