Proto commits in man-group/ArcticDB

These commits are when the Protocol Buffers files have changed: (only the last 100 relevant commits are shown)

Commit:cc49adf
Author:Vasil Pashov
Committer:Vasil Pashov

Implement pickle on failure as an override Due to legacy ArcticDB has multiple ways of selecting whether pickling will happen. ## Current state ### use_norm_failure_handler_known_types `use_norm_failure_handler_known_types` is a protobuf field in `VersionStoreConfig`. It's read when the `NativeVersionStore` class is initialized. `NativeVersionStore._normalizer` is a `CompositeNormalizer` which gets two parameters in its constructor a `normalization_failure_handler` and `use_norm_failure_handler_known_types`. Currently in all user facing use cases the `normalization_failure_handler` is the `MsgPackNormalizer` (it can be None in case of writing dataframe to file in file.py::normalization_failure_handler). If `use_norm_failure_handler_known_types` is `True` then every normalizer, (except `ArrowTableNormalizer`, which is probably a bug) is wrapped in `KnownTypeFallbackOnError` which first tries to do the normalization and if an exception is thrown it tries to run the `MsgPackNormalizer`. Arctic Explorer allows setting `use_norm_failure_handler_known_types`. There aren't any checks for existence of environment variable corresponding to `use_norm_failure_handler_known_types` nor are there any function kwargs. ### pickle_on_failure `pickle_on_failure` is a protobuf field of `VersionStoreConfig.WriteOptions`. There is no user-facing API for setting it. However, the kwarg used to control pickling per method is named `pickle_on_failure` and the environment variable used is also `PICKLE_ON_FAILURE`. ### disallow_pickle and recursive_normalize_msgpack_no_pickle_fallback `recursive_normalize_msgpack_no_pickle_fallback` is a toplevel V1 kwarg which is not officially documented. The naming is confusing. It has wider implications. This parameter controls how msgpack works when it encounters leaf which it cannot normalize. If `recursive_normalize_msgpack_no_pickle_fallback` is `True` it will raise. If it's not set it falls back to `VersionStoreConfig.MsgPack.strict_mode` and uses Python pickling only if `strict_mode` is off (the default). Internally it is passed to the `MsgPackNormalizer` as `disallow_pickle`. ## New behavior ### Precedence Pickling is controlled in the following order of precedence: 1. kwarg `pickle_on_failure` 2. environment variable `PICKLE_ON_FAILURE` 3. `use_norm_failure_handler_known_types` 4. If none of the above is set `pickle_on_failure` defaults to `False` Note: there is no environment variable corresponding to `use_norm_failure_handler_known_types`. `USE_NORM_FAILURE_HANDLER_KNOWN_TYPES` is not checked. ### What pickle_on_failure controls Due to legacy and unfortunate turn of events `pickle_on_failure`'s scope is not trivial. It affects what happens when **native normalization** fails. Native normalization is defined as types which have defined normalizer in the ArcticDB codebase. These normalizers are the normalizers `CompositeNormalizer` tries using before falling back to the fallback normalizer (if any): * `DataFrameNormalizer` * `SeriesNormalizer` * `TimeFrameNormalizer` * `NdArrayNormalizer` * `ArrowTableNormalizer` If any of these normalizers is selected and fails `pickle_on_failure` controls whether to retry normalizing the input via `MsgPack` ### MsgPack Historically `MsgPack` was also referred as pickling, but this is not entirely correct. `MsgPack` can normalize JSON-like python objects and store them as a binary blob inside the symbol. The commonality between `MsgPack` and Python pickling is that a binary blob is stored. This blob cannot be processed via processing pipeline, it cannot be appended, it cannot take date/row range. From C++ point of view `MsgPack` is the same as the Python native pickling and this is why the two are used interchangeably. The big difference is that MsgPack can process only a limited set of types. These are only types that JSON-like Python object contain: dict, array, numbers (int and float), strings, booleans. The `CompositeNormalizer` used by write methods of ArcticDB uses `MsgPack` as a fallback normalizer. This means that if the passed item's type is not suitable for any of the native normalizers `MsgPack` will be used. Note: there's a difference between native normalizer failing and a type not being suitable for a native normalizer. Example: * `lib.write("test", pd.DataFrame([[1]], columns=[pd.Timestamp(0)]))` the type of the item is `DataFrame` it's suitable for the `DataFrameNormalizer` so the `DataFrameNormalizer` is selected. However, the normalizer will fail because column names cannot be `pd.Timestamp`. If `pickle_on_failure=True` this will be pickled using Python native pickling and stored as a binary blob * `lib.write("test", "hello")` the type of the item is `str`, none of the native normalizers accepts strings. The fallback normalizer will be invoked. For our methods the fallback normalizer is `MsgPackNormalizer`. `MsgPack` can normalize strings. The data will be stored as a binary blob created by `MsgPack`, not by Python native pickling. #### `enable_python_native_pickling` As stated earlier `MsgPack` cannot normalize all types. Example: `[pd.DataFrame()]`. `enable_python_native_pickling` controls how the `MsgPackNormalizer` behaves when `MsgPack` reaches a leaf that cannot be normalized by `MsgPack`. If `enable_python_native_pickling=True` then `MsgPack` will invoke Python native pickling and store the leaf as Python pickled binary blob, otherwise the `MsgPackNormalizer` will throw. `enable_python_native_pickling` was named `recursive_normalize_msgpack_no_pickle_fallback` in the V1 API kwargs which suggest more narrow impact than the actual impact. Internally the `MsgPackNormalizer` used the name `disallow_pickle`. This PR unifies the name and also makes it more clear. #### `pickle_on_failure` respects `enable_python_native_pickling` Previously when native normalization failed and `pickle_on_failure=True` the item was passed to the `MsgPackNormalizer` without forwarding `recursive_normalize_msgpack_no_pickle_fallback`. This means that passing `pickle_on_failure=True` and `recursive_normalize_msgpack_no_pickle_fallback=True` resulted in Python native pickling even though it was explicitly disabled. Now `enable_python_native_pickling` is forwarded to the `MsgPackNormalizer` in all cases, so `pickle_on_failure=True` with `enable_python_native_pickling=False` will raise. #### Empty DataFrame and Series with non-normalizable column names **This is a breaking change** If a DataFrame contains a column whose name is not a `str` or an `int` it can't be normalized natively. Pickling/MsgPack normalization must be used. This is obvious for DataFrames having more than one row. However ArcticDB used to silently enable pickling no matter the value of `use_norm_failure_handler_known_types` in case the DataFrame or Series had 0 rows. This is now not true. There's no more silent normalization. This affects only the V1 API, V2 `write` already raised. ### Protobuf changes `WriteOptions.pickle_on_failure` is removed and both the name and the field number are reserved. Only one internal library used this protobuf setting. ### V2 API The following changes affect only libraries with `use_norm_failure_handler_known_types=True`. They are caused by `KnownTypeFallbackOnError` no longer being used. The same changes apply to the V1 API. * `stage` will now throw. `stage` always passes `pickle_on_failure=False`, previously the library setting silently normalized the data via `MsgPack` and nothing was written. * `append` and `update` will properly raise `ArcticDbNotYetImplemented`: "Not supported: normalizing, symbol: sym, Reason: ...." instead of `SchemaException` * `write` raises a different (more descriptive) exception. Instead of "Normalizing data by pickling has been disabled" it reports the actual reason the native normalization failed. | Input | Before | After | |---|---|---| | `pd.DataFrame([[1]], columns=[pd.Timestamp(0)])` | `ArcticException`: "Error while normalizing symbol=sym, Normalizing data by pickling has been disabled." | `ArcticException`: "Error while normalizing symbol=sym, Column and index names must be of type str or int, received 1970-01-01 00:00:00 of type <class 'pandas._libs.tslibs.timestamps.Timestamp'>" | | `pd.DataFrame({"a": [pd.DataFrame()]})` | `ArcticException`: "Error while normalizing symbol=sym, Normalizing data by pickling has been disabled." | `ArcticDbNotYetImplemented`: "Not supported: normalizing, symbol: sym, Reason: Failed to normalize column 'a' with dtype 'object'. Found first non-null value of type '<class 'pandas.core.frame.DataFrame'>', but only strings, unicode, ..." | | `"hello"`, `[pd.DataFrame()]` | `ArcticUnsupportedDataTypeException`: "Data is of a type that cannot be normalized. Consider using write_pickle instead." | unchanged |

Commit:b5a08e6
Author:George Slavov
Committer:George Slavov

Get rid of proto field for mock storage This was a copy paste that I didn't think through. I shan't be using it.

Commit:4c95201
Author:George Slavov
Committer:George Slavov

Fixup

Commit:0619808
Author:George Slavov
Committer:George Slavov

Insert nearly empty implementation of FileBackedStorage Everything builds and it's possible to invoke Arctic("file://storage_dir") but nearly every virtual method raises an exception. Solid foundation to build on. Next step, create a fixture and run the test suite. :^)

Commit:9e0fbcb
Author:George Slavov
Committer:George Slavov

Get rid of proto field for mock storage This was a copy paste that I didn't think through. I shan't be using it.

Commit:2cd0f75
Author:George Slavov
Committer:George Slavov

Fixup

Commit:24bdf8b
Author:George Slavov
Committer:George Slavov

Insert nearly empty implementation of FileBackedStorage Everything builds and it's possible to invoke Arctic("file://storage_dir") but nearly every virtual method raises an exception. Solid foundation to build on. Next step, create a fixture and run the test suite. :^)

Commit:3634801
Author:IvoDD
Committer:IvoDD

Rename ExperimentalArrow normalization metadata to Arrow (#3506) #### Reference Issues/PRs [12968104728](https://man312219.monday.com/boards/7852509418/pulses/12968104728) #### What does this implement or fix? Follow-up to #3504. Renames the Arrow normalization metadata, now that Arrow writes are no longer experimental: - `NormalizationMetadata.ExperimentalArrow` becomes `NormalizationMetadata.Arrow` - The `input_type` oneof field `experimental_arrow = 6` becomes `arrow = 7`. Field 6 is left unused, with a comment that it must not be reused. The rest is a mechanical rename across C++, Python and `docs/claude`. #### Any other comments? Compatibility: - Data written with the experimental field 6 does not read back as Arrow on this version, and data written by this version does not read as Arrow on older clients. This is deliberate for the 7.0.0 major release. - `get_description` / `get_info` report `input_type` as `"arrow"` instead of `"experimental_arrow"`. #### Checklist <details> <summary> Checklist for code changes... </summary> - [ ] Have you updated the relevant docstrings, documentation and copyright notice? - [ ] Is this contribution tested against [all ArcticDB's features](../docs/mkdocs/docs/technical/contributing.md)? - [ ] Do all exceptions introduced raise appropriate [error messages](https://docs.arcticdb.io/error_messages/)? - [ ] Are API changes highlighted in the PR description? - [ ] Is the PR labelled as enhancement or bug so it appears in autogenerated release notes? </details> <!-- Thanks for contributing a Pull Request to ArcticDB! Please ensure you have taken a look at: - ArcticDB's Code of Conduct: https://github.com/man-group/ArcticDB/blob/master/CODE_OF_CONDUCT.md - ArcticDB's Contribution Licensing: https://github.com/man-group/ArcticDB/blob/master/docs/mkdocs/docs/technical/contributing.md#contribution-licensing --> Co-authored-by: Ivo <ivo.dilov@man.com>

The documentation is generated from this commit.

Commit:b2a04f0
Author:IvoDD
Committer:IvoDD

Arrow and pandas interoperability (#3447) Monday ref: 11325694339 Monday ref: 18053173176 Allows coherent interoperability between arrow and pandas input and output format: - Consistent way to read as pandas data written as arrow (all timezones are applied correctly) - Consistent way to read as arrow data written as pandas (moves some of the conversion logic to C++ but behavior wise is unchanged) - Consistent way to combine arrow and pandas data via any of `append`, `update`, `concat`, `finalize`. Note that to combine arrow and pandas, the pandas dataframe needs to have the same column names. Any duplicate columns or unnamed indices will be handled via #3418 column renaming. Reading pandas written series with range index as arrow will now produce a pyarrow array / polars series ```python lib.write("symbol", pd.Series([1, 2, 3])) result = lib.read("symbol", output_format=OutputFormat.PYARROW).data ``` The PR is split in 5 commits for easier review: 1. Extends Arrow/pandas interoperability tests to cover staged writes and finalization. 2. Changes the way Arrow one dimansional structures are normalized to be consistent with pandas. This allows follow up combination easier. 3. Embeds pandas metadata in Arrow normalization metadata. Adds conversion utilites. Moves pandas to arrow conversion in C++ layer so arrow python normalization only needs to understand arrow metadata format. 4. Allows combining arrow and pandas in `combine_schema`. Refactors it so that: - When a mix of arrow and pandas dataframes is given all get converted to arrow with pandas upfront. - Resolve any required name mismatches upfront (so later normalization stages can rely on consistent column name to normalization mappings). 6. Aligns Arrow multi-index field names with pandas’ stored `__idx__` names for append, update, and merge-update. - Note that `stage` and `finalize` don't correctly combine pandas multiindex with arrow. Monday ref: 13124164631 - For `concat` this works without changes. It just renames the stored multiindex levels to unnamed. <details> <summary> Checklist for code changes... </summary> - [ ] Have you updated the relevant docstrings, documentation and copyright notice? - [ ] Is this contribution tested against [all ArcticDB's features](../docs/mkdocs/docs/technical/contributing.md)? - [ ] Do all exceptions introduced raise appropriate [error messages](https://docs.arcticdb.io/error_messages/)? - [ ] Are API changes highlighted in the PR description? - [ ] Is the PR labelled as enhancement or bug so it appears in autogenerated release notes? </details> <!-- Thanks for contributing a Pull Request to ArcticDB! Please ensure you have taken a look at: - ArcticDB's Code of Conduct: https://github.com/man-group/ArcticDB/blob/master/CODE_OF_CONDUCT.md - ArcticDB's Contribution Licensing: https://github.com/man-group/ArcticDB/blob/master/docs/mkdocs/docs/technical/contributing.md#contribution-licensing --> --------- Co-authored-by: Ivo <ivo.dilov@man.com>

Commit:b890c72
Author:Alex Owens
Committer:IvoDD

Enhancement 12844033169: Method to rename indexes and columns to be Arrow compatible

Commit:e509d10
Author:Alex Owens
Committer:IvoDD

Enhancement 18362804801: Support for round-tripping pyarrow record batches, chunked arrays, and arrays, and polars series (#3399) #### Reference Issues/PRs [9360502457](https://man312219.monday.com/boards/7852509418/pulses/9360502457) #### What does this implement or fix? Adds support for writing `pyarrow.RecordBatch/ChunkedArray/Array` and `polars.Series`. Due to our on-disk slicing, `RecordBatch` always reads back as a `Table`, and `Array` always reads back as a `ChunkedArray`, even if they only have 1 chunk per column. --------- Co-authored-by: Alex Owens <alex.owens@man.com>

Commit:fb726d5
Author:IvoDD
Committer:Ivo

Arrow and pandas interoperability (#3447) Monday ref: 11325694339 Monday ref: 18053173176 Allows coherent interoperability between arrow and pandas input and output format: - Consistent way to read as pandas data written as arrow (all timezones are applied correctly) - Consistent way to read as arrow data written as pandas (moves some of the conversion logic to C++ but behavior wise is unchanged) - Consistent way to combine arrow and pandas data via any of `append`, `update`, `concat`, `finalize`. Note that to combine arrow and pandas, the pandas dataframe needs to have the same column names. Any duplicate columns or unnamed indices will be handled via #3418 column renaming. Reading pandas written series with range index as arrow will now produce a pyarrow array / polars series ```python lib.write("symbol", pd.Series([1, 2, 3])) result = lib.read("symbol", output_format=OutputFormat.PYARROW).data ``` The PR is split in 5 commits for easier review: 1. Extends Arrow/pandas interoperability tests to cover staged writes and finalization. 2. Changes the way Arrow one dimansional structures are normalized to be consistent with pandas. This allows follow up combination easier. 3. Embeds pandas metadata in Arrow normalization metadata. Adds conversion utilites. Moves pandas to arrow conversion in C++ layer so arrow python normalization only needs to understand arrow metadata format. 4. Allows combining arrow and pandas in `combine_schema`. Refactors it so that: - When a mix of arrow and pandas dataframes is given all get converted to arrow with pandas upfront. - Resolve any required name mismatches upfront (so later normalization stages can rely on consistent column name to normalization mappings). 6. Aligns Arrow multi-index field names with pandas’ stored `__idx__` names for append, update, and merge-update. - Note that `stage` and `finalize` don't correctly combine pandas multiindex with arrow. Monday ref: 13124164631 - For `concat` this works without changes. It just renames the stored multiindex levels to unnamed. <details> <summary> Checklist for code changes... </summary> - [ ] Have you updated the relevant docstrings, documentation and copyright notice? - [ ] Is this contribution tested against [all ArcticDB's features](../docs/mkdocs/docs/technical/contributing.md)? - [ ] Do all exceptions introduced raise appropriate [error messages](https://docs.arcticdb.io/error_messages/)? - [ ] Are API changes highlighted in the PR description? - [ ] Is the PR labelled as enhancement or bug so it appears in autogenerated release notes? </details> <!-- Thanks for contributing a Pull Request to ArcticDB! Please ensure you have taken a look at: - ArcticDB's Code of Conduct: https://github.com/man-group/ArcticDB/blob/master/CODE_OF_CONDUCT.md - ArcticDB's Contribution Licensing: https://github.com/man-group/ArcticDB/blob/master/docs/mkdocs/docs/technical/contributing.md#contribution-licensing --> --------- Co-authored-by: Ivo <ivo.dilov@man.com>

Commit:8aedf70
Author:Alex Owens
Committer:Ivo

Enhancement 12844033169: Method to rename indexes and columns to be Arrow compatible

Commit:e8529db
Author:Alex Owens
Committer:Ivo

Enhancement 18362804801: Support for round-tripping pyarrow record batches, chunked arrays, and arrays, and polars series (#3399) #### Reference Issues/PRs [9360502457](https://man312219.monday.com/boards/7852509418/pulses/9360502457) #### What does this implement or fix? Adds support for writing `pyarrow.RecordBatch/ChunkedArray/Array` and `polars.Series`. Due to our on-disk slicing, `RecordBatch` always reads back as a `Table`, and `Array` always reads back as a `ChunkedArray`, even if they only have 1 chunk per column. --------- Co-authored-by: Alex Owens <alex.owens@man.com>

Commit:8a8a8b9
Author:IvoDD
Committer:Ivo

Arrow and pandas interoperability (#3447) Monday ref: 11325694339 Monday ref: 18053173176 Allows coherent interoperability between arrow and pandas input and output format: - Consistent way to read as pandas data written as arrow (all timezones are applied correctly) - Consistent way to read as arrow data written as pandas (moves some of the conversion logic to C++ but behavior wise is unchanged) - Consistent way to combine arrow and pandas data via any of `append`, `update`, `concat`, `finalize`. Note that to combine arrow and pandas, the pandas dataframe needs to have the same column names. Any duplicate columns or unnamed indices will be handled via #3418 column renaming. Reading pandas written series with range index as arrow will now produce a pyarrow array / polars series ```python lib.write("symbol", pd.Series([1, 2, 3])) result = lib.read("symbol", output_format=OutputFormat.PYARROW).data ``` The PR is split in 5 commits for easier review: 1. Extends Arrow/pandas interoperability tests to cover staged writes and finalization. 2. Changes the way Arrow one dimansional structures are normalized to be consistent with pandas. This allows follow up combination easier. 3. Embeds pandas metadata in Arrow normalization metadata. Adds conversion utilites. Moves pandas to arrow conversion in C++ layer so arrow python normalization only needs to understand arrow metadata format. 4. Allows combining arrow and pandas in `combine_schema`. Refactors it so that: - When a mix of arrow and pandas dataframes is given all get converted to arrow with pandas upfront. - Resolve any required name mismatches upfront (so later normalization stages can rely on consistent column name to normalization mappings). 6. Aligns Arrow multi-index field names with pandas’ stored `__idx__` names for append, update, and merge-update. - Note that `stage` and `finalize` don't correctly combine pandas multiindex with arrow. Monday ref: 13124164631 - For `concat` this works without changes. It just renames the stored multiindex levels to unnamed. <details> <summary> Checklist for code changes... </summary> - [ ] Have you updated the relevant docstrings, documentation and copyright notice? - [ ] Is this contribution tested against [all ArcticDB's features](../docs/mkdocs/docs/technical/contributing.md)? - [ ] Do all exceptions introduced raise appropriate [error messages](https://docs.arcticdb.io/error_messages/)? - [ ] Are API changes highlighted in the PR description? - [ ] Is the PR labelled as enhancement or bug so it appears in autogenerated release notes? </details> <!-- Thanks for contributing a Pull Request to ArcticDB! Please ensure you have taken a look at: - ArcticDB's Code of Conduct: https://github.com/man-group/ArcticDB/blob/master/CODE_OF_CONDUCT.md - ArcticDB's Contribution Licensing: https://github.com/man-group/ArcticDB/blob/master/docs/mkdocs/docs/technical/contributing.md#contribution-licensing --> --------- Co-authored-by: Ivo <ivo.dilov@man.com>

Commit:81a2a12
Author:Alex Owens
Committer:Ivo

Enhancement 12844033169: Method to rename indexes and columns to be Arrow compatible

Commit:c841380
Author:Alex Owens
Committer:Ivo

Enhancement 18362804801: Support for round-tripping pyarrow record batches, chunked arrays, and arrays, and polars series (#3399) #### Reference Issues/PRs [9360502457](https://man312219.monday.com/boards/7852509418/pulses/9360502457) #### What does this implement or fix? Adds support for writing `pyarrow.RecordBatch/ChunkedArray/Array` and `polars.Series`. Due to our on-disk slicing, `RecordBatch` always reads back as a `Table`, and `Array` always reads back as a `ChunkedArray`, even if they only have 1 chunk per column. --------- Co-authored-by: Alex Owens <alex.owens@man.com>

Commit:2fbdc26
Author:IvoDD
Committer:Ivo

Arrow and pandas interoperability (#3447) Monday ref: 11325694339 Monday ref: 18053173176 Allows coherent interoperability between arrow and pandas input and output format: - Consistent way to read as pandas data written as arrow (all timezones are applied correctly) - Consistent way to read as arrow data written as pandas (moves some of the conversion logic to C++ but behavior wise is unchanged) - Consistent way to combine arrow and pandas data via any of `append`, `update`, `concat`, `finalize`. Note that to combine arrow and pandas, the pandas dataframe needs to have the same column names. Any duplicate columns or unnamed indices will be handled via #3418 column renaming. Reading pandas written series with range index as arrow will now produce a pyarrow array / polars series ```python lib.write("symbol", pd.Series([1, 2, 3])) result = lib.read("symbol", output_format=OutputFormat.PYARROW).data ``` The PR is split in 5 commits for easier review: 1. Extends Arrow/pandas interoperability tests to cover staged writes and finalization. 2. Changes the way Arrow one dimansional structures are normalized to be consistent with pandas. This allows follow up combination easier. 3. Embeds pandas metadata in Arrow normalization metadata. Adds conversion utilites. Moves pandas to arrow conversion in C++ layer so arrow python normalization only needs to understand arrow metadata format. 4. Allows combining arrow and pandas in `combine_schema`. Refactors it so that: - When a mix of arrow and pandas dataframes is given all get converted to arrow with pandas upfront. - Resolve any required name mismatches upfront (so later normalization stages can rely on consistent column name to normalization mappings). 6. Aligns Arrow multi-index field names with pandas’ stored `__idx__` names for append, update, and merge-update. - Note that `stage` and `finalize` don't correctly combine pandas multiindex with arrow. Monday ref: 13124164631 - For `concat` this works without changes. It just renames the stored multiindex levels to unnamed. <details> <summary> Checklist for code changes... </summary> - [ ] Have you updated the relevant docstrings, documentation and copyright notice? - [ ] Is this contribution tested against [all ArcticDB's features](../docs/mkdocs/docs/technical/contributing.md)? - [ ] Do all exceptions introduced raise appropriate [error messages](https://docs.arcticdb.io/error_messages/)? - [ ] Are API changes highlighted in the PR description? - [ ] Is the PR labelled as enhancement or bug so it appears in autogenerated release notes? </details> <!-- Thanks for contributing a Pull Request to ArcticDB! Please ensure you have taken a look at: - ArcticDB's Code of Conduct: https://github.com/man-group/ArcticDB/blob/master/CODE_OF_CONDUCT.md - ArcticDB's Contribution Licensing: https://github.com/man-group/ArcticDB/blob/master/docs/mkdocs/docs/technical/contributing.md#contribution-licensing --> --------- Co-authored-by: Ivo <ivo.dilov@man.com>

Commit:4c33737
Author:Alex Owens
Committer:Ivo

Enhancement 12844033169: Method to rename indexes and columns to be Arrow compatible

Commit:1ee900a
Author:Alex Owens
Committer:Ivo

Enhancement 18362804801: Support for round-tripping pyarrow record batches, chunked arrays, and arrays, and polars series (#3399) #### Reference Issues/PRs [9360502457](https://man312219.monday.com/boards/7852509418/pulses/9360502457) #### What does this implement or fix? Adds support for writing `pyarrow.RecordBatch/ChunkedArray/Array` and `polars.Series`. Due to our on-disk slicing, `RecordBatch` always reads back as a `Table`, and `Array` always reads back as a `ChunkedArray`, even if they only have 1 chunk per column. --------- Co-authored-by: Alex Owens <alex.owens@man.com>

Commit:76793ad
Author:Ivo
Committer:Ivo

Describe pandas data in Arrow terms before generating Arrow output ExperimentalArrow gains an optional pandas metadata, in a oneof mirroring the top-level input_type one, so that a pandas metadata is embedded, and recovered, by copying the message across. arrow_norm_from_pandas builds that description: the pandas message embedded whole, plus what Arrow keeps separately - whether the first column is the index, whether the object is one-dimensional, a Series' name, and a column metadata entry for every timestamp column carrying its timezone. It lives in its own file rather than in normalization_utils, because stream_descriptor.hpp includes that header for is_pandas_input_type, so a function there taking a descriptor would invert the dependency. Arrow output is now always generated from Arrow metadata. create_python_read_result converts before it builds the frame, for the result and for each node of a recursively normalized read, and _modify_schema does the same, so that the schema _collect_schema reports and the schema a read returns cannot disagree. A test pins that invariant for pandas data with a timezone, which is the case that would have caught the missing second call site. Because the timezone is then already on the Arrow column, Python stops applying timezones altogether: the timezones map, its plumbing through ArrowNormalizationOperations, and the assume_timezone call all go, and ArrowTableNormalizer.denormalize works from one shape, whose pandas part is the embedded metadata. Timezone lookup failures now surface from date::locate_zone, which is where the Windows tzdata message moves to, since that path is what serves every symbol now rather than only Arrow-written ones. pandas_common reaches the embedded metadata, so that a converted metadata behaves like the pandas one it came from - required_fields_info counts the index levels of an embedded multi-index rather than the single index Arrow records, which is what keeps column selection over an unnamed multi-index correct. is_pandas_input_type is the narrow question, asked only by is_inferred_from_empty_pandas, whose answer is about the frame the schema was inferred from rather than about the metadata's shape. The pandas output path keeps its own Arrow branch: pandas metadata cannot express a column timezone, and C++ cannot apply one to a numpy array, so Python lifts the embedded metadata where there is one, synthesizes a pandas view where there is not, and localizes the columns from the Arrow column metadata. That is what the two timezone tests needed.

Commit:b20290c
Author:Petar Markovski
Committer:Alex Owens

Initial implementation

Commit:be7184a
Author:Petar Markovski
Committer:Alex Owens

Initial implementation

Commit:f0510ae
Author:Alex Owens
Committer:Alex Owens

Address Claude's comments

Commit:4cf3a3f
Author:Ivo
Committer:Ivo

Do not trust is_physically_stored for a timestamp index From 6.18.1 until 7.0.0, append and update kept the normalization metadata of the version they modified. A symbol first written empty therefore keeps the empty frame's account of its index once it has rows: - A DatetimeIndex keeps is_physically_stored=False. - A RangeIndex keeps step=0 and the placeholder name "index". Add is_index_physically_stored, which treats a TIMESTAMP index descriptor as stored whatever the flag says. Use it wherever the flag decides the required fields, the datetime-against-range check, Arrow conversion or timezone handling. For a RangeIndex, update_rowrange_norm_for_append adopts the new step when the stored step is 0, rather than assuming 1. PandasName::of_index treats an empty RangeIndex name as unnamed, so that it agrees with the stale placeholder.

Commit:85a9c8c
Author:Ivo
Committer:Ivo

Describe pandas data in Arrow terms before generating Arrow output ExperimentalArrow gains an optional pandas metadata, in a oneof mirroring the top-level input_type one, so that a pandas metadata is embedded, and recovered, by copying the message across. arrow_norm_from_pandas builds that description: the pandas message embedded whole, plus what Arrow keeps separately - whether the first column is the index, whether the object is one-dimensional, a Series' name, and a column metadata entry for every timestamp column carrying its timezone. It lives in its own file rather than in normalization_utils, because stream_descriptor.hpp includes that header for is_pandas_input_type, so a function there taking a descriptor would invert the dependency. Arrow output is now always generated from Arrow metadata. create_python_read_result converts before it builds the frame, for the result and for each node of a recursively normalized read, and _modify_schema does the same, so that the schema _collect_schema reports and the schema a read returns cannot disagree. A test pins that invariant for pandas data with a timezone, which is the case that would have caught the missing second call site. Because the timezone is then already on the Arrow column, Python stops applying timezones altogether: the timezones map, its plumbing through ArrowNormalizationOperations, and the assume_timezone call all go, and ArrowTableNormalizer.denormalize works from one shape, whose pandas part is the embedded metadata. Timezone lookup failures now surface from date::locate_zone, which is where the Windows tzdata message moves to, since that path is what serves every symbol now rather than only Arrow-written ones. pandas_common reaches the embedded metadata, so that a converted metadata behaves like the pandas one it came from - required_fields_info counts the index levels of an embedded multi-index rather than the single index Arrow records, which is what keeps column selection over an unnamed multi-index correct. is_pandas_input_type is the narrow question, asked only by is_inferred_from_empty_pandas, whose answer is about the frame the schema was inferred from rather than about the metadata's shape. The pandas output path keeps its own Arrow branch: pandas metadata cannot express a column timezone, and C++ cannot apply one to a numpy array, so Python lifts the embedded metadata where there is one, synthesizes a pandas view where there is not, and localizes the columns from the Arrow column metadata. That is what the two timezone tests needed.

Commit:88f6d5f
Author:Ivo
Committer:Ivo

Describe pandas data in Arrow terms before generating Arrow output ExperimentalArrow gains an optional pandas metadata, in a oneof mirroring the top-level input_type one, so that a pandas metadata is embedded, and recovered, by copying the message across. arrow_norm_from_pandas builds that description: the pandas message embedded whole, plus what Arrow keeps separately - whether the first column is the index, whether the object is one-dimensional, a Series' name, and a column metadata entry for every timestamp column carrying its timezone. It lives in its own file rather than in normalization_utils, because stream_descriptor.hpp includes that header for is_pandas_input_type, so a function there taking a descriptor would invert the dependency. Arrow output is now always generated from Arrow metadata. create_python_read_result converts before it builds the frame, for the result and for each node of a recursively normalized read, and _modify_schema does the same, so that the schema _collect_schema reports and the schema a read returns cannot disagree. A test pins that invariant for pandas data with a timezone, which is the case that would have caught the missing second call site. Because the timezone is then already on the Arrow column, Python stops applying timezones altogether: the timezones map, its plumbing through ArrowNormalizationOperations, and the assume_timezone call all go, and ArrowTableNormalizer.denormalize works from one shape, whose pandas part is the embedded metadata. Timezone lookup failures now surface from date::locate_zone, which is where the Windows tzdata message moves to, since that path is what serves every symbol now rather than only Arrow-written ones. pandas_common reaches the embedded metadata, so that a converted metadata behaves like the pandas one it came from - required_fields_info counts the index levels of an embedded multi-index rather than the single index Arrow records, which is what keeps column selection over an unnamed multi-index correct. is_pandas_input_type is the narrow question, asked only by is_inferred_from_empty_pandas, whose answer is about the frame the schema was inferred from rather than about the metadata's shape. The pandas output path keeps its own Arrow branch: pandas metadata cannot express a column timezone, and C++ cannot apply one to a numpy array, so Python lifts the embedded metadata where there is one, synthesizes a pandas view where there is not, and localizes the columns from the Arrow column metadata. That is what the two timezone tests needed.

Commit:aeb9fb4
Author:Ivo
Committer:Ivo

Describe pandas data in Arrow terms before generating Arrow output ExperimentalArrow gains an optional pandas metadata, in a oneof mirroring the top-level input_type one, so that a pandas metadata is embedded, and recovered, by copying the message across. arrow_norm_from_pandas builds that description: the pandas message embedded whole, plus what Arrow keeps separately - whether the first column is the index, whether the object is one-dimensional, a Series' name, and a column metadata entry for every timestamp column carrying its timezone. It lives in its own file rather than in normalization_utils, because stream_descriptor.hpp includes that header for is_pandas_input_type, so a function there taking a descriptor would invert the dependency. Arrow output is now always generated from Arrow metadata. create_python_read_result converts before it builds the frame, for the result and for each node of a recursively normalized read, and _modify_schema does the same, so that the schema _collect_schema reports and the schema a read returns cannot disagree. A test pins that invariant for pandas data with a timezone, which is the case that would have caught the missing second call site. Because the timezone is then already on the Arrow column, Python stops applying timezones altogether: the timezones map, its plumbing through ArrowNormalizationOperations, and the assume_timezone call all go, and ArrowTableNormalizer.denormalize works from one shape, whose pandas part is the embedded metadata. Timezone lookup failures now surface from date::locate_zone, which is where the Windows tzdata message moves to, since that path is what serves every symbol now rather than only Arrow-written ones. pandas_common reaches the embedded metadata, so that a converted metadata behaves like the pandas one it came from - required_fields_info counts the index levels of an embedded multi-index rather than the single index Arrow records, which is what keeps column selection over an unnamed multi-index correct. is_pandas_input_type is the narrow question, asked only by is_inferred_from_empty_pandas, whose answer is about the frame the schema was inferred from rather than about the metadata's shape. The pandas output path keeps its own Arrow branch: pandas metadata cannot express a column timezone, and C++ cannot apply one to a numpy array, so Python lifts the embedded metadata where there is one, synthesizes a pandas view where there is not, and localizes the columns from the Arrow column metadata. That is what the two timezone tests needed.

Commit:e89eb6d
Author:Ivo
Committer:Ivo

Describe pandas data in Arrow terms before generating Arrow output ExperimentalArrow gains an optional pandas metadata, in a oneof mirroring the top-level input_type one, so that a pandas metadata is embedded, and recovered, by copying the message across. arrow_norm_from_pandas builds that description: the pandas message embedded whole, plus what Arrow keeps separately - whether the first column is the index, whether the object is one-dimensional, a Series' name, and a column metadata entry for every timestamp column carrying its timezone. It lives in its own file rather than in normalization_utils, because stream_descriptor.hpp includes that header for is_pandas_input_type, so a function there taking a descriptor would invert the dependency. Arrow output is now always generated from Arrow metadata. create_python_read_result converts before it builds the frame, for the result and for each node of a recursively normalized read, and _modify_schema does the same, so that the schema _collect_schema reports and the schema a read returns cannot disagree. A test pins that invariant for pandas data with a timezone, which is the case that would have caught the missing second call site. Because the timezone is then already on the Arrow column, Python stops applying timezones altogether: the timezones map, its plumbing through ArrowNormalizationOperations, and the assume_timezone call all go, and ArrowTableNormalizer.denormalize works from one shape, whose pandas part is the embedded metadata. Timezone lookup failures now surface from date::locate_zone, which is where the Windows tzdata message moves to, since that path is what serves every symbol now rather than only Arrow-written ones. pandas_common reaches the embedded metadata, so that a converted metadata behaves like the pandas one it came from - required_fields_info counts the index levels of an embedded multi-index rather than the single index Arrow records, which is what keeps column selection over an unnamed multi-index correct. is_pandas_input_type is the narrow question, asked only by is_inferred_from_empty_pandas, whose answer is about the frame the schema was inferred from rather than about the metadata's shape. The pandas output path keeps its own Arrow branch: pandas metadata cannot express a column timezone, and C++ cannot apply one to a numpy array, so Python lifts the embedded metadata where there is one, synthesizes a pandas view where there is not, and localizes the columns from the Arrow column metadata. That is what the two timezone tests needed.

Commit:835f730
Author:Alex Owens
Committer:Ivo

Enhancement 18362804801: Support for round-tripping pyarrow record batches, chunked arrays, and arrays, and polars series (#3399) #### Reference Issues/PRs [9360502457](https://man312219.monday.com/boards/7852509418/pulses/9360502457) #### What does this implement or fix? Adds support for writing `pyarrow.RecordBatch/ChunkedArray/Array` and `polars.Series`. Due to our on-disk slicing, `RecordBatch` always reads back as a `Table`, and `Array` always reads back as a `ChunkedArray`, even if they only have 1 chunk per column. --------- Co-authored-by: Alex Owens <alex.owens@man.com>

Commit:f5f0e1f
Author:Ivo

Describe pandas data in Arrow terms before generating Arrow output ExperimentalArrow gains an optional pandas metadata, in a oneof mirroring the top-level input_type one, so that a pandas metadata is embedded, and recovered, by copying the message across. arrow_norm_from_pandas builds that description: the pandas message embedded whole, plus what Arrow keeps separately - whether the first column is the index, whether the object is one-dimensional, a Series' name, and a column metadata entry for every timestamp column carrying its timezone. It lives in its own file rather than in normalization_utils, because stream_descriptor.hpp includes that header for is_pandas_input_type, so a function there taking a descriptor would invert the dependency. Arrow output is now always generated from Arrow metadata. create_python_read_result converts before it builds the frame, for the result and for each node of a recursively normalized read, and _modify_schema does the same, so that the schema _collect_schema reports and the schema a read returns cannot disagree. A test pins that invariant for pandas data with a timezone, which is the case that would have caught the missing second call site. Because the timezone is then already on the Arrow column, Python stops applying timezones altogether: the timezones map, its plumbing through ArrowNormalizationOperations, and the assume_timezone call all go, and ArrowTableNormalizer.denormalize works from one shape, whose pandas part is the embedded metadata. Timezone lookup failures now surface from date::locate_zone, which is where the Windows tzdata message moves to, since that path is what serves every symbol now rather than only Arrow-written ones. pandas_common reaches the embedded metadata, so that a converted metadata behaves like the pandas one it came from - required_fields_info counts the index levels of an embedded multi-index rather than the single index Arrow records, which is what keeps column selection over an unnamed multi-index correct. is_pandas_input_type is the narrow question, asked only by is_inferred_from_empty_pandas, whose answer is about the frame the schema was inferred from rather than about the metadata's shape. The pandas output path keeps its own Arrow branch: pandas metadata cannot express a column timezone, and C++ cannot apply one to a numpy array, so Python lifts the embedded metadata where there is one, synthesizes a pandas view where there is not, and localizes the columns from the Arrow column metadata. That is what the two timezone tests needed.

Commit:75b0de8
Author:Petar Markovski

Add MIN_STR_V1 and MAX_STR_V1 column stat types

Commit:8fbb72c
Author:Alex Owens
Committer:Vasil Pashov

Enhancement 18362804801: Support for round-tripping pyarrow record batches, chunked arrays, and arrays, and polars series (#3399) #### Reference Issues/PRs [9360502457](https://man312219.monday.com/boards/7852509418/pulses/9360502457) #### What does this implement or fix? Adds support for writing `pyarrow.RecordBatch/ChunkedArray/Array` and `polars.Series`. Due to our on-disk slicing, `RecordBatch` always reads back as a `Table`, and `Array` always reads back as a `ChunkedArray`, even if they only have 1 chunk per column. --------- Co-authored-by: Alex Owens <alex.owens@man.com>

Commit:eb4f01f
Author:Alex Owens
Committer:GitHub

Enhancement 18362804801: Support for round-tripping pyarrow record batches, chunked arrays, and arrays, and polars series (#3399) #### Reference Issues/PRs [9360502457](https://man312219.monday.com/boards/7852509418/pulses/9360502457) #### What does this implement or fix? Adds support for writing `pyarrow.RecordBatch/ChunkedArray/Array` and `polars.Series`. Due to our on-disk slicing, `RecordBatch` always reads back as a `Table`, and `Array` always reads back as a `ChunkedArray`, even if they only have 1 chunk per column. --------- Co-authored-by: Alex Owens <alex.owens@man.com>

Commit:af4d690
Author:Alex Owens
Committer:Alex Owens

Enhancement 18362804801: Support for round-tripping pyarrow record batches, chunked arrays, and arrays, and polars series

Commit:73d06bc
Author:Alex Owens
Committer:Alex Owens

Enhancement 18362804801: Support for round-tripping pyarrow record batches, chunked arrays, and arrays, and polars series

Commit:d580e28
Author:Alex Owens

Speculative impl

Commit:9477f6e
Author:Alex Seaton
Committer:Alex Seaton

Col stats: Unify NaN and Null counts in to one ISNULL count

Commit:bb7ef1c
Author:Alex Seaton
Committer:Alex Seaton

Col stats: Unify NaN and Null counts in to one ISNULL count

Commit:558d1b7
Author:Alex Seaton

Col stats: Unify NaN and Null counts in to one ISNULL count

Commit:1004b8f
Author:Alex Owens
Committer:GitHub

Enhancement 11864701485: arrow string formatting norm metadata (#3333) #### Reference Issues/PRs [11864701485](https://man312219.monday.com/boards/7852509418/pulses/11864701485) #### What does this implement or fix? Unless explicitly overridden at the library/read call/column level, string columns written using either pyarrow or polars will return as the same type (small string/large string/categorical). The exception is small string columns written with pyarrow being read back into polars, as polars does not support small strings. These are mapped to large string columns.

Commit:16644c0
Author:Alex Owens
Committer:Alex Owens

Enhancement 11864701485: arrow string formats roundtripped by default

Commit:b2958f0
Author:Alex Owens
Committer:Alex Owens

Enhancement 11864701485: arrow string formats roundtripped by default

Commit:441e170
Author:Alex Owens
Committer:Alex Owens

Enhancement 11864701485: arrow string formats roundtripped by default

Commit:88dbc9d
Author:Alex Owens
Committer:Alex Owens

Enhancement 11864701485: arrow string formats roundtripped by default

Commit:00df4cb
Author:Alex Owens

Move enum

Commit:ac85889
Author:Alex Owens

Add protobuf

Commit:e87e00a
Author:Alex Seaton
Committer:Alex Seaton

Draft: Support a duration type in query builder

Commit:ce5117f
Author:IvoDD
Committer:GitHub

Add arrow and pandas interop normalization tests (#3228) #### Reference Issues/PRs <!--Example: Fixes #1234. See also #3456.--> #### What does this implement or fix? It introduces a comprehensive test suite for expected behavior for arrow interoperability with pandas. This will serve as the base to design arrow normalization metadata. This PR also includes comprehensive pandas normalization metadata tests, which serve to find all usecases of each field of pandas normalization metadata. #### Any other comments? #### Checklist <details> <summary> Checklist for code changes... </summary> - [ ] Have you updated the relevant docstrings, documentation and copyright notice? - [ ] Is this contribution tested against [all ArcticDB's features](../docs/mkdocs/docs/technical/contributing.md)? - [ ] Do all exceptions introduced raise appropriate [error messages](https://docs.arcticdb.io/error_messages/)? - [ ] Are API changes highlighted in the PR description? - [ ] Is the PR labelled as enhancement or bug so it appears in autogenerated release notes? </details> <!-- Thanks for contributing a Pull Request to ArcticDB! Please ensure you have taken a look at: - ArcticDB's Code of Conduct: https://github.com/man-group/ArcticDB/blob/master/CODE_OF_CONDUCT.md - ArcticDB's Contribution Licensing: https://github.com/man-group/ArcticDB/blob/master/docs/mkdocs/docs/technical/contributing.md#contribution-licensing --> --------- Co-authored-by: Ivo <ivo.dilov@man.com>

Commit:c6e2690
Author:Ivo
Committer:Ivo

Address Alex's comments

Commit:2581bd4
Author:Ivo
Committer:Ivo

Add arrow and pandas interop normalization tests

Commit:221db22
Author:Petar Markovski
Committer:GitHub

Add NaN and NULL count to MIN/MAX stats (#3139) ## Add NaN and null counts to MINMAX column stats > [!IMPORTANT] > **The regression in the benchmark commit is EXPECTED and was reviewed by me and Ivo Dilov.** > It happened because the size of the `COLUMN_STATS_KEY` became **~2× bigger** due to the two > new columns added for **`null_count`** and **`nan_count`**. ### Summary Extends the `MINMAX` column statistic to also record per-segment counts of `NaN` (floating point) and null values (NaT timestamps, plus sparse-map gaps such as those from Arrow validity bitmaps), alongside the existing min/max values. ### Changes - **Proto**: Added `NAN_COUNT_V1 = 3` and `NULL_COUNT_V1 = 4` entries to the `ColumnStatsType` enum in `column_stats.proto`. - **Aggregation** (`unsorted_aggregation.{hpp,cpp}`): `MinMaxAggregatorData` now tracks `nan_count_` (floating-point NaNs) and `null_count_` (NaT cells plus sparse-map gaps). For sparse columns, the gap count `last_row()+1 - row_count()` is added to `null_count_` so validity-bitmap nulls that the dense iterator never visits are still counted. `MinMaxAggregator` takes two additional output column names and `finalize` produces 4 output columns (`MIN`, `MAX`, `NAN_COUNT`, `NULL_COUNT`) instead of 2. - **Pipeline** (`column_stats.cpp`): `MINMAX` external stat now maps to the four internal stat types; `v1_NAN_COUNT` / `v1_NULL_COUNT` operator strings and segment column naming wired through. - **Tests**: Added `test_column_stats_nan_and_null_counts` covering multi-segment symbols with mixed NaN, NaT, and valid values across floating-point and timestamp columns. Widened the shared `assert_stats_equal` helper to subselect the received frame to the columns the test cares about (so existing min/max-only tests don't have to spell out all-zero count columns), and updated `test_column_stats_header_metadata` for the 4-entries-per-column header layout.

Commit:5bbc476
Author:Alex Owens
Committer:GitHub

Enhancement 9929831600: arrow timezone support (#3225) #### Reference Issues/PRs [9929831600](https://man312219.monday.com/boards/7852509418/pulses/9929831600) #### What does this implement or fix? Adds support for storing timestamp columns' timezones with data written as Arrow. Behaviour with append/update is as follows: - Static schema - timezone of column in the new table must match that of the existing data exactly (or both be timezone-naive) - Dynamic schema - timezones can differ arbitrarily. If they are different, the result will be timezone-naive --------- Co-authored-by: Alex Owens <alex.owens@man.com>

Commit:0c638aa
Author:Alex Owens
Committer:Alex Owens

Enhancement 9929831600: Arrow timezone support

Commit:c249757
Author:Ivo
Committer:Ivo

Add arrow and pandas interop normalization tests

Commit:5593e10
Author:Petar Markovski
Committer:Petar Markovski

Add NaN and NaT counter to column stats Adds a null counter to column statistics, counting NaN/NaT values, and uses it during query pruning: - Count nulls for null-only column-stats segments. - Prune only-null column-stats slices for == / isin, but keep wholly-null row-slices for != / isnotin instead of pruning them. - C++ and Python regression tests, including Arrow coverage.

Commit:5867f92
Author:Alex Owens
Committer:Alex Owens

Enhancement 9929831600: Arrow timezone support

Commit:c4b6dc7
Author:Alex Owens
Committer:Alex Owens

Enhancement 9929831600: Arrow timezone support

Commit:7030dae
Author:Ivo

Arrow norm meta RFC

Commit:e409207
Author:Alex Owens

WIP tidy up the implementation

Commit:3914d45
Author:Alex Owens

Persist timezones in norm metadata

Commit:32d84fc
Author:Petar Markovski

Add NaN and NaT counter to column stats Adds a null counter to column statistics, counting NaN/NaT values, and uses it during query pruning: - Count nulls for null-only column-stats segments. - Prune only-null column-stats slices for == / isin, but keep wholly-null row-slices for != / isnotin instead of pruning them. - C++ and Python regression tests, including Arrow coverage.

Commit:d1f3b09
Author:Petar Markovski
Committer:Petar Markovski

Add NaN and NaT counter to column stats Adds NaN/NaT/null counting to min/max column stats, counted inline with minmax computation, and fixes column stats pruning silently dropping NaN/NaT/null rows. Includes tests covering null and nan counts across sparse floats and multiple segments.

Commit:ca6e100
Author:Petar Markovski

Add NaN and NaT counter to column stats Adds NaN/NaT/null counting to min/max column stats, counted inline with minmax computation, and fixes column stats pruning silently dropping NaN/NaT/null rows. Includes tests covering null and nan counts across sparse floats and multiple segments.

Commit:640627e
Author:Petar Markovski

change nat_count to null_count

Commit:a1d379a
Author:Petar Markovski

add nan and nat count to min and max stats

Commit:b56df48
Author:IvoDD
Committer:GitHub

Polars write support and change index_column to bool (#3037) #### Reference Issues/PRs Monday ref: 18362798253 #### What does this implement or fix? Two main changes: #### 1. `index_column` change from `str` to `bool` Changes the `index_column` argument from `str` to `bool` meaning we now only accept the first column to be the timeseries index. This is NOT a breaking change because it only affects api strictly labeled as experimental. This is done for two reaons: - Simplifies drastically denormalization logic. We don't have to worry about where to put the index column after schema changes. - It is future proof. We can re-enable the custom index column position in the future without being a braking change. The `index_column` arg could just start accepting strings alongside bools. We would not be able to change our minds in the opisite direction. Change involves: - Modification of experimental protobuf. This is ok as arrow writes are experimental - Removal of all column reordering logic - Updates tests and docs #### 2. Accept `polars.DataFrame`s to write like methods if arrow input is enabled Make `ArrowTableNormalizer` work with `polars.DataFrame`s. This allows all write methods to work with polars input. Currently this works by converting `polars.DataFrame` to `pyarrow.Table` and works the same as pyarrow tables going forward. In the future this will be better done via the `PyCapsule` polars interface, when sparrow fully supports it. Change invloves: - Normalizer accepting polars dataframes - update with date_range truncation of polars dataframes via pyarrow (was benchmarked faster than polars because pyarrow can work with views) - Parametrizing a subset of tests to work with polars (done in a separate commit for easier review) #### Note to reviewers The polars test parametrization is done in a separate commit for easier review. #### Checklist <details> <summary> Checklist for code changes... </summary> - [ ] Have you updated the relevant docstrings, documentation and copyright notice? - [ ] Is this contribution tested against [all ArcticDB's features](../docs/mkdocs/docs/technical/contributing.md)? - [ ] Do all exceptions introduced raise appropriate [error messages](https://docs.arcticdb.io/error_messages/)? - [ ] Are API changes highlighted in the PR description? - [ ] Is the PR labelled as enhancement or bug so it appears in autogenerated release notes? </details> <!-- Thanks for contributing a Pull Request to ArcticDB! Please ensure you have taken a look at: - ArcticDB's Code of Conduct: https://github.com/man-group/ArcticDB/blob/master/CODE_OF_CONDUCT.md - ArcticDB's Contribution Licensing: https://github.com/man-group/ArcticDB/blob/master/docs/mkdocs/docs/technical/contributing.md#contribution-licensing --> --------- Co-authored-by: Ivo <ivo.dilov@man.com>

Commit:7f23711
Author:Alex Seaton
Committer:GitHub

Encode column stats metadata in the segment header rather than column names, and support multi-index (#2990) Monday: 11292562756 11292649800 We believe that no one has created column stats yet as there is no benefit to users. #2958 will start to support using them at read time. We want to make sure that we get the column stats format right on disk before we announce column stats to users and they start to serialize them. This PR is making changes to how we save column stats on disk. It also allows us to create column stats over multi-indexed dataframes, which was not possible before. Global search in Man to check no one ever used `create_column_stats`: https://chat-man.slack.com/archives/CKQBVA96D/p1774986019195379 Column stats information is currently written in `KeyType::COLUMN_STATS` with column names like, `v1.0_min(col_three)`. This is not ideal because at read time we need to parse these string column names to understand what each statistic means. This PR adds a structure in `descriptors.proto`: ``` enum ColumnStatsType { // Older clients reading a new enum value written by a newer client will decode to this first // element, so make sure they don't mis-interpret it as a statistic they understand. // https://protobuf.dev/best-practices/dos-donts/#unspecified-enum COLUMN_STATS_UNKNOWN = 0; // The version numbers here refer to the format of a given statistic. For example we might start // off saving string min and max truncated to 8 bytes in a uint64_t, and later change to saving it truncated // to a different length. That would necessitate a COLUMN_STATS_MIN_V2 so that old readers do not // misinterpret the new statistics format. COLUMN_STATS_MIN_V1 = 1; COLUMN_STATS_MAX_V1 = 2; } message StatEntry { uint32 stats_seg_offset = 1; // offset in to fields in the KeyType::COLUMN_STATS's StreamDescriptor ColumnStatsType type = 2; } message StatEntryList { repeated StatEntry entries = 1; } // Stored in the user defined metadata for KeyType::COLUMN_STATS message ColumnStatsHeader { // This version number refers to the format of this header structure. // Increment the version number if the meaning of the fields changes, to help older clients // avoid mis-interpreting data written by newer clients. uint32 version = 1; // key = data_col_offset (offset in to TimeseriesDescriptor#fields_ in the Index key) map<uint32, StatEntryList> stats_by_column = 2; // end of fields in version 1 } ``` that we save in `KeyType::COLUMN_STATS` header, in the user defined metadata field. This lets us understand the contents of column stats keys without needing to parse a particular string format. We keep meaningful strings as the names of the column stats segment columns for debugging. ### Compat Testing [Manual testing of what happens if this wheel sees unknown stats](https://github.com/man-group/ArcticDB/blob/aseaton-stats-compat-testing/compat_testing_notes.md) --------- Co-authored-by: Alex Seaton <alex.seaton@man.com>

Commit:5c7cd6f
Author:Alex Seaton

Changes for manual compat testing

Commit:5d156df
Author:Alex Seaton
Committer:Alex Seaton

New encoding for column stats, using protos to encode column stats information in the segment header's user metadata field

Commit:271d785
Author:Alex Seaton

Move COLUMN_STATS metadata into segment header Store version info and stat-to-column mappings in a protobuf ColumnStatsHeader packed into the segment metadata, rather than encoding them in column names. Column names simplify from "v1.0_MIN(col)" to "MIN(col)". No backwards compatibility needed as there are no existing COLUMN_STATS keys in use.

Commit:61ffd0b
Author:Alex Owens
Committer:GitHub

Update copyright notices to 2026 (#2848) #### What does this implement or fix? Update all years in copyright notices to 2026. Done mechanically by running: `find . -type f -exec sed -i 's/Copyright 2023 Man Group Operations Limited/Copyright 2026 Man Group Operations Limited/' {} \;` for 2023/4/5 in `cpp/arcticdb`, `cpp/proto` and `python` directories.

Commit:85d59a7
Author:Alex Owens
Committer:GitHub

Enhancement 8987170104: arrow write numeric dense data (#2663) #### Reference Issues/PRs [8987170104](https://man312219.monday.com/boards/7852509418/pulses/8987170104) #### What does this implement or fix? Allows writing of `pyarrow.Table` objects provided they contain only dense numeric data. This is by no means the final API and so must currently be opted into by calling `_set_allow_arrow_input()` on `NativeVersionStore`. Everything is zero copy as far as possible, with the unavoidable exception of bool columns, as our internal representation uses 1 byte per value, whereas Arrow uses 1 bit per value. Performance is pretty much identical to Pandas in comparable use cases, with the exception of bool columns for the reason stated above, and wide, shallow dataframes, which are a bit faster to write with Arrow (presumably because of normalization stuff happening in the Python layer with Pandas that we can skip with Arrow). #### Any other comments? In addition to not supporting strings or sparse data, there are still a lot of improvements to be made: - `update` with a `date_range` overlapping the provided table index not currently supported - Converting from Arrow's packed bitset bool column representation to our internal format is not parallelised - Non-nanosecond timestamp columns can be written, but are returned as nanosecond timestamp columns - Non-UTC timestamp columns also not handled correctly - The index column (if specified) is only guaranteed to be at the correct index when reading all data, not when selecting specific columns or performing processing - Only `pyarrow.Table` is supported right now, not lower level primitives such as `RecordBatch/Array/ChunkedArray` - On read, processing pipeline operations on data written as Arrow probably won't work - `get_info/get_description` output will not be accurate for data written as Arrow - No testing of appending/updating Arrow data with Pandas data and vice-versa, so probably doesn't work - Index column must always be specified for timeseries data, even when it could be inferred in `append/update` calls

Commit:5ee70c6
Author:Alex Owens
Committer:Alex Owens

Enhancement 8987170104: Write Arrow Tables consisting only of dense numeric data

Commit:af86a3e
Author:Alex Owens
Committer:Alex Owens

Address Ivo's comments

Commit:5fbadb0
Author:Alex Owens
Committer:Alex Owens

Enhancement 8987170104: Write Arrow Tables consisting only of dense numeric data

Commit:48fa022
Author:Alex Owens
Committer:Alex Owens

Enhancement 8987170104: Write Arrow Tables consisting only of dense numeric data

Commit:8c7791b
Author:Alex Owens
Committer:Alex Owens

Enhancement 8987170104: Write Arrow Tables consisting only of dense numeric data

Commit:e8058fb
Author:Alex Owens
Committer:Alex Owens

Enhancement 8987170104: Write Arrow Tables consisting only of dense numeric data

Commit:c14f5b6
Author:Alex Owens
Committer:Alex Owens

Enhancement 8987170104: Write Arrow Tables consisting only of dense numeric data

Commit:4baa61e
Author:Alex Owens

Specifiying index column wip

Commit:0b6f945
Author:Ognyan Stoimenov
Committer:GitHub

Storage lock increase wait time and add artificial slow writes (#2497) #### Reference Issues/PRs <!--Example: Fixes #1234. See also #3456.--> #### What does this implement or fix? In continuation of https://github.com/man-group/ArcticDB/pull/2359 which was reverted. This is the same but without the check for slow writes, as that was causing the lock to never be taken on inherently slow storages. This is the diff with the reverted PR https://github.com/man-group/ArcticDB/compare/symbol_list_slow_writes_tests_before_revert..symbol_list_slow_writes_tests_after_revert Changelog is similar to #2359: * Storage lock wait increased to 1000ms * ~Locking gives up on slow writes~ * StorageFailureSimulator extended to simulate slow writes * Stress tests (some are removed from #2359) * Refactoring the storage lock share common code #### Any other comments? #### Checklist <details> <summary> Checklist for code changes... </summary> - [ ] Have you updated the relevant docstrings, documentation and copyright notice? - [ ] Is this contribution tested against [all ArcticDB's features](../docs/mkdocs/docs/technical/contributing.md)? - [ ] Do all exceptions introduced raise appropriate [error messages](https://docs.arcticdb.io/error_messages/)? - [ ] Are API changes highlighted in the PR description? - [ ] Is the PR labelled as enhancement or bug so it appears in autogenerated release notes? </details> <!-- Thanks for contributing a Pull Request to ArcticDB! Please ensure you have taken a look at: - ArcticDB's Code of Conduct: https://github.com/man-group/ArcticDB/blob/master/CODE_OF_CONDUCT.md - ArcticDB's Contribution Licensing: https://github.com/man-group/ArcticDB/blob/master/docs/mkdocs/docs/technical/contributing.md#contribution-licensing -->

Commit:d568688
Author:Ognyan Stoimenov
Committer:Ognyan Stoimenov

Symbol list and storage lock improvements (#2359) #### Reference Issues/PRs <!--Example: Fixes #1234. See also #3456.--> #### What does this implement or fix? * Storage lock wait increased to 1000ms * Locking gives up on slow writes * StorageFailureSimulator extended to simulate slow writes * Stress tests #### Any other comments? https://github.com/man-group/ArcticDB/actions/runs/16205330703?pr=2359 proof of all tests passing (except macOS ones which seem problematic) #### Checklist <details> <summary> Checklist for code changes... </summary> - [ ] Have you updated the relevant docstrings, documentation and copyright notice? - [ ] Is this contribution tested against [all ArcticDB's features](../docs/mkdocs/docs/technical/contributing.md)? - [ ] Do all exceptions introduced raise appropriate [error messages](https://docs.arcticdb.io/error_messages/)? - [ ] Are API changes highlighted in the PR description? - [ ] Is the PR labelled as enhancement or bug so it appears in autogenerated release notes? </details> <!-- Thanks for contributing a Pull Request to ArcticDB! Please ensure you have taken a look at: - ArcticDB's Code of Conduct: https://github.com/man-group/ArcticDB/blob/master/CODE_OF_CONDUCT.md - ArcticDB's Contribution Licensing: https://github.com/man-group/ArcticDB/blob/master/docs/mkdocs/docs/technical/contributing.md#contribution-licensing -->

Commit:4ee3d53
Author:Ognyan Stoimenov

Merge branch 'master' of github.com:man-group/ArcticDB into symbol_list_slow_writes_tests

Commit:ba64bc1
Author:Ognyan Stoimenov

Merge branch 'master' of github.com:man-group/ArcticDB into symbol_list_slow_writes_tests_after_revert

Commit:495f12e
Author:IvoDD
Committer:GitHub

Arrow output normalization (#2488) Makes sure arrow output is consistent with pandas normalization metadata. Change includes: - properly constructing timestamp type in C++ layer - Introduction of ArrowTableNormalizer - Handles index column timezones - Handles index column renames - Constructs `pyarrow.Schema.pandas_metadata` which `pyarrow` can then use to reconstruct the pandas dataframe. - Arrow normalization tests - Simplifies testing logic for existing tests by allowing to reuse `assert_frame_equal` - Pyarrow requires [extra installation of timezone data](https://arrow.apache.org/docs/cpp/build_system.html#download-timezone-database) on Windows. Adds github action step to download it. #### Reference Issues/PRs Monday ref: 8987164697 #### What does this implement or fix? #### Any other comments? #### Checklist <details> <summary> Checklist for code changes... </summary> - [ ] Have you updated the relevant docstrings, documentation and copyright notice? - [ ] Is this contribution tested against [all ArcticDB's features](../docs/mkdocs/docs/technical/contributing.md)? - [ ] Do all exceptions introduced raise appropriate [error messages](https://docs.arcticdb.io/error_messages/)? - [ ] Are API changes highlighted in the PR description? - [ ] Is the PR labelled as enhancement or bug so it appears in autogenerated release notes? </details> <!-- Thanks for contributing a Pull Request to ArcticDB! Please ensure you have taken a look at: - ArcticDB's Code of Conduct: https://github.com/man-group/ArcticDB/blob/master/CODE_OF_CONDUCT.md - ArcticDB's Contribution Licensing: https://github.com/man-group/ArcticDB/blob/master/docs/mkdocs/docs/technical/contributing.md#contribution-licensing -->

Commit:096ab25
Author:Ivo Dilov
Committer:Ivo Dilov

Arrow output normalization Makes sure arrow output is consistent with pandas normalization metadata. Change includes: - properly constructing timestamp type in C++ layer - Introduction of ArrowTableNormalizer - Handles index column timezones - Handles index column renames - Constructs `pyarrow.Schema.pandas_metadata` which `pyarrow` can then use to reconstruct the pandas dataframe. - Arrow normalization tests - Simplifies testing logic for existing tests by allowing to reuse `assert_frame_equal`

Commit:7754a6d
Author:Ivo Dilov
Committer:Ivo Dilov

Arrow output normalization Makes sure arrow output is consistent with pandas normalization metadata. Change includes: - properly constructing timestamp type in C++ layer - Introduction of ArrowTableNormalizer - Handles index column timezones - Handles index column renames - Constructs `pyarrow.Schema.pandas_metadata` which `pyarrow` can then use to reconstruct the pandas dataframe. - Arrow normalization tests - Simplifies testing logic for existing tests by allowing to reuse `assert_frame_equal`

Commit:1580235
Author:Ivo Dilov

Arrow output normalization Makes sure arrow output is consistent with pandas normalization metadata. Change includes: - properly constructing timestamp type in C++ layer - Introduction of ArrowTableNormalizer - Handles index column timezones - Handles index column renames - Constructs `pyarrow.Schema.pandas_metadata` which `pyarrow` can then use to reconstruct the pandas dataframe. - Arrow normalization tests - Simplifies testing logic for existing tests by allowing to reuse `assert_frame_equal`

Commit:f857c0d
Author:Ivo Dilov
Committer:Ivo Dilov

Read arrow frontend

Commit:67a0f02
Author:Alex Owens
Committer:GitHub

Revert "Symbol list and storage lock improvements (#2359)" (#2467) This reverts commit 7a5f09b75adfd8ff4d34f8c6501db7d5bd25e35b (merge of #2359) due to issues seen in real S3 storage tests.

Commit:30627b3
Author:Alex Owens

Revert "Symbol list and storage lock improvements (#2359)" This reverts commit 7a5f09b75adfd8ff4d34f8c6501db7d5bd25e35b.

Commit:88e9285
Author:Ivo Dilov
Committer:Ivo Dilov

Read arrow frontend

Commit:7a5f09b
Author:Ognyan Stoimenov
Committer:GitHub

Symbol list and storage lock improvements (#2359) #### Reference Issues/PRs <!--Example: Fixes #1234. See also #3456.--> #### What does this implement or fix? * Storage lock wait increased to 1000ms * Locking gives up on slow writes * StorageFailureSimulator extended to simulate slow writes * Stress tests #### Any other comments? https://github.com/man-group/ArcticDB/actions/runs/16205330703?pr=2359 proof of all tests passing (except macOS ones which seem problematic) #### Checklist <details> <summary> Checklist for code changes... </summary> - [ ] Have you updated the relevant docstrings, documentation and copyright notice? - [ ] Is this contribution tested against [all ArcticDB's features](../docs/mkdocs/docs/technical/contributing.md)? - [ ] Do all exceptions introduced raise appropriate [error messages](https://docs.arcticdb.io/error_messages/)? - [ ] Are API changes highlighted in the PR description? - [ ] Is the PR labelled as enhancement or bug so it appears in autogenerated release notes? </details> <!-- Thanks for contributing a Pull Request to ArcticDB! Please ensure you have taken a look at: - ArcticDB's Code of Conduct: https://github.com/man-group/ArcticDB/blob/master/CODE_OF_CONDUCT.md - ArcticDB's Contribution Licensing: https://github.com/man-group/ArcticDB/blob/master/docs/mkdocs/docs/technical/contributing.md#contribution-licensing -->

Commit:b296888
Author:Ognyan Stoimenov
Committer:Ognyan Stoimenov

Temp, dont merge

Commit:1f46e3b
Author:William Dealtry
Committer:William Dealtry

Add some lightweight encodings

Commit:f5eb08c
Author:William Dealtry
Committer:William Dealtry

Add some lightweight encodings

Commit:8d6e952
Author:Ognyan Stoimenov
Committer:Ognyan Stoimenov

Temp, dont merge

Commit:1ed0643
Author:William Dealtry
Committer:William Dealtry

Add some lightweight encodings

Commit:ecb7bdb
Author:Ognyan Stoimenov

Temp, dont merge