package arcticc.pb2.descriptors_pb2

Get desktop application:
View/edit binary Protocol Buffers messages

Used in: FrameMetadata, TimeSeriesDescriptor

• oneof id

  • string string_id = 1

  • int64 numeric_id = 2

• uint64 version_id = 3

• uint64 creation_ts = 4

• uint64 content_hash = 5

• oneof index_start

  • string string_start = 6

  • int64 numeric_start = 7

• oneof index_end

  • string string_end = 8

  • int64 numeric_end = 9

• KeyType key_type = 10

Used in: TimeSeriesDescriptor

• repeated uint32 group_ends = 1

  exclusive boundary of the group, ie for 2 elements: [0,group_ends[0][, [group_ends[0], group_ends[1][

  not used

• uint32 num_buckets = 2

  not used

• bool enabled = 3

• optional NormalizationMetadata normalization = 1

• optional UserDefinedMetadata user_meta = 2

• optional AtomKey next_key = 3

• optional UserDefinedMetadata multi_key_meta = 4

Used in: StreamDescriptor, mapped_file_storage_pb2.Config

• uint32 field_count = 1

• IndexDescriptor.Type kind = 2

Used in: IndexDescriptor

• UNKNOWN = 0

• EMPTY = 69

  Used then the dataframe has 0 rows. Convertible to any other type

  'E'

• ROWCOUNT = 82

  'R'

• STRING = 83

  'S'

• TIMESTAMP = 84

  'T'

• TIME_SYMBOL = 85

  'U'

Used in: AtomKey

• STREAM_GROUP = 0

• GENERATION = 1

• TABLE_DATA = 2

• TABLE_INDEX = 3

• VERSION = 4

• VERSION_JOURNAL = 5

• METRICS = 6

• SNAPSHOT = 7

• SYMBOL_LIST = 8

• VERSION_REF = 9

• STORAGE_INFO = 10

• APPEND_REF = 11

• MULTI_KEY = 12

• LOCK = 13

• SNAPSHOT_REF = 14

• TOMBSTONE = 15

• APPEND_DATA = 16

• LOG = 17

• PARTITION = 18

• UNDEFINED = 19

(message has no fields)

• UNSUPPORTED = 0

• PD_TIMESTAMP = 32

• PY_DATETIME = 33

• PY_TIMEDELTA = 34

• PY_PICKLE_2 = 102

• PY_PICKLE_3 = 103

* This type represents all the additional info required to reconstruct the object that has been written. The modelling reflects the extra info necessary to cater for the input specific data. For now focused on pandas, but support for any other type without a 1:1 mapping to the first class StreamDescriptor model should be handled through normalization and appropriate metadata additions here.

Used in: FrameMetadata, TimeSeriesDescriptor

• oneof input_type

  • NormalizationMetadata.PandasDataFrame df = 1

  • NormalizationMetadata.PandasDataFrame series = 2

  • NormalizationMetadata.NormalisedTimeSeries ts = 3

  • NormalizationMetadata.MsgPackFrame msg_pack_frame = 4

  • NormalizationMetadata.NdArray np = 5

• optional NormalizationMetadata.CustomNormalizerMeta custom = 64

Used in: NormalizationMetadata

• string class_name = 1

• bytes meta = 2

• bool py_datetime_marker = 3

• bool daterange_has_datetime_marker = 4

Used in: NormalizationMetadata

• uint32 version = 1

  To set at least one field

• uint64 size_bytes = 2

Used in: NormalizationMetadata

• repeated uint64 shape = 1

Used in: NormalizationMetadata

• bool mark = 1

• optional Pandas common = 2

• optional PandasMultiColumn multi_columns = 3

• bool has_synthetic_columns = 4

* This represents the common metadata for pandas types which are shared between series and dataframes.

Used in: NormalisedTimeSeries, PandasDataFrame

• string name = 1

  Pandas series support arbitrary objects but restricting to a string seems reasonable

• bool has_name = 9

  Backwards-compatible way of treating name as optional

• oneof index_type

  • PandasIndex index = 2

  • PandasMultiIndex multi_index = 3

• bool mark = 4

  only here to have one field to set if nothing else is necessary

• map<string, Pandas.Categories> categories = 5

• map<string, Pandas.IntCategories> int_categories = 6

• map<string, Pandas.ColumnName> col_names = 7

• optional PandasIndex columns = 8

  Note that 9 is taken above by has_name, so the next added field should be 10

Used in: Pandas

• repeated string category = 1

Used in: Pandas

• bool is_none = 1

• bool is_empty = 2

• string original_name = 3

• bool is_int = 4

This shouldn't be used much

Used in: Pandas

• repeated uint64 category = 1

Used in: NormalizationMetadata

• optional Pandas common = 1

• optional PandasMultiColumn multi_columns = 2

• bool has_synthetic_columns = 3

Used in: Pandas

• string name = 1

  RangeIndex are not represented as a field, hence no name is associated

• string tz = 2

• bool fake_name = 3

• bool is_physically_stored = 4

  True if the index is stored on disk. False if it can be reconstruced from the metadata at read time.

• int64 start = 5

  Used for RangeIndex

• int64 step = 6

  Used for RangeIndex

• bool is_int = 7

mapping the structure of the multi column. unlike the multi index where the data can be stored inside the block

Used in: NormalisedTimeSeries, PandasDataFrame

• repeated string levels = 1

• repeated PandasMultiColumn.Labels labels = 2

• map<uint32, string> timezone = 3

Used in: PandasMultiColumn

• repeated uint64 val = 1

Used in: Pandas

• uint32 version = 1

• map<uint32, string> timezone = 2

  timezone by index field

• uint32 field_count = 3

• string name = 4

  RangeIndex are not represented as a field, hence no name is associated

• string tz = 5

• repeated uint32 fake_field_pos = 6

• bool is_int = 7

Used in: StreamDescriptor

• UNKNOWN = 0

• UNSORTED = 1

• ASCENDING = 2

• DESCENDING = 3

Used in: TimeSeriesDescriptor, encoding_pb2.SegmentHeader

• oneof id

  Channel can output a single type of data possibly containing multiple fields

  • uint64 num_id = 1

  • string str_id = 2

• optional IndexDescriptor index = 3

• repeated StreamDescriptor.FieldDescriptor fields = 4

• uint64 type_hash = 5

• string timezone = 6

  unused but reserved in case first class support is needed

• SortedValue sorted = 7

• uint64 in_bytes = 8

• uint64 out_bytes = 9

• uint64 row_count = 10

Used in: StreamDescriptor

• optional TypeDescriptor type_desc = 1

• string name = 2

• bool enabled = 1

• uint64 total_rows = 1

• optional StreamDescriptor stream_descriptor = 2

• optional NormalizationMetadata normalization = 3

• optional ColumnGroups column_groups = 4

• optional UserDefinedMetadata user_meta = 5

• optional AtomKey next_key = 6

• optional UserDefinedMetadata multi_key_meta = 7

Used in: StreamDescriptor.FieldDescriptor

• TypeDescriptor.ValueType value_type = 1

• TypeDescriptor.SizeBits size_bits = 2

  useful to specify 8/16/32/64 for [u]int, and float{32,64}. Defaults to 0 for unused

• uint32 dimension = 3

  Dimension of a single tick stored 0: scalar 1: array of values 2: matrix ... This maps numpy ndarray concept

Used in: TypeDescriptor

• UNKNOWN_SIZE_BITS = 0

• S8 = 1

• S16 = 2

• S32 = 3

• S64 = 4

Used in: TypeDescriptor

• UNKNOWN_VALUE_TYPE = 0

• UINT = 1

• INT = 2

• FLOAT = 3

• BOOL = 4

• NANOSECONDS_UTC = 5

• SYMBOL = 6

  categorical string of low cardinality suitable for dictionary encoding

• ASCII_STRING = 7

  fixed size string when dim > 1, inputs of type uint8_t, no encoding

• UTF8_STRING = 8

  fixed size string when dim > 1, inputs of type uint8_t, utf8 encoding

• BYTES = 9

  implies fixed size bytes array when dim > 1, opaque

• PICKLE = 10

  BYTES + pickle specific encoding

• DYNAMIC_STRING = 11

  offset reference to deduplicated string pool

• DYNAMIC_UTF8 = 12

• EMPTY = 13

• BOOL_OBJECT = 14

  Nullable boolean

Used in: FrameMetadata, TimeSeriesDescriptor

• UserDefinedMetadata.SerializationType type = 1

• uint32 version = 2

  in case logic needs to be versioned

• oneof storage_type

  • bytes inline_payload = 3

    capped to fixed size in code

Used in: UserDefinedMetadata

• MSG_PACK = 0

  defaults to msg pack