package arcticc.pb2.encoding_pb2

Get desktop application:
View/edit binary Protocol Buffers messages

Used in: NDArrayEncodedField

• uint32 in_bytes = 1

  number of bytes before any codec is applied

• uint32 out_bytes = 2

  number of bytes outputted by the last codec

• uint64 hash = 3

  Hash value used to verify data integrity

• uint32 encoder_version = 4

• optional VariantCodec codec = 5

Used in: EncodedField

• optional NDArrayEncodedField values = 1

  The use case is to encode and array of type T where T is an object with size >> 4 bytes into 2 arrays, one containing unique values of T, and a second with the position of the unique values. If T has a low cardinality, this yields a good space reduction. l contains the unique T values, and labels contains the index of T. values.item_count == len(unique(array_T)) positions.item_count == array_T.shape[0] This can be used to represent single field of a pandas multindex.

  aka level in pandas multi index

• optional NDArrayEncodedField positions = 2

  aka label in pandas multi index

Used in: SegmentHeader

• oneof encoding

  Captures the variant field encoding

  • NDArrayEncodedField ndarray = 2

  • DictionaryEncodedField dictionary = 3

• uint32 offset = 4

• uint32 num_elements = 5

• optional FieldStats stats = 6

Used in: EncodedField

• uint64 min = 1

• uint64 max = 2

• uint32 unique_count = 3

• bool sorted = 4

• uint32 set = 5

• FieldStats.UniqueCountPrecision unique_count_precision = 6

Used in: FieldStats

• PRECISE = 0

• HYPERLOGLOG = 1

Used in: DictionaryEncodedField, EncodedField

• uint32 items_count = 1

• repeated Block shapes = 2

• repeated Block values = 3

• uint32 sparse_map_bytes = 4

Storage block that describes the encoding for multiple fields for a certain period of time.

• uint64 start_ts = 1

  min(timestamps)

• uint64 end_ts = 2

  max(timestamps)

• optional descriptors_pb2.StreamDescriptor stream_descriptor = 3

• repeated EncodedField fields = 5

  encoding description of the various fields

• optional EncodedField metadata_field = 7

  optional metadata encoding

• optional EncodedField string_pool_field = 8

  string pool

• bool compacted = 9

  is the segment the result of a compaction

• uint32 encoding_version = 13

• ROWCOUNT = 0

• XX_HASH = 1

Used in: Block, mapped_file_storage_pb2.Config

• oneof codec

  • VariantCodec.Zstd zstd = 16

  • VariantCodec.TurboPfor tp4 = 17

  • VariantCodec.Lz4 lz4 = 18

  • VariantCodec.Passthrough passthrough = 19

Used in: VariantCodec

• int32 acceleration = 1

Used in: VariantCodec

• bool mark = 1

Used in: VariantCodec

• TurboPfor.SubCodecs sub_codec = 1

Used in: TurboPfor

• UNKNOWN = 0

• P4 = 16

• P4_DELTA = 17

• P4_DELTA_RLE = 18

• P4_ZZ = 20

• FP_DELTA = 32

  fpp

• FP_DELTA2_ZZ = 33

  fpzz

• FP_GORILLA_RLE = 34

  fpg

• FP_ZZ = 36

  bvz

• FP_ZZ_DELTA = 40

  bvz

Used in: VariantCodec

• int32 level = 1

  See https://github.com/facebook/zstd

  from -20 to 20

• bool is_streaming = 2