package pixels.proto

Get desktop application:
View/edit binary Protocol Buffers messages

Used in: ColumnStatistic

• optional sint64 sum = 1

  sum will store the total binary blob length in a stripe

Used in: ColumnStatistic

• repeated uint64 count = 1

ColumnChunk index

Used in: RowGroupIndex

• optional uint64 chunkOffset = 1

• optional uint32 chunkLength = 2

  the number of bytes of this column chunk (including the isNull bitmap) in the storage

• optional uint32 isNullOffset = 3

  the offset of the isNull bitmap within this column chunk

• repeated uint32 pixelPositions = 4

  starting offsets of each pixel in this column chunk

• repeated PixelStatistic pixelStatistics = 5

  the statistics of the pixels in this column chunk

• optional bool littleEndian = 6

  whether this column chunk is stored in little endian

• optional bool nullsPadding = 7

  whether the null positions are padded with arbitrary value, only take effects if non-random-accessible encoding (i.e., run-length) is not used

• optional uint32 isNullAlignment = 8

  the number of bytes the isNullOffset is align to

Used in: RowGroupEncoding

• required ColumnEncoding.Kind kind = 1

• optional uint32 dictionarySize = 2

  the number of distinct elements in the dictionary, only valid when dictionary encoding is in use

• optional ColumnEncoding cascadeEncoding = 3

  the explicit cascade encoding scheme specified by pixels writer

Used in: ColumnEncoding

• NONE = 0

• RUNLENGTH = 1

  pixels adaptively cascades delta encoding on run-length encoding, so there is no explicit delta encoding

• DICTIONARY = 2

  since v0.2.0, dictionary encoding does not cascade other encoding schemes such as run-length by default

  pixels applies bit-packing automatically on all boolean data, so there is no explicit bit-packing encoding

Used in: Footer, PixelStatistic, RowGroupStatistic

• optional uint64 numberOfValues = 1

• optional IntegerStatistic intStatistics = 2

• optional DoubleStatistic doubleStatistics = 3

• optional StringStatistic stringStatistics = 4

• optional BucketStatistic bucketStatistics = 5

• optional BinaryStatistic binaryStatistics = 6

• optional TimestampStatistic timestampStatistics = 7

• optional DateStatistic dateStatistics = 9

  Date and Time types are added in Issue #94, use field number 9 and 10 for compatibility.

• optional TimeStatistic timeStatistics = 10

• optional Integer128Statistic int128Statistics = 11

  Integer128 is added for long decimal in Issue #203.

• optional bool hasNull = 8

Used in: PostScript

• NONE = 0

• ZLIB = 1

• SNAPPY = 2

• LZO = 3

• LZ4 = 4

• ZSTD = 5

Used in: ColumnStatistic

• optional sint32 minimum = 1

  min,max values saved as milliseconds since epoch

• optional sint32 maximum = 2

Used in: ColumnStatistic

• optional double minimum = 1

• optional double maximum = 2

• optional double sum = 3

The content of the file tail that must be serialized and stored at the end of each file, followed by an eight-byte integer in big endian indicating the start offset of the file tail.

• optional Footer footer = 1

• optional PostScript postscript = 2

• optional uint32 footerLength = 3

• optional uint32 postscriptLength = 4

File Footer

Used in: FileTail

• repeated Type types = 1

  data schema of this file

• repeated ColumnStatistic columnStats = 2

  file level column statistics

• repeated RowGroupInformation rowGroupInfos = 3

  row group information

• repeated RowGroupStatistic rowGroupStats = 4

  row group statistics

• optional Type hiddenType = 5

  hidden timestamp column type

• optional ColumnStatistic hiddenColumnStats = 6

  file level hidden timestamp column statistics

Used in: ColumnStatistic

• optional uint64 minimum_high = 1

  high stores the highest 64 bits whereas low stores the lowest 64 bits

• optional uint64 minimum_low = 2

• optional sint64 maximum_high = 3

• optional sint64 maximum_low = 4

Used in: ColumnStatistic

• optional sint64 minimum = 1

• optional sint64 maximum = 2

• optional sint64 sum = 3

The information about a data hash partition. Each row group can be used to store a hash partition of the data in the file.

Used in: RowGroupInformation, stream.proto.StreamRowGroupFooter

• repeated uint32 columnIds = 1

  the id (index in Footer.types) of the columns that are used as the partition key

• optional int32 hashValue = 2

  the hash value of the partition keys in this partition

Pixel statistic statistic: statistic for this pixel

Used in: ColumnChunkIndex

• optional ColumnStatistic statistic = 1

PostScript

Used in: FileTail

• optional uint32 version = 1

  Pixels file version

• optional uint64 contentLength = 2

  file content length (everything except FileTail)

• optional uint32 numberOfRows = 3

  number of rows in the file

• optional CompressionKind compression = 4

  compression kind, currently NOT USED

• optional uint32 compressionBlockSize = 5

  compression block size, currently NOT USED

• optional uint32 pixelStride = 6

  the maximum number of rows in a pixel

• optional string writerTimezone = 7

  the timezone in which the file was written

• optional bool partitioned = 8

  whether this is a hash partitioned file, each row group is a hash partition if it is true

• optional uint32 columnChunkAlignment = 9

  the number of bytes the start offsets of the column chunks are align to

• optional bool hasHiddenColumn = 10

  whether the file contains hidden timestamp columns

• optional string magic = 8000

  it is always "PIXELS", leave this last in the record

Used in: RowGroupFooter, stream.proto.StreamRowGroupFooter

• repeated ColumnEncoding columnChunkEncodings = 1

• optional ColumnEncoding hiddenColumnChunkEncoding = 2

• optional RowGroupIndex rowGroupIndexEntry = 1

• optional RowGroupEncoding rowGroupEncoding = 2

Used in: RowGroupFooter, stream.proto.StreamRowGroupFooter

• repeated ColumnChunkIndex columnChunkIndexEntries = 1

• optional ColumnChunkIndex hiddenColumnChunkIndexEntry = 2

Row group information

Used in: Footer

• optional uint64 footerOffset = 1

  row group start offset

• optional uint32 dataLength = 2

  row group serialized content length

• optional uint32 footerLength = 3

  serialized RowGroupFooter length

• optional uint32 numberOfRows = 4

  number of rows in this row group

• optional PartitionInformation partitionInfo = 5

  the information of the hash partition, if Footer.partitioned is true

Row group statistic

Used in: Footer

• repeated ColumnStatistic columnChunkStats = 1

  statistic for each column chunk that can be used to filter row groups

• optional ColumnStatistic hiddenColumnChunkStats = 2

  statistics for the column chunk of hidden timestamp column that can be used to filter row groups

Used in: ColumnStatistic

• optional string minimum = 1

• optional string maximum = 2

• optional sint64 sum = 3

  sum will store the total length of all strings in a stripe

Used in: ColumnStatistic

• optional sint32 minimum = 1

  min,max values saved as milliseconds since epoch

• optional sint32 maximum = 2

Used in: ColumnStatistic

• optional sint64 minimum = 1

  min,max values saved as milliseconds since epoch

• optional sint64 maximum = 2

Used in: Footer, stream.proto.StreamHeader

• optional Type.Kind kind = 1

• optional string name = 2

• repeated uint32 subtypes = 3

• optional uint32 maximumLength = 4

• optional uint32 precision = 5

• optional uint32 scale = 6

• optional uint32 dimension = 7

Used in: Type

• BOOLEAN = 0

• BYTE = 1

• SHORT = 2

• INT = 3

• LONG = 4

• FLOAT = 5

• DOUBLE = 6

• STRING = 7

• BINARY = 8

• TIMESTAMP = 9

• ARRAY = 10

• MAP = 11

• STRUCT = 12

• VARBINARY = 13

• DECIMAL = 14

• DATE = 15

• VARCHAR = 16

• CHAR = 17

• TIME = 18

• VECTOR = 19