package lance.file.v2

Get desktop application:
View/edit binary Protocol Buffers messages

Each column has a metadata block that is placed at the end of the file. These may be read individually to allow for column projection.

• optional Encoding encoding = 1

  Encoding information about the column itself. This typically describes how to interpret the column metadata buffers. For example, it could describe how statistics or dictionaries are stored in the column metadata.

• repeated ColumnMetadata.Page pages = 2

  The pages in the column

• repeated uint64 buffer_offsets = 3

  The file offsets of each of the column metadata buffers There may be zero buffers.

• repeated uint64 buffer_sizes = 4

  The size (in bytes) of each of the column metadata buffers This field will have the same length as `buffer_offsets` and may be empty.

This describes a page of column data.

Used in: ColumnMetadata

• repeated uint64 buffer_offsets = 1

  The file offsets for each of the page buffers The number of buffers is variable and depends on the encoding. There may be zero buffers (e.g. constant encoded data) in which case this could be empty.

• repeated uint64 buffer_sizes = 2

  The size (in bytes) of each of the page buffers This field will have the same length as `buffer_offsets` and may be empty.

• uint64 length = 3

  Logical length (e.g. # rows) of the page

• optional Encoding encoding = 4

  The encoding used to encode the page

• uint64 priority = 5

  The priority of the page For tabular data this will be the top-level row number of the first row in the page (and top-level rows should not split across pages).

The deferred encoding is used to place the encoding itself in a different part of the file. This is most commonly used to allow encodings to be shared across different columns. For example, when writing a file with thousands of columns, where many pages have the exact same encoding, it can be useful to cut down on the size of the metadata by using a deferred encoding.

Used in: Encoding

• uint64 buffer_location = 1

  Location of the buffer containing the encoding. * If sharing encodings across columns then this will be in a global buffer * If sharing encodings across pages within a column this could be in a column metadata buffer. * This could also be a page buffer if the encoding is not shared, needs to be written before the file ends, and the encoding is too large to load unless we first determine the page needs to be read. This combination seems unusual.

• uint64 buffer_length = 2

The encoding is placed directly in the metadata section

Used in: Encoding

• bytes encoding = 1

  The bytes that make up the encoding embedded directly in the metadata This is the most common approach.

An encoding stores the information needed to decode a column or page For example, it could describe if the page is using bit packing, and how many bits there are in each individual value. At the column level it can be used to wrap columns with dictionaries or statistics.

Used in: ColumnMetadata, ColumnMetadata.Page

• oneof location

  • DeferredEncoding indirect = 1

    The encoding is stored elsewhere and not part of this protobuf message

  • DirectEncoding direct = 2

    The encoding is stored within this protobuf message

  • google.protobuf.Empty none = 3

    There is no encoding information