package datafusion_common

Get desktop application:
View/edit binary Protocol Buffers messages

(message has no fields)

Serialized data type

Used in: datafusion.CastNode, datafusion.PartitionColumn, datafusion.PhysicalCastNode, datafusion.PhysicalScalarUdfNode, datafusion.PhysicalTryCastNode, datafusion.PlaceholderNode, datafusion.PrepareNode, datafusion.TryCastNode, Dictionary, Field, ScalarDictionaryValue, ScalarValue

• oneof arrow_type_enum

  • EmptyMessage NONE = 1

    arrow::Type::NA

  • EmptyMessage BOOL = 2

    arrow::Type::BOOL

  • EmptyMessage UINT8 = 3

    arrow::Type::UINT8

  • EmptyMessage INT8 = 4

    arrow::Type::INT8

  • EmptyMessage UINT16 = 5

    represents arrow::Type fields in src/arrow/type.h

  • EmptyMessage INT16 = 6

  • EmptyMessage UINT32 = 7

  • EmptyMessage INT32 = 8

  • EmptyMessage UINT64 = 9

  • EmptyMessage INT64 = 10

  • EmptyMessage FLOAT16 = 11

  • EmptyMessage FLOAT32 = 12

  • EmptyMessage FLOAT64 = 13

  • EmptyMessage UTF8 = 14

  • EmptyMessage UTF8_VIEW = 35

  • EmptyMessage LARGE_UTF8 = 32

  • EmptyMessage BINARY = 15

  • EmptyMessage BINARY_VIEW = 34

  • int32 FIXED_SIZE_BINARY = 16

  • EmptyMessage LARGE_BINARY = 31

  • EmptyMessage DATE32 = 17

  • EmptyMessage DATE64 = 18

  • TimeUnit DURATION = 19

  • Timestamp TIMESTAMP = 20

  • TimeUnit TIME32 = 21

  • TimeUnit TIME64 = 22

  • IntervalUnit INTERVAL = 23

  • Decimal DECIMAL = 24

  • Decimal256Type DECIMAL256 = 36

  • List LIST = 25

  • List LARGE_LIST = 26

  • FixedSizeList FIXED_SIZE_LIST = 27

  • Struct STRUCT = 28

  • Union UNION = 29

  • Dictionary DICTIONARY = 30

  • Map MAP = 33

Used in: datafusion.ListingTableScanNode

(message has no fields)

(message has no fields)

Used in: datafusion.ColumnUnnestListRecursion, datafusion.LogicalExprNode, datafusion.RecursionUnnestOption, datafusion.UnnestNode

• string name = 1

• optional ColumnRelation relation = 2

Used in: Column, DfField

• string relation = 1

Used in: datafusion.PartitionStats, Statistics

• optional Precision min_value = 1

• optional Precision max_value = 2

• optional Precision sum_value = 5

• optional Precision null_count = 3

• optional Precision distinct_count = 4

Used in: CsvOptions, CsvWriterOptions, JsonOptions, JsonWriterOptions

• GZIP = 0

• BZIP2 = 1

• XZ = 2

• ZSTD = 3

• UNCOMPRESSED = 4

Used in: Constraints

• oneof constraint_mode

  • PrimaryKeyConstraint primary_key = 1

  • UniqueConstraint unique = 2

Used in: datafusion.CreateExternalTableNode, datafusion.FileScanExecConf

• repeated Constraint constraints = 1

Used in: datafusion.ListingTableScanNode

• optional CsvOptions options = 5

Options controlling CSV format

Used in: CsvFormat

• bytes has_header = 1

  Indicates if the CSV has a header row

• bytes delimiter = 2

  Delimiter character as a byte

• bytes quote = 3

  Quote character as a byte

• bytes escape = 4

  Optional escape character as a byte

• CompressionTypeVariant compression = 5

  Compression type

• optional uint64 schema_infer_max_rec = 6

  Optional max records for schema inference

• string date_format = 7

  Optional date format

• string datetime_format = 8

  Optional datetime format

• string timestamp_format = 9

  Optional timestamp format

• string timestamp_tz_format = 10

  Optional timestamp with timezone format

• string time_format = 11

  Optional time format

• string null_value = 12

  Optional representation of null value

• string null_regex = 13

  Optional representation of null loading regex

• bytes comment = 14

  Optional comment character as a byte

• bytes double_quote = 15

  Indicates if quotes are doubled

• bytes newlines_in_values = 16

  Indicates if newlines are supported in values

• bytes terminator = 17

  Optional terminator character as a byte

Used in: datafusion.CsvSink

• CompressionTypeVariant compression = 1

  Compression type

• string delimiter = 2

  Optional column delimiter. Defaults to `b','`

• bool has_header = 3

  Whether to write column names as file headers. Defaults to `true`

• string date_format = 4

  Optional date format for date arrays

• string datetime_format = 5

  Optional datetime format for datetime arrays

• string timestamp_format = 6

  Optional timestamp format for timestamp arrays

• string time_format = 7

  Optional time format for time arrays

• string null_value = 8

  Optional value to represent null

• string quote = 9

  Optional quote. Defaults to `b'"'`

• string escape = 10

  Optional escape. Defaults to `'\\'`

• bool double_quote = 11

  Optional flag whether to double quotes, instead of escaping. Defaults to `true`

Used in: ArrowType

• uint32 precision = 3

• int32 scale = 4

Used in: ScalarValue

• bytes value = 1

• int64 p = 2

• int64 s = 3

Used in: ScalarValue

• bytes value = 1

• int64 p = 2

• int64 s = 3

Used in: ArrowType

• uint32 precision = 3

• int32 scale = 4

Used in: DfSchema

• optional Field field = 1

• optional ColumnRelation qualifier = 2

Used in: datafusion.CreateCatalogNode, datafusion.CreateCatalogSchemaNode, datafusion.CreateExternalTableNode, datafusion.DmlNode, datafusion.DropViewNode, datafusion.UnnestNode

• repeated DfField columns = 1

• map<string, string> metadata = 2

Used in: ArrowType

• optional ArrowType key = 1

• optional ArrowType value = 2

Useful for representing an empty enum variant in rust E.G. enum example{One, Two(i32)} maps to message example{ oneof{ EmptyMessage One = 1; i32 Two = 2; } }

Used in: datafusion.PlanType, datafusion.WindowAggExecNode, ArrowType

(message has no fields)

Used in: DfField, FixedSizeList, List, Map, Schema, Struct, Union, UnionField

• string name = 1

  name of the field

• optional ArrowType arrow_type = 2

• bool nullable = 3

• repeated Field children = 4

  for complex data types like structs, unions

• map<string, string> metadata = 5

• int64 dict_id = 6

• bool dict_ordered = 7

Used in: ArrowType

• optional Field field_type = 1

• int32 list_size = 2

Used in: ScalarValue

• int32 days = 1

• int32 milliseconds = 2

Used in: ScalarValue

• int32 months = 1

• int32 days = 2

• int64 nanos = 3

Used in: ArrowType

• YearMonth = 0

• DayTime = 1

• MonthDayNano = 2

Used in: datafusion.JoinNode

• ON = 0

• USING = 1

Used in: datafusion.ColumnIndex

• LEFT_SIDE = 0

• RIGHT_SIDE = 1

• NONE = 2

Used in: datafusion.HashJoinExecNode, datafusion.JoinNode, datafusion.NestedLoopJoinExecNode, datafusion.SymmetricHashJoinExecNode

• INNER = 0

• LEFT = 1

• RIGHT = 2

• FULL = 3

• LEFTSEMI = 4

• LEFTANTI = 5

• RIGHTSEMI = 6

• RIGHTANTI = 7

• LEFTMARK = 8

Options controlling CSV format

Used in: NdJsonFormat

• CompressionTypeVariant compression = 1

  Compression type

• optional uint64 schema_infer_max_rec = 2

  Optional max records for schema inference

Used in: datafusion.JsonSink

• CompressionTypeVariant compression = 1

Used in: ArrowType

• optional Field field_type = 1

Used in: ArrowType

• optional Field field_type = 1

• bool keys_sorted = 2

Used in: datafusion.ListingTableScanNode

• optional JsonOptions options = 1

Used in: ParquetColumnSpecificOptions

• oneof bloom_filter_enabled_opt

  • bool bloom_filter_enabled = 1

• oneof encoding_opt

  • string encoding = 2

• oneof dictionary_enabled_opt

  • bool dictionary_enabled = 3

• oneof compression_opt

  • string compression = 4

• oneof statistics_enabled_opt

  • string statistics_enabled = 5

• oneof bloom_filter_fpp_opt

  • double bloom_filter_fpp = 6

• oneof bloom_filter_ndv_opt

  • uint64 bloom_filter_ndv = 7

• oneof max_statistics_size_opt

  • uint32 max_statistics_size = 8

Used in: TableParquetOptions

• string column_name = 1

• optional ParquetColumnOptions options = 2

Used in: datafusion.ListingTableScanNode

• optional TableParquetOptions options = 2

Used in: TableParquetOptions

• bool enable_page_index = 1

  Regular fields

  default = true

• bool pruning = 2

  default = true

• bool skip_metadata = 3

  default = true

• bool pushdown_filters = 5

  default = false

• bool reorder_filters = 6

  default = false

• uint64 data_pagesize_limit = 7

  default = 1024 * 1024

• uint64 write_batch_size = 8

  default = 1024

• string writer_version = 9

  default = "1.0"

• bool allow_single_file_parallelism = 23

  bool bloom_filter_enabled = 20; // default = false

  default = true

• uint64 maximum_parallel_row_group_writers = 24

  default = 1

• uint64 maximum_buffered_record_batches_per_stream = 25

  default = 2

• bool bloom_filter_on_read = 26

  default = true

• bool bloom_filter_on_write = 27

  default = false

• bool schema_force_view_types = 28

  default = false

• bool binary_as_string = 29

  default = false

• bool skip_arrow_metadata = 30

  default = false

• oneof metadata_size_hint_opt

  • uint64 metadata_size_hint = 4

• oneof compression_opt

  • string compression = 10

• oneof dictionary_enabled_opt

  • bool dictionary_enabled = 11

• oneof statistics_enabled_opt

  • string statistics_enabled = 13

• oneof max_statistics_size_opt

  • uint64 max_statistics_size = 14

• oneof column_index_truncate_length_opt

  • uint64 column_index_truncate_length = 17

• oneof encoding_opt

  • string encoding = 19

• oneof bloom_filter_fpp_opt

  • double bloom_filter_fpp = 21

• oneof bloom_filter_ndv_opt

  • uint64 bloom_filter_ndv = 22

• uint64 dictionary_page_size_limit = 12

• uint64 data_page_row_count_limit = 18

• uint64 max_row_group_size = 15

• string created_by = 16

Used in: ColumnStats, Statistics

• PrecisionInfo precision_info = 1

• optional ScalarValue val = 2

Used in: Precision

• EXACT = 0

• INEXACT = 1

• ABSENT = 2

Used in: Constraint

• repeated uint64 indices = 1

Used in: ScalarValue

• optional ArrowType index_type = 1

• optional ScalarValue value = 2

Used in: ScalarValue

• bytes values = 1

• int32 length = 2

Used for List/FixedSizeList/LargeList/Struct/Map

Used in: ScalarValue

• bytes ipc_message = 1

• bytes arrow_data = 2

• optional Schema schema = 3

• repeated ScalarNestedValue.Dictionary dictionaries = 4

Used in: ScalarNestedValue

• bytes ipc_message = 1

• bytes arrow_data = 2

Used in: ScalarValue

• oneof value

  • int32 time32_second_value = 1

  • int32 time32_millisecond_value = 2

Used in: ScalarValue

• oneof value

  • int64 time64_microsecond_value = 1

  • int64 time64_nanosecond_value = 2

Used in: ScalarValue

• oneof value

  • int64 time_microsecond_value = 1

  • int64 time_nanosecond_value = 2

  • int64 time_second_value = 3

  • int64 time_millisecond_value = 4

• string timezone = 5

Used in: datafusion.LogicalExprNode, datafusion.NamedStructField, datafusion.PartitionedFile, datafusion.PhysicalExprNode, datafusion.WindowFrameBound, Precision, ScalarDictionaryValue, UnionValue

• oneof value

  • ArrowType null_value = 33

    was PrimitiveScalarType null_value = 19; Null value of any type

  • bool bool_value = 1

  • string utf8_value = 2

  • string large_utf8_value = 3

  • string utf8_view_value = 23

  • int32 int8_value = 4

  • int32 int16_value = 5

  • int32 int32_value = 6

  • int64 int64_value = 7

  • uint32 uint8_value = 8

  • uint32 uint16_value = 9

  • uint32 uint32_value = 10

  • uint64 uint64_value = 11

  • float float32_value = 12

  • double float64_value = 13

  • int32 date_32_value = 14

    Literal Date32 value always has a unit of day

  • ScalarTime32Value time32_value = 15

  • ScalarNestedValue large_list_value = 16

  • ScalarNestedValue list_value = 17

  • ScalarNestedValue fixed_size_list_value = 18

  • ScalarNestedValue struct_value = 32

  • ScalarNestedValue map_value = 41

  • Decimal128 decimal128_value = 20

  • Decimal256 decimal256_value = 39

  • int64 date_64_value = 21

  • int32 interval_yearmonth_value = 24

  • int64 duration_second_value = 35

  • int64 duration_millisecond_value = 36

  • int64 duration_microsecond_value = 37

  • int64 duration_nanosecond_value = 38

  • ScalarTimestampValue timestamp_value = 26

  • ScalarDictionaryValue dictionary_value = 27

  • bytes binary_value = 28

  • bytes large_binary_value = 29

  • bytes binary_view_value = 22

  • ScalarTime64Value time64_value = 30

  • IntervalDayTimeValue interval_daytime_value = 25

  • IntervalMonthDayNanoValue interval_month_day_nano = 31

  • ScalarFixedSizeBinary fixed_size_binary_value = 34

  • UnionValue union_value = 42

Used in: datafusion.AggregateExecNode, datafusion.AnalyzeExecNode, datafusion.CsvSinkExecNode, datafusion.CteWorkTableScanNode, datafusion.CustomTableScanNode, datafusion.EmptyExecNode, datafusion.ExplainExecNode, datafusion.FileScanExecConf, datafusion.FileSinkConfig, datafusion.JoinFilter, datafusion.JsonSinkExecNode, datafusion.ListingTableScanNode, datafusion.ParquetSinkExecNode, datafusion.PlaceholderRowExecNode, datafusion.UnnestExecNode, datafusion.ViewTableScanNode, ScalarNestedValue, datafusion_ray.protobuf.DFRayStageReaderExecNode

• repeated Field columns = 1

• map<string, string> metadata = 2

Used in: datafusion.FileScanExecConf, datafusion.PartitionedFile

• optional Precision num_rows = 1

• optional Precision total_byte_size = 2

• repeated ColumnStats column_stats = 3

Used in: ArrowType

• repeated Field sub_field_types = 1

Used in: datafusion.ParquetSink, ParquetFormat

• optional ParquetOptions global = 1

• repeated ParquetColumnSpecificOptions column_specific_options = 2

• map<string, string> key_value_metadata = 3

Used in: ArrowType, Timestamp

• Second = 0

• Millisecond = 1

• Microsecond = 2

• Nanosecond = 3

Used in: ArrowType

• TimeUnit time_unit = 1

• string timezone = 2

Used in: ArrowType

• repeated Field union_types = 1

• UnionMode union_mode = 2

• repeated int32 type_ids = 3

Used in: UnionValue

• int32 field_id = 1

• optional Field field = 2

Used in: Union, UnionValue

• sparse = 0

• dense = 1

Used in: ScalarValue

• int32 value_id = 1

  Note that a null union value must have one or more fields, so we encode a null UnionValue as one with value_id == 128

• optional ScalarValue value = 2

• repeated UnionField fields = 3

• UnionMode mode = 4

Used in: Constraint

• repeated uint64 indices = 1