package spark.spark_partitioning

Get desktop application:
View/edit binary Protocol Buffers messages

Used in: RangePartition

• repeated spark_expression.Expr partition_bounds = 1

Used in: Partitioning

• repeated spark_expression.Expr hash_expression = 1

• int32 num_partitions = 2

The basic message representing a Spark partitioning.

Used in: spark_operator.ShuffleWriter

• oneof partitioning_struct

  • HashPartition hash_partition = 1

  • SinglePartition single_partition = 2

  • RangePartition range_partition = 3

  • RoundRobinPartition round_robin_partition = 4

Used in: Partitioning

• repeated spark_expression.Expr sort_orders = 1

• int32 num_partitions = 2

• repeated BoundaryRow boundary_rows = 4

Used in: Partitioning

• int32 num_partitions = 1

• int32 max_hash_columns = 2

  Maximum number of columns to hash. 0 means no limit (hash all columns).

• bool positional = 3

  When true, place rows by position rather than by hashing their contents: the row at task-global ordinal i goes to (positional_start_partition + i / positional_group_rows) % num_partitions. Only set where the driver has established that the map task replays its rows in the same order; see "Round Robin Partitioning" in the contributor guide's native_shuffle.md.

• int32 positional_group_rows = 4

  Rows per contiguous group under positional placement. Always positive: the driver resolves the default once, so that a retried task cannot derive a different one.

• int32 positional_start_partition = 5

  Output partition this map task's first group goes to, set per task from `CometShuffleExchangeExec.positionalStartPartition`.

Used in: Partitioning

(message has no fields)