package spark.spark_native_executors

Get desktop application:
View/edit binary Protocol Buffers messages

How a scheduler sends a stage to its executors, inside the task plans Ballista ships. The scheduler and the executors come from one build, so this needs no version check of its own.

• bytes block = 1

  The block every partition runs. Empty for a stage that reads files, where each partition runs its own copy, which the partition descriptor carries.

• repeated spark_operator.SparkStructField output_schema = 2

• repeated bool broadcast = 3

  Whether every partition reads all of each input, one entry per input.

• map<string, string> conf = 4

A query submitted by a Spark driver to a Comet Native Executor cluster, as the command of an Arrow Flight GetFlightInfo request.

• string comet_version = 1

  The Comet version and proto fingerprint of the client's build. The scheduler requires both to match its own.

• string proto_fingerprint = 2

• map<string, string> conf = 3

  Spark and Comet configs that native execution reads, as a Spark task would pass them.

• repeated Stage stages = 4

  Stages in topological order: every stage's producers come before it. The last stage is the root, and its output is the query result.

Used in: QueryRequest

• optional spark_operator.Operator plan = 1

  The stage's native block. Its input leaves (Scan and ShuffleScan) read the stage's inputs, matched in the order a depth-first walk of the block meets them.

• repeated StageInput inputs = 2

• repeated bytes partition_plans = 3

  For a stage that reads files: the block with each partition's files injected, one serialized Operator per partition. Empty for a stage whose partitions all run `plan`.

• uint32 num_partitions = 4

  The number of partitions of a stage that reads no exchange: the length of partition_plans for a stage that reads files, or otherwise the block's own, such as a range's number of slices. A stage that reads exchanges has as many partitions as they deliver.

• repeated spark_operator.SparkStructField output_schema = 5

  The block's output as Spark declares it. Each partition's output is cast to it before it leaves the stage, as a Spark task's shuffle writer does, because a hash exchange places rows by the types of their values too.

Used in: Stage

• uint32 producer = 1

  Index into QueryRequest.stages of the stage producing this input.

• optional spark_partitioning.Partitioning partitioning = 2

  How the producer's output is partitioned for this stage. Unset for a broadcast.

• bool broadcast = 3

  Every partition of this stage reads the producer's whole output.