package impala

Get desktop application:
View/edit binary Protocol Buffers messages

• rpc CancelQueryFInstances (CancelQueryFInstancesRequestPB, CancelQueryFInstancesResponsePB)

  control_service.proto:230

  Called by coordinator to cancel execution of a single query's fragment instances, which the coordinator initiated with a prior call to ExecQueryFInstances. Cancellation is asynchronous (in the sense that this call may return before the fragment instance has completely stopped executing).

  message CancelQueryFInstancesRequestPB

  control_service.proto:176

  • optional UniqueIdPB query_id = 1

    The query id of the query being cancelled.

  message CancelQueryFInstancesResponsePB

  control_service.proto:181

  • optional StatusPB status = 1

• rpc RemoteShutdown (RemoteShutdownParamsPB, RemoteShutdownResultPB)

  control_service.proto:234

  Called to initiate shutdown of this backend.

  message RemoteShutdownParamsPB

  control_service.proto:185

  • optional int64 deadline_s = 1

    Deadline for the shutdown. After this deadline expires (starting at the time when this remote shutdown command is received), the Impala daemon exits immediately regardless of whether queries are still executing.

  message RemoteShutdownResultPB

  control_service.proto:211

  • optional StatusPB status = 1

    Success or failure of the operation.

  • optional ShutdownStatusPB shutdown_status = 2

    If status is OK, additional info about the shutdown status.

• rpc ReportExecStatus (ReportExecStatusRequestPB, ReportExecStatusResponsePB)

  control_service.proto:224

  Update the coordinator with the query status of the backend.

  message ReportExecStatusRequestPB

  control_service.proto:149

  • optional UniqueIdPB query_id = 1

    The query id which this report is for.

  • optional int32 coord_state_idx = 2

    same as TExecQueryFInstancesParams.coord_state_idx

  • repeated FragmentInstanceExecStatusPB instance_exec_status = 3

  • optional int32 thrift_profiles_sidecar_idx = 4

    Sidecar index of the cumulative profiles of all fragment instances in instance_exec_status.

  • optional StatusPB overall_status = 5

    Cumulative status for this backend. See QueryState::overall_status for details.

  • optional UniqueIdPB fragment_instance_id = 6

    The fragment instance id of the first failed fragment instance. This corresponds to the fragment which sets 'overall_status' above. Not set if 'overall_status' is a general error (e.g. failure to start fragment instances).

  message ReportExecStatusResponsePB

  control_service.proto:172

  • optional StatusPB status = 1

Handles data transmission between fragment instances.

• rpc EndDataStream (EndDataStreamRequestPB, EndDataStreamResponsePB)

  data_stream_service.proto:92

  Called by a sender to close the channel between fragment instances.

  message EndDataStreamRequestPB

  data_stream_service.proto:62

  All fields are required in V1.

  • optional UniqueIdPB dest_fragment_instance_id = 1

    The fragment instance id of the receiver.

  • optional int32 sender_id = 2

    Sender instance id, unique within a fragment.

  • optional int32 dest_node_id = 3

    PlanNodeId of the exchange node which owns the receiver.

  message EndDataStreamResponsePB

  data_stream_service.proto:74

  All fields are required in V1.

  • optional StatusPB status = 1

  • optional int64 receiver_latency_ns = 2

    Latency for response in the receiving daemon in nanoseconds.

• rpc TransmitData (TransmitDataRequestPB, TransmitDataResponsePB)

  data_stream_service.proto:89

  Called by sender to transmit a single row batch. Returns error indication if params.fragmentId or params.destNodeId are unknown or if data couldn't be read.

  message TransmitDataRequestPB

  data_stream_service.proto:29

  All fields are required in V1.

  • optional UniqueIdPB dest_fragment_instance_id = 1

    The fragment instance id of the receiver.

  • optional int32 sender_id = 2

    Sender instance id, unique within a fragment.

  • optional int32 dest_node_id = 3

    PlanNodeId of the exchange node which owns the receiver.

  • optional RowBatchHeaderPB row_batch_header = 4

    The header which contains the meta-data of the row batch.

  • optional int32 tuple_offsets_sidecar_idx = 5

    The sidecar index of tuple offsets' buffer which is an array of int32 containing the offsets of tuples into the buffer pointed to by tuple data's sidecar below. There are num_rows * num_tuples_per_row offsets in total. An offset of -1 records a NULL.

  • optional int32 tuple_data_sidecar_idx = 6

    The sidecar index of the tuple's data which is a (compressed) row batch. The details of the row batch (e.g. # of rows) is in 'row_batch_header' above.

  message TransmitDataResponsePB

  data_stream_service.proto:53

  All fields are required in V1.

  • optional StatusPB status = 1

    Status::OK() on success; Error status on failure.

  • optional int64 receiver_latency_ns = 2

    Latency for response in the receiving daemon in nanoseconds.

• rpc Ping (PingRequestPB, PingResponsePB)

  rpc_test.proto:36

  message PingRequestPB

  rpc_test.proto:26

  Definitions for service used for rpc-mgr-test.

  (message has no fields)

  message PingResponsePB

  rpc_test.proto:29

  • required int32 int_response = 1

• rpc ScanMem (ScanMemRequestPB, ScanMemResponsePB)

  rpc_test.proto:50

  message ScanMemRequestPB

  rpc_test.proto:39

  • required int32 pattern = 1

  • required int32 sidecar_idx = 2

  message ScanMemResponsePB

  rpc_test.proto:44

  (message has no fields)

The compression codec. Currently used in row batch's header to indicate the type of compression applied to the row batch.

Used in: RowBatchHeaderPB

• NONE = 0

  No compression.

• LZ4 = 1

The results of a DML statement, sent to the coordinator as part of ReportExecStatusRequestPB

Used in: FragmentInstanceExecStatusPB

• map<string, string> files_to_move = 1

  A map from temporary absolute file path to final absolute destination. The coordinator performs these updates after the query completes.

• map<string, DmlPartitionStatusPB> per_partition_status = 2

  Per-partition details, used in finalization and reporting. The keys represent partitions to create, coded as k1=v1/k2=v2/k3=v3..., with the root's key in an unpartitioned table being ROOT_PARTITION_KEY. The target table name is recorded in the corresponding TQueryExecRequest

Per-partition statistics and metadata resulting from DML statements.

Used in: DmlExecStatusPB

• optional int64 id = 1

  The id of the partition written to (may be -1 if the partition is created by this query). See THdfsTable.partitions.

• optional int64 num_modified_rows = 2

  The number of rows modified in this partition

• optional DmlStatsPB stats = 3

  Detailed statistics gathered by table writers for this partition

• optional string partition_base_dir = 4

  Fully qualified URI to the base directory for this partition.

• optional int64 kudu_latest_observed_ts = 5

  The latest observed Kudu timestamp reported by the local KuduSession. This value is an unsigned int64.

Per partition DML stats TODO: this should include the table stats that we update the metastore with.

Used in: DmlPartitionStatusPB

• optional int64 bytes_written = 1

• optional ParquetDmlStatsPB parquet_stats = 2

• optional KuduDmlStatsPB kudu_stats = 3

Error message exchange format

Used in: StatefulStatusPB

• optional int32 count = 1

  Number of error messages reported using the above identifier

• repeated string messages = 2

  Sample messages from the above error code

Represents the states that a fragment instance goes through during its execution. The current state gets sent back to the coordinator and will be presented to users through the debug webpages. The states are listed in the order to which they are transitioned. Not all states are necessarily transitioned through when there are errors.

Used in: FragmentInstanceExecStatusPB

• WAITING_FOR_EXEC = 0

• WAITING_FOR_PREPARE = 1

• WAITING_FOR_CODEGEN = 2

• WAITING_FOR_OPEN = 3

• WAITING_FOR_FIRST_BATCH = 4

• FIRST_BATCH_PRODUCED = 5

• PRODUCING_DATA = 6

• LAST_BATCH_SENT = 7

• FINISHED = 8

Used in: ReportExecStatusRequestPB

• optional int64 report_seq_no = 1

  Sequence number prevents out-of-order or duplicated updates from being applied.

• optional UniqueIdPB fragment_instance_id = 2

  The ID of the fragment instance which this report contains

• optional bool done = 3

  If true, fragment finished executing.

• optional FInstanceExecStatePB current_state = 4

  The current state of this fragment instance's execution.

• optional DmlExecStatusPB dml_exec_status = 5

  Cumulative structural changes made by the table sink of this fragment instance. This is sent only when 'done' above is true. Not idempotent.

• repeated StatefulStatusPB stateful_report = 6

  The non-idempotent parts of the report, and any prior reports that are not known to have been received by the coordinator.

Used in: DmlStatsPB

• optional int64 num_row_errors = 1

  The number of reported per-row errors, i.e. this many rows were not modified. Note that this aggregate is less useful than a breakdown of the number of errors by error type, e.g. number of rows with duplicate key conflicts, number of rows violating nullability constraints, etc., but it isn't possible yet to differentiate all error types in the KuduTableSink yet.

Used in: DmlStatsPB

• map<string, int64> per_column_size = 1

  For each column, the on disk byte size

The serialized version of a header of a RowBatch (in be/src/runtime/row-batch.h). It contains the meta-data of a row batch. The actual data of a row batch is sent as KRPC sidecars. Please see TransmitDataRequestPB for details. All fields are required in V1.

Used in: TransmitDataRequestPB

• optional int32 num_rows = 1

  Total number of rows contained in this batch.

• optional int32 num_tuples_per_row = 2

  Number of tuples per row in this batch.

• optional int64 uncompressed_size = 3

  Size of 'tuple_data' in bytes before any compression is applied.

• optional CompressionType compression_type = 4

  The compression codec (if any) used for compressing the row batch.

The current status of a shutdown operation.

Used in: RemoteShutdownResultPB

• optional int64 grace_remaining_ms = 1

  Milliseconds remaining in startup grace period. 0 if the period has expired.

• optional int64 deadline_remaining_ms = 2

  Milliseconds remaining in shutdown deadline. 0 if the deadline has expired.

• optional int64 finstances_executing = 3

  Number of fragment instances still executing.

• optional int64 client_requests_registered = 4

  Number of client requests still registered with the Impala server that is being shut down.

• optional int64 backend_queries_executing = 5

  Number of queries still executing on backend.

Represents any part of the status report that isn't idempotent. If the executor thinks the report failed, we'll retransmit these parts, and this allows us to keep them associated with their original sequence number so that if the coordinator actually did receive the original report it won't reapply them.

Used in: FragmentInstanceExecStatusPB

• optional int64 report_seq_no = 1

  Sequence number prevents out-of-order or duplicated updates from being applied. 'report_seq_no' will be <= the 'report_seq_no' in the FragmentInstanceExecStatusPB that contains this StatefulStatusPB.

• map<int32, ErrorLogEntryPB> error_log = 2

  Map of TErrorCode to ErrorLogEntryPB; New errors that have not been reported to the coordinator by this fragment instance. Not idempotent.

Proto-serialized version of Impala's Status object.

Used in: CancelQueryFInstancesResponsePB, EndDataStreamResponsePB, RemoteShutdownResultPB, ReportExecStatusRequestPB, ReportExecStatusResponsePB, TransmitDataResponsePB

• optional int32 status_code = 1

• repeated string error_msgs = 2

128-bit ID (equivalent to TUniqueID).

Used in: CancelQueryFInstancesRequestPB, EndDataStreamRequestPB, FragmentInstanceExecStatusPB, ReportExecStatusRequestPB, TransmitDataRequestPB

• required fixed64 hi = 1

• required fixed64 lo = 2