package spark.connect
Get desktop application:
View/edit binary Protocol Buffers messages
Main interface for the SparkConnect service.
rpc ExecutePlan (, stream )
base.proto:1065Executes a request that contains the query and returns a stream of [[Response]]. It is guaranteed that there is at least one ARROW batch returned even if the result set is empty.
message
base.proto:281A request to be executed by the service.
string session_id = 1
(Required) The session_id specifies a spark session for a user id (which is specified by user_context.user_id). The session_id is set by the client to be able to collate streaming responses from different queries within the dedicated session. The id should be an UUID string of the format `00112233-4455-6677-8899-aabbccddeeff`
optional string client_observed_server_side_session_id = 8
(Optional) Server-side generated idempotency key from the previous responses (if any). Server can use this to validate that the server side session has not changed.
optional user_context = 2
(Required) User context user_context.user_id and session+id both identify a unique remote spark session on the server side.
optional string operation_id = 6
(Optional) Provide an id for this request. If not provided, it will be generated by the server. It is returned in every ExecutePlanResponse.operation_id of the ExecutePlan response stream. The id must be an UUID string of the format `00112233-4455-6677-8899-aabbccddeeff`
optional plan = 3
(Required) The logical plan to be executed / analyzed.
optional string client_type = 4
Provides optional information about the client sending the request. This field can be used for language or version specific information and is only intended for logging purposes and will not be interpreted by the server.
repeated request_options = 5
Repeated element for options that can be passed to the request. This element is currently unused but allows to pass in an extension value used for arbitrary options.
repeated string tags = 7
Tags to tag the given execution with. Tags cannot contain ',' character and cannot be empty strings. Used by Interrupt with interrupt.tag.
rpc AnalyzePlan (, )
base.proto:1068Analyzes a query and returns a [[AnalyzeResponse]] containing metadata about the query.
message
base.proto:60Request to perform plan analyze, optionally to explain the plan.
string session_id = 1
(Required) The session_id specifies a spark session for a user id (which is specified by user_context.user_id). The session_id is set by the client to be able to collate streaming responses from different queries within the dedicated session. The id should be an UUID string of the format `00112233-4455-6677-8899-aabbccddeeff`
optional string client_observed_server_side_session_id = 17
(Optional) Server-side generated idempotency key from the previous responses (if any). Server can use this to validate that the server side session has not changed.
optional user_context = 2
(Required) User context
optional string client_type = 3
Provides optional information about the client sending the request. This field can be used for language or version specific information and is only intended for logging purposes and will not be interpreted by the server.
oneof analyze
schema = 4
explain = 5
tree_string = 6
is_local = 7
is_streaming = 8
input_files = 9
spark_version = 10
ddl_parse = 11
same_semantics = 12
semantic_hash = 13
persist = 14
unpersist = 15
get_storage_level = 16
message
base.proto:207Response to performing analysis of the query. Contains relevant metadata to be able to reason about the performance. Next ID: 16
string session_id = 1
string server_side_session_id = 15
Server-side generated idempotency key that the client can use to assert that the server side session has not changed.
oneof result
schema = 2
explain = 3
tree_string = 4
is_local = 5
is_streaming = 6
input_files = 7
spark_version = 8
ddl_parse = 9
same_semantics = 10
semantic_hash = 11
persist = 12
unpersist = 13
get_storage_level = 14
rpc Config (, )
base.proto:1071Update or fetch the configurations and returns a [[ConfigResponse]] containing the result.
message
base.proto:476Request to update or fetch the configurations.
string session_id = 1
(Required) The session_id specifies a spark session for a user id (which is specified by user_context.user_id). The session_id is set by the client to be able to collate streaming responses from different queries within the dedicated session. The id should be an UUID string of the format `00112233-4455-6677-8899-aabbccddeeff`
optional string client_observed_server_side_session_id = 8
(Optional) Server-side generated idempotency key from the previous responses (if any). Server can use this to validate that the server side session has not changed.
optional user_context = 2
(Required) User context
optional operation = 3
(Required) The operation for the config.
optional string client_type = 4
Provides optional information about the client sending the request. This field can be used for language or version specific information and is only intended for logging purposes and will not be interpreted by the server.
message
base.proto:552Response to the config request. Next ID: 5
string session_id = 1
string server_side_session_id = 4
Server-side generated idempotency key that the client can use to assert that the server side session has not changed.
repeated pairs = 2
(Optional) The result key-value pairs. Available when the operation is 'Get', 'GetWithDefault', 'GetOption', 'GetAll'. Also available for the operation 'IsModifiable' with boolean string "true" and "false".
repeated string warnings = 3
(Optional) Warning messages for deprecated or unsupported configurations.
rpc AddArtifacts (stream , )
base.proto:1075Add artifacts to the session and returns a [[AddArtifactsResponse]] containing metadata about the added artifacts.
message
base.proto:571Request to transfer client-local artifacts.
string session_id = 1
(Required) The session_id specifies a spark session for a user id (which is specified by user_context.user_id). The session_id is set by the client to be able to collate streaming responses from different queries within the dedicated session. The id should be an UUID string of the format `00112233-4455-6677-8899-aabbccddeeff`
optional user_context = 2
User context
optional string client_observed_server_side_session_id = 7
(Optional) Server-side generated idempotency key from the previous responses (if any). Server can use this to validate that the server side session has not changed.
optional string client_type = 6
Provides optional information about the client sending the request. This field can be used for language or version specific information and is only intended for logging purposes and will not be interpreted by the server.
oneof payload
The payload is either a batch of artifacts or a partial chunk of a large artifact.
batch = 3
begin_chunk = 4
The metadata and the initial chunk of a large artifact chunked into multiple requests. The server side is notified about the total size of the large artifact as well as the number of chunks to expect.
chunk = 5
A chunk of an artifact excluding metadata. This can be any chunk of a large artifact excluding the first chunk (which is included in `BeginChunkedArtifact`).
message
base.proto:654Response to adding an artifact. Contains relevant metadata to verify successful transfer of artifact(s). Next ID: 4
string session_id = 2
Session id in which the AddArtifact was running.
string server_side_session_id = 3
Server-side generated idempotency key that the client can use to assert that the server side session has not changed.
repeated artifacts = 1
The list of artifact(s) seen by the server.
rpc ArtifactStatus (, )
base.proto:1078Check statuses of artifacts in the session and returns them in a [[ArtifactStatusesResponse]]
message
base.proto:675Request to get current statuses of artifacts at the server side.
string session_id = 1
(Required) The session_id specifies a spark session for a user id (which is specified by user_context.user_id). The session_id is set by the client to be able to collate streaming responses from different queries within the dedicated session. The id should be an UUID string of the format `00112233-4455-6677-8899-aabbccddeeff`
optional string client_observed_server_side_session_id = 5
(Optional) Server-side generated idempotency key from the previous responses (if any). Server can use this to validate that the server side session has not changed.
optional user_context = 2
User context
optional string client_type = 3
Provides optional information about the client sending the request. This field can be used for language or version specific information and is only intended for logging purposes and will not be interpreted by the server.
repeated string names = 4
The name of the artifact is expected in the form of a "Relative Path" that is made up of a sequence of directories and the final file element. Examples of "Relative Path"s: "jars/test.jar", "classes/xyz.class", "abc.xyz", "a/b/X.jar". The server is expected to maintain the hierarchy of files as defined by their name. (i.e The relative path of the file on the server's filesystem will be the same as the name of the provided artifact)
message
base.proto:709Response to checking artifact statuses. Next ID: 4
string session_id = 2
Session id in which the ArtifactStatus was running.
string server_side_session_id = 3
Server-side generated idempotency key that the client can use to assert that the server side session has not changed.
map<string, > statuses = 1
A map of artifact names to their statuses.
rpc Interrupt (, )
base.proto:1081Interrupts running executions
message
base.proto:724string session_id = 1
(Required) The session_id specifies a spark session for a user id (which is specified by user_context.user_id). The session_id is set by the client to be able to collate streaming responses from different queries within the dedicated session. The id should be an UUID string of the format `00112233-4455-6677-8899-aabbccddeeff`
optional string client_observed_server_side_session_id = 7
(Optional) Server-side generated idempotency key from the previous responses (if any). Server can use this to validate that the server side session has not changed.
optional user_context = 2
(Required) User context
optional string client_type = 3
Provides optional information about the client sending the request. This field can be used for language or version specific information and is only intended for logging purposes and will not be interpreted by the server.
InterruptRequest.InterruptType interrupt_type = 4
(Required) The type of interrupt to execute.
oneof interrupt
string operation_tag = 5
if interrupt_tag == INTERRUPT_TYPE_TAG, interrupt operation with this tag.
string operation_id = 6
if interrupt_tag == INTERRUPT_TYPE_OPERATION_ID, interrupt operation with this operation_id.
message
base.proto:773Next ID: 4
string session_id = 1
Session id in which the interrupt was running.
string server_side_session_id = 3
Server-side generated idempotency key that the client can use to assert that the server side session has not changed.
repeated string interrupted_ids = 2
Operation ids of the executions which were interrupted.
rpc ReattachExecute (, stream )
base.proto:1087Reattach to an existing reattachable execution. The ExecutePlan must have been started with ReattachOptions.reattachable=true. If the ExecutePlanResponse stream ends without a ResultComplete message, there is more to continue. If there is a ResultComplete, the client should use ReleaseExecute with
message
base.proto:797string session_id = 1
(Required) The session_id of the request to reattach to. This must be an id of existing session.
optional string client_observed_server_side_session_id = 6
(Optional) Server-side generated idempotency key from the previous responses (if any). Server can use this to validate that the server side session has not changed.
optional user_context = 2
(Required) User context user_context.user_id and session+id both identify a unique remote spark session on the server side.
string operation_id = 3
(Required) Provide an id of the request to reattach to. This must be an id of existing operation.
optional string client_type = 4
Provides optional information about the client sending the request. This field can be used for language or version specific information and is only intended for logging purposes and will not be interpreted by the server.
optional string last_response_id = 5
(Optional) Last already processed response id from the response stream. After reattach, server will resume the response stream after that response. If not specified, server will restart the stream from the start. Note: server controls the amount of responses that it buffers and it may drop responses, that are far behind the latest returned response, so this can't be used to arbitrarily scroll back the cursor. If the response is no longer available, this will result in an error.
rpc ReleaseExecute (, )
base.proto:1093Release an reattachable execution, or parts thereof. The ExecutePlan must have been started with ReattachOptions.reattachable=true. Non reattachable executions are released automatically and immediately after the ExecutePlan RPC and ReleaseExecute may not be used.
message
base.proto:837string session_id = 1
(Required) The session_id of the request to reattach to. This must be an id of existing session.
optional string client_observed_server_side_session_id = 7
(Optional) Server-side generated idempotency key from the previous responses (if any). Server can use this to validate that the server side session has not changed.
optional user_context = 2
(Required) User context user_context.user_id and session+id both identify a unique remote spark session on the server side.
string operation_id = 3
(Required) Provide an id of the request to reattach to. This must be an id of existing operation.
optional string client_type = 4
Provides optional information about the client sending the request. This field can be used for language or version specific information and is only intended for logging purposes and will not be interpreted by the server.
oneof release
release_all = 5
release_until = 6
message
base.proto:886Next ID: 4
string session_id = 1
Session id in which the release was running.
string server_side_session_id = 3
Server-side generated idempotency key that the client can use to assert that the server side session has not changed.
optional string operation_id = 2
Operation id of the operation on which the release executed. If the operation couldn't be found (because e.g. it was concurrently released), will be unset. Otherwise, it will be equal to the operation_id from request.
rpc ReleaseSession (, )
base.proto:1099Release a session. All the executions in the session will be released. Any further requests for the session with that session_id for the given user_id will fail. If the session didn't exist or was already released, this is a noop.
message
base.proto:899string session_id = 1
(Required) The session_id of the request to reattach to. This must be an id of existing session.
optional user_context = 2
(Required) User context user_context.user_id and session+id both identify a unique remote spark session on the server side.
optional string client_type = 3
Provides optional information about the client sending the request. This field can be used for language or version specific information and is only intended for logging purposes and will not be interpreted by the server.
message
base.proto:919Next ID: 3
string session_id = 1
Session id of the session on which the release executed.
string server_side_session_id = 2
Server-side generated idempotency key that the client can use to assert that the server side session has not changed.
rpc FetchErrorDetails (, )
base.proto:1102FetchErrorDetails retrieves the matched exception with details based on a provided error id.
message
base.proto:927string session_id = 1
(Required) The session_id specifies a Spark session for a user identified by user_context.user_id. The id should be a UUID string of the format `00112233-4455-6677-8899-aabbccddeeff`.
optional string client_observed_server_side_session_id = 5
(Optional) Server-side generated idempotency key from the previous responses (if any). Server can use this to validate that the server side session has not changed.
optional user_context = 2
User context
string error_id = 3
(Required) The id of the error.
optional string client_type = 4
Provides optional information about the client sending the request. This field can be used for language or version specific information and is only intended for logging purposes and will not be interpreted by the server.
message
base.proto:954Next ID: 5
string server_side_session_id = 3
Server-side generated idempotency key that the client can use to assert that the server side session has not changed.
string session_id = 4
optional int32 root_error_idx = 1
The index of the root error in errors. The field will not be set if the error is not found.
repeated errors = 2
A list of errors.
message
base.proto:596A chunk of an Artifact.
Used in: , ,
bytes data = 1
Data chunk.
int64 crc = 2
CRC to allow server to verify integrity of the chunk.
message
base.proto:618A number of `SingleChunkArtifact` batched into a single RPC.
Used in:
repeated artifacts = 1
message
base.proto:625Signals the beginning/start of a chunked artifact. A large artifact is transferred through a payload of `BeginChunkedArtifact` followed by a sequence of `ArtifactChunk`s.
Used in:
string name = 1
Name of the artifact undergoing chunking. Follows the same conventions as the `name` in the `Artifact` message.
int64 total_bytes = 2
Total size of the artifact in bytes.
int64 num_chunks = 3
Number of chunks the artifact is split into. This includes the `initial_chunk`.
optional initial_chunk = 4
The first/initial chunk.
message
base.proto:605An artifact that is contained in a single `ArtifactChunk`. Generally, this message represents tiny artifacts such as REPL-generated class files.
Used in:
string name = 1
The name of the artifact is expected in the form of a "Relative Path" that is made up of a sequence of directories and the final file element. Examples of "Relative Path"s: "jars/test.jar", "classes/xyz.class", "abc.xyz", "a/b/X.jar". The server is expected to maintain the hierarchy of files as defined by their name. (i.e The relative path of the file on the server's filesystem will be the same as the name of the provided artifact)
optional data = 2
A single data chunk.
message
base.proto:665Metadata of an artifact.
Used in:
string name = 1
bool is_crc_successful = 2
Whether the CRC (Cyclic Redundancy Check) is successful on server verification. The server discards any artifact that fails the CRC. If false, the client may choose to resend the artifact specified by `name`.
message
relations.proto:338Relation of type [[Aggregate]].
Used in:
optional input = 1
(Required) Input relation for a RelationalGroupedDataset.
Aggregate.GroupType group_type = 2
(Required) How the RelationalGroupedDataset was built.
repeated grouping_expressions = 3
(Required) Expressions for grouping keys
repeated aggregate_expressions = 4
(Required) List of values that will be translated to columns in the output DataFrame.
optional pivot = 5
(Optional) Pivots a column of the current `DataFrame` and performs the specified aggregation.
repeated grouping_sets = 6
(Optional) List of values that will be translated to columns in the output DataFrame.
Used in:
GROUP_TYPE_UNSPECIFIED = 0
GROUP_TYPE_GROUPBY = 1
GROUP_TYPE_ROLLUP = 2
GROUP_TYPE_CUBE = 3
GROUP_TYPE_PIVOT = 4
GROUP_TYPE_GROUPING_SETS = 5
message
relations.proto:377Used in:
repeated grouping_set = 1
(Required) Individual grouping set
message
relations.proto:366Used in:
optional col = 1
(Required) The column to pivot
repeated values = 2
(Optional) List of values that will be translated to columns in the output DataFrame. Note that if it is empty, the server side will immediately trigger a job to collect the distinct values of the column.
message
base.proto:162Used in:
string ddl_string = 1
(Required) The DDL formatted string to be parsed.
message
base.proto:105Explains the input plan based on a configurable mode.
Used in:
optional plan = 1
(Required) The logical plan to be analyzed.
Explain.ExplainMode explain_mode = 2
(Required) For analyzePlan rpc calls, configure the mode to explain plan in strings.
Plan explanation mode.
Used in:
EXPLAIN_MODE_UNSPECIFIED = 0
EXPLAIN_MODE_SIMPLE = 1
Generates only physical plan.
EXPLAIN_MODE_EXTENDED = 2
Generates parsed logical plan, analyzed logical plan, optimized logical plan and physical plan. Parsed Logical plan is a unresolved plan that extracted from the query. Analyzed logical plans transforms which translates unresolvedAttribute and unresolvedRelation into fully typed objects. The optimized logical plan transforms through a set of optimization rules, resulting in the physical plan.
EXPLAIN_MODE_CODEGEN = 3
Generates code for the statement, if any and a physical plan.
EXPLAIN_MODE_COST = 4
If plan node statistics are available, generates a logical plan and also the statistics.
EXPLAIN_MODE_FORMATTED = 5
Generates a physical plan outline and also node details.
message
base.proto:198Used in:
optional relation = 1
(Required) The logical plan to get the storage level.
message
base.proto:155Used in:
optional plan = 1
(Required) The logical plan to be analyzed.
message
base.proto:145Used in:
optional plan = 1
(Required) The logical plan to be analyzed.
message
base.proto:150Used in:
optional plan = 1
(Required) The logical plan to be analyzed.
message
base.proto:182Used in:
optional relation = 1
(Required) The logical plan to persist.
optional storage_level = 2
(Optional) The storage level.
message
base.proto:169Returns `true` when the logical query plans are equal and therefore return same results.
Used in:
optional target_plan = 1
(Required) The plan to be compared.
optional other_plan = 2
(Required) The other plan to be compared.
message
base.proto:99Used in:
optional plan = 1
(Required) The logical plan to be analyzed.
message
base.proto:177Used in:
optional plan = 1
(Required) The logical plan to get a hashCode.
message
base.proto:160Used in:
(message has no fields)
message
base.proto:137Used in:
optional plan = 1
(Required) The logical plan to be analyzed.
optional int32 level = 2
(Optional) Max level of the schema.
message
base.proto:190Used in:
optional relation = 1
(Required) The logical plan to unpersist.
optional bool blocking = 2
(Optional) Whether to block until all blocks are deleted.
message
base.proto:258Used in:
optional parsed = 1
message
base.proto:233Used in:
string explain_string = 1
message
base.proto:274Used in:
optional storage_level = 1
(Required) The StorageLevel as a result of get_storage_level request.
message
base.proto:249Used in:
repeated string files = 1
A best-effort snapshot of the files that compose this Dataset
message
base.proto:241Used in:
bool is_local = 1
message
base.proto:245Used in:
bool is_streaming = 1
message
base.proto:270Used in:
(message has no fields)
message
base.proto:262Used in:
bool result = 1
message
base.proto:229Used in:
optional schema = 1
message
base.proto:266Used in:
int32 result = 1
message
base.proto:254Used in:
string version = 1
message
base.proto:237Used in:
string tree_string = 1
message
base.proto:272Used in:
(message has no fields)
message
relations.proto:993Used in:
optional input = 1
(Required) Input relation for applyInPandasWithState.
repeated grouping_expressions = 2
(Required) Expressions for grouping keys.
optional func = 3
(Required) Input user-defined function.
string output_schema = 4
(Required) Schema for the output DataFrame.
string state_schema = 5
(Required) Schema for the state.
string output_mode = 6
(Required) The output mode of the function.
string timeout_conf = 7
(Required) Timeout configuration for groups that do not receive data for a while.
message
base.proto:718Used in:
bool exists = 1
Exists or not particular artifact at the server.
message
relations.proto:1097Relation of type [[AsOfJoin]]. `left` and `right` must be present.
Used in:
optional left = 1
(Required) Left input relation for a Join.
optional right = 2
(Required) Right input relation for a Join.
optional left_as_of = 3
(Required) Field to join on in left DataFrame
optional right_as_of = 4
(Required) Field to join on in right DataFrame
optional join_expr = 5
(Optional) The join condition. Could be unset when `using_columns` is utilized. This field does not co-exist with using_columns.
repeated string using_columns = 6
Optional. using_columns provides a list of columns that should present on both sides of the join inputs that this Join will join on. For example A JOIN B USING col_name is equivalent to A JOIN B on A.col_name = B.col_name. This field does not co-exist with join_condition.
string join_type = 7
(Required) The join type.
optional tolerance = 8
(Optional) The asof tolerance within this range.
bool allow_exact_matches = 9
(Required) Whether allow matching with the same value or not.
string direction = 10
(Required) Whether to search for prior, subsequent, or closest matches.
message
catalog.proto:201See `spark.catalog.cacheTable`
Used in:
string table_name = 1
(Required)
optional storage_level = 2
(Optional)
message
relations.proto:444A local relation that has been cached already.
Used in:
string hash = 3
(Required) A sha-256 hash of the serialized local relation in proto, see LocalRelation.
message
relations.proto:456Represents a remote relation that has been cached on server.
Used in: , ,
string relation_id = 1
(Required) ID of the remote related (assigned by the service).
message
expressions.proto:411Used in:
string function_name = 1
(Required) Unparsed name of the SQL function.
repeated arguments = 2
(Optional) Function arguments. Empty arguments are allowed.
message
catalog.proto:30Catalog messages are marked as unstable.
Used in:
oneof cat_type
current_database = 1
set_current_database = 2
list_databases = 3
list_tables = 4
list_functions = 5
list_columns = 6
get_database = 7
get_table = 8
get_function = 9
database_exists = 10
table_exists = 11
function_exists = 12
create_external_table = 13
create_table = 14
drop_temp_view = 15
drop_global_temp_view = 16
recover_partitions = 17
is_cached = 18
cache_table = 19
uncache_table = 20
clear_cache = 21
refresh_table = 22
refresh_by_path = 23
current_catalog = 24
set_current_catalog = 25
list_catalogs = 26
message
commands.proto:500Used in:
optional relation = 1
(Required) The logical plan to checkpoint.
bool local = 2
(Required) Locally checkpoint using a local temporary directory in Spark Connect server (Spark Driver)
bool eager = 3
(Required) Whether to checkpoint this dataframe immediately.
message
base.proto:1054Used in:
optional relation = 1
(Required) The logical plan checkpointed.
message
catalog.proto:216See `spark.catalog.clearCache`
Used in:
(message has no fields)
message
relations.proto:970Used in:
optional input = 1
(Required) One input relation for CoGroup Map API - applyInPandas.
repeated input_grouping_expressions = 2
Expressions for grouping keys of the first input relation.
optional other = 3
(Required) The other input relation.
repeated other_grouping_expressions = 4
Expressions for grouping keys of the other input relation.
optional func = 5
(Required) Input user-defined function.
repeated input_sorting_expressions = 6
(Optional) Expressions for sorting. Only used by Scala Sorted CoGroup Map API.
repeated other_sorting_expressions = 7
(Optional) Expressions for sorting. Only used by Scala Sorted CoGroup Map API.
message
relations.proto:1065Collect arbitrary (named) metrics from a dataset.
Used in:
optional input = 1
(Required) The input relation.
string name = 2
(Required) Name of the metrics.
repeated metrics = 3
(Required) The metric sequence.
message
commands.proto:33A [[Command]] is an operation that is executed by the server that does not directly consume or produce a relational result.
Used in:
oneof command_type
register_function = 1
write_operation = 2
create_dataframe_view = 3
write_operation_v2 = 4
sql_command = 5
write_stream_operation_start = 6
streaming_query_command = 7
get_resources_command = 8
streaming_query_manager_command = 9
register_table_function = 10
streaming_query_listener_bus_command = 11
register_data_source = 12
create_resource_profile_command = 13
checkpoint_command = 14
remove_cached_remote_relation_command = 15
merge_into_table_command = 16
extension = 999
This field is used to mark extensions to the protocol. When plugins generate arbitrary Commands they can add them here. During the planning the correct resolution is done.
message
relations.proto:1046Used in: ,
string name = 1
(Required) Name of the data source.
oneof data_source
(Required) The data source type.
python_data_source = 2
message
expressions.proto:354Used in: , , , , ,
string function_name = 1
(Required) Name of the user-defined function.
bool deterministic = 2
(Optional) Indicate if the user-defined function is deterministic.
repeated arguments = 3
(Optional) Function arguments. Empty arguments are allowed.
oneof function
(Required) Indicate the function type of the user-defined function.
python_udf = 4
scalar_scala_udf = 5
java_udf = 6
message
relations.proto:1016Used in: ,
string function_name = 1
(Required) Name of the user-defined table function.
bool deterministic = 2
(Optional) Whether the user-defined table function is deterministic.
repeated arguments = 3
(Optional) Function input arguments. Empty arguments are allowed.
oneof function
(Required) Type of the user-defined table function.
python_udtf = 4
message
base.proto:519Used in:
repeated string keys = 1
(Required) The config keys to get.
message
base.proto:534Used in:
optional string prefix = 1
(Optional) The prefix of the config key to get.
message
base.proto:529Used in:
repeated string keys = 1
(Required) The config keys to get optionally.
message
base.proto:524Used in:
repeated pairs = 1
(Required) The config key-value pairs to get. The value will be used as the default value.
message
base.proto:544Used in:
repeated string keys = 1
(Required) The config keys to check the config is modifiable.
message
base.proto:502Used in:
oneof op_type
set = 1
get = 2
get_with_default = 3
get_option = 4
get_all = 5
unset = 6
is_modifiable = 7
message
base.proto:514Used in:
repeated pairs = 1
(Required) The config key-value pairs to set.
message
base.proto:539Used in:
repeated string keys = 1
(Required) The config keys to unset.
message
commands.proto:88A command that can create DataFrame global temp view or local temp view.
Used in:
optional input = 1
(Required) The relation that this view will be built on.
string name = 2
(Required) View name.
bool is_global = 3
(Required) Whether this is global temp view or local temp view.
bool replace = 4
(Required) If true, and if the view already exists, updates it; if false, and if the view already exists, throws exception.
message
catalog.proto:145See `spark.catalog.createExternalTable`
Used in:
string table_name = 1
(Required)
optional string path = 2
(Optional)
optional string source = 3
(Optional)
optional schema = 4
(Optional)
map<string, string> options = 5
Options could be empty for valid data source format. The map key is case insensitive.
message
commands.proto:483Command to create ResourceProfile
Used in:
optional profile = 1
(Required) The ResourceProfile to be built on the server-side.
message
commands.proto:489Response for command 'CreateResourceProfileCommand'.
Used in:
int32 profile_id = 1
(Required) Server-side generated resource profile id.
message
catalog.proto:160See `spark.catalog.createTable`
Used in:
string table_name = 1
(Required)
optional string path = 2
(Optional)
optional string source = 3
(Optional)
optional string description = 4
(Optional)
optional schema = 5
(Optional)
map<string, string> options = 6
Options could be empty for valid data source format. The map key is case insensitive.
message
catalog.proto:231See `spark.catalog.currentCatalog`
Used in:
(message has no fields)
message
catalog.proto:62See `spark.catalog.currentDatabase`
Used in:
(message has no fields)
message
types.proto:28This message describes the logical [[DataType]] of something. It does not carry the value itself but only describes it.
Used in: , , , , , , , , , , , , , , , , , , ,
oneof kind
null = 1
binary = 2
boolean = 3
byte = 4
Numeric types
short = 5
integer = 6
long = 7
float = 8
double = 9
decimal = 10
string = 11
String types
char = 12
var_char = 13
date = 14
Datatime types
timestamp = 15
timestamp_ntz = 16
calendar_interval = 17
Interval types
year_month_interval = 18
day_time_interval = 19
array = 20
Complex types
struct = 21
map = 22
variant = 25
udt = 23
UserDefinedType
unparsed = 24
UnparsedDataType
message
types.proto:172Used in:
optional element_type = 1
bool contains_null = 2
uint32 type_variation_reference = 3
message
types.proto:107Used in:
uint32 type_variation_reference = 1
message
types.proto:74Used in:
uint32 type_variation_reference = 1
message
types.proto:78Used in:
uint32 type_variation_reference = 1
message
types.proto:127Used in:
uint32 type_variation_reference = 1
message
types.proto:144Start compound types.
Used in:
int32 length = 1
uint32 type_variation_reference = 2
message
types.proto:119Used in:
uint32 type_variation_reference = 1
message
types.proto:137Used in:
optional int32 start_field = 1
optional int32 end_field = 2
uint32 type_variation_reference = 3
message
types.proto:154Used in:
optional int32 scale = 1
optional int32 precision = 2
uint32 type_variation_reference = 3
message
types.proto:98Used in:
uint32 type_variation_reference = 1
message
types.proto:94Used in:
uint32 type_variation_reference = 1
message
types.proto:86Used in:
uint32 type_variation_reference = 1
message
types.proto:90Used in:
uint32 type_variation_reference = 1
message
types.proto:178Used in:
optional key_type = 1
optional value_type = 2
bool value_contains_null = 3
uint32 type_variation_reference = 4
message
types.proto:111Used in:
uint32 type_variation_reference = 1
message
types.proto:82Used in:
uint32 type_variation_reference = 1
message
types.proto:102Used in:
uint32 type_variation_reference = 1
string collation = 2
message
types.proto:167Used in:
repeated fields = 1
uint32 type_variation_reference = 2
message
types.proto:160Used in:
string name = 1
optional data_type = 2
bool nullable = 3
optional string metadata = 4
message
types.proto:115Used in:
uint32 type_variation_reference = 1
message
types.proto:123Used in:
uint32 type_variation_reference = 1
message
types.proto:189Used in:
string type = 1
optional string jvm_class = 2
optional string python_class = 3
optional string serialized_python_class = 4
optional sql_type = 5
message
types.proto:197Used in:
string data_type_string = 1
(Required) The unparsed data type string
message
types.proto:149Used in:
int32 length = 1
uint32 type_variation_reference = 2
message
types.proto:185Used in:
uint32 type_variation_reference = 1
message
types.proto:131Used in:
optional int32 start_field = 1
optional int32 end_field = 2
uint32 type_variation_reference = 3
message
catalog.proto:123See `spark.catalog.databaseExists`
Used in:
string db_name = 1
(Required)
message
relations.proto:411Relation of type [[Deduplicate]] which have duplicate rows removed, could consider either only the subset of columns or all the columns.
Used in:
optional input = 1
(Required) Input relation for a Deduplicate.
repeated string column_names = 2
(Optional) Deduplicate based on a list of column names. This field does not co-use with `all_columns_as_keys`.
optional bool all_columns_as_keys = 3
(Optional) Deduplicate based on all the columns of the input relation. This field does not co-use with `column_names`.
optional bool within_watermark = 4
(Optional) Deduplicate within the time range of watermark.
message
relations.proto:397Drop specified columns.
Used in:
optional input = 1
(Required) The input relation.
repeated columns = 2
(Optional) columns to drop.
repeated string column_names = 3
(Optional) names of columns to drop.
message
catalog.proto:183See `spark.catalog.dropGlobalTempView`
Used in:
string view_name = 1
(Required)
message
catalog.proto:177See `spark.catalog.dropTempView`
Used in:
string view_name = 1
(Required)
message
example_plugins.proto:40string custom_field = 1
message
example_plugins.proto:35optional child = 1
string custom_field = 2
message
example_plugins.proto:29optional input = 1
string custom_field = 2
message
base.proto:320Used in:
oneof request_option
reattach_options = 1
extension = 999
Extension type for request options
message
base.proto:337The response of a query, can be one or more for each request. Responses belonging to the same input query, carry the same `session_id`. Next ID: 17
Used as response type in: SparkConnectService.ExecutePlan, SparkConnectService.ReattachExecute
string session_id = 1
string server_side_session_id = 15
Server-side generated idempotency key that the client can use to assert that the server side session has not changed.
string operation_id = 12
Identifies the ExecutePlan execution. If set by the client in ExecutePlanRequest.operationId, that value is returned. Otherwise generated by the server. It is an UUID string of the format `00112233-4455-6677-8899-aabbccddeeff`
string response_id = 13
Identified the response in the stream. The id is an UUID string of the format `00112233-4455-6677-8899-aabbccddeeff`
oneof response_type
Union type for the different response messages.
arrow_batch = 2
sql_command_result = 5
Special case for executing SQL commands.
write_stream_operation_start_result = 8
Response for a streaming query.
streaming_query_command_result = 9
Response for commands on a streaming query.
get_resources_command_result = 10
Response for 'SparkContext.resources'.
streaming_query_manager_command_result = 11
Response for commands on the streaming query manager.
streaming_query_listener_events_result = 16
Response for commands on the client side streaming query listener.
result_complete = 14
Response type informing if the stream is complete in reattachable execution.
create_resource_profile_command_result = 17
Response for command that creates ResourceProfile.
execution_progress = 18
(Optional) Intermediate query progress reports.
checkpoint_command_result = 19
Response for command that checkpoints a DataFrame.
extension = 999
Support arbitrary result objects.
optional metrics = 4
Metrics for the query execution. Typically, this field is only present in the last batch of results and then represent the overall state of the query execution.
repeated observed_metrics = 6
The metrics observed during the execution of the query plan.
optional schema = 7
(Optional) The Spark schema. This field is available when `collect` is called.
message
base.proto:408Batch results of metrics.
Used in:
int64 row_count = 1
Count rows in `data`. Must match the number of rows inside `data`.
bytes data = 2
Serialized Arrow data.
optional int64 start_offset = 3
If set, row offset of the start of this ArrowBatch in execution results.
message
base.proto:450This message is used to communicate progress about the query progress during the execution.
Used in:
repeated stages = 1
Captures the progress of each individual stage.
int64 num_inflight_tasks = 2
Captures the currently in progress tasks.
message
base.proto:457Used in:
int64 stage_id = 1
int64 num_tasks = 2
int64 num_completed_tasks = 3
int64 input_bytes_read = 4
bool done = 5
message
base.proto:418Used in:
repeated metrics = 1
message
base.proto:422Used in:
string name = 1
int64 plan_id = 2
int64 parent = 3
map<string, > execution_metrics = 4
message
base.proto:429Used in:
string name = 1
int64 value = 2
string metric_type = 3
message
base.proto:436Used in:
string name = 1
repeated values = 2
repeated string keys = 3
int64 plan_id = 4
message
base.proto:443If present, in a reattachable execution this means that after server sends onComplete, the execution is complete. If the server sends onComplete without sending a ResultComplete, it means that there is more, and the client should use ReattachExecute RPC to continue.
Used in:
(message has no fields)
message
base.proto:403A SQL command returns an opaque Relation that can be directly used as input for the next call.
Used in:
optional relation = 1
message
common.proto:51An executor resource request.
Used in:
string resource_name = 1
(Required) resource name.
int64 amount = 2
(Required) resource amount requesting.
optional string discovery_script = 3
Optional script used to discover the resources.
optional string vendor = 4
Optional vendor, required for some cluster managers.
message
expressions.proto:32Expression used to refer to fields, functions and similar. This can be used everywhere expressions in SQL appear.
Used in: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ,
optional common = 18
oneof expr_type
literal = 1
unresolved_attribute = 2
unresolved_function = 3
expression_string = 4
unresolved_star = 5
alias = 6
cast = 7
unresolved_regex = 8
sort_order = 9
lambda_function = 10
window = 11
unresolved_extract_value = 12
update_fields = 13
unresolved_named_lambda_variable = 14
common_inline_user_defined_function = 15
call_function = 16
named_argument_expression = 17
merge_action = 19
typed_aggregate_expression = 20
extension = 999
This field is used to mark extensions to the protocol. When plugins generate arbitrary relations they can add them here. During the planning the correct resolution is done.
message
expressions.proto:318Used in: ,
optional expr = 1
(Required) The expression that alias will be added on.
repeated string name = 2
(Required) a list of name parts for the alias. Scalar columns only has one name that presents.
optional string metadata = 3
(Optional) Alias metadata expressed as a JSON map.
message
expressions.proto:144Used in:
optional expr = 1
(Required) the expression to be casted.
oneof cast_to_type
(Required) the data type that the expr to be casted to.
type = 2
string type_str = 3
If this is set, Server will use Catalyst parser to parse this string to DataType.
Cast.EvalMode eval_mode = 4
(Optional) The expression evaluation mode.
Used in:
EVAL_MODE_UNSPECIFIED = 0
EVAL_MODE_LEGACY = 1
EVAL_MODE_ANSI = 2
EVAL_MODE_TRY = 3
message
expressions.proto:265Expression as string.
Used in:
string expression = 1
(Required) A SQL expression that will be parsed by Catalyst parser.
message
expressions.proto:331Used in:
optional function = 1
(Required) The lambda function. The function body should use 'UnresolvedAttribute' as arguments, the sever side will replace 'UnresolvedAttribute' with 'UnresolvedNamedLambdaVariable'.
repeated arguments = 2
(Required) Function variables. Must contains 1 ~ 3 variables.
message
expressions.proto:166Used in: , , , , , , , , , ,
oneof literal_type
null = 1
bytes binary = 2
bool boolean = 3
int32 byte = 4
int32 short = 5
int32 integer = 6
int64 long = 7
float float = 10
double double = 11
decimal = 12
string string = 13
int32 date = 16
Date in units of days since the UNIX epoch.
int64 timestamp = 17
Timestamp in units of microseconds since the UNIX epoch.
int64 timestamp_ntz = 18
Timestamp in units of microseconds since the UNIX epoch (without timezone information).
calendar_interval = 19
int32 year_month_interval = 20
int64 day_time_interval = 21
array = 22
map = 23
struct = 24
message
expressions.proto:213Used in:
optional element_type = 1
repeated elements = 2
message
expressions.proto:207Used in:
int32 months = 1
int32 days = 2
int64 microseconds = 3
message
expressions.proto:197Used in:
string value = 1
the string representation.
optional int32 precision = 2
The maximum number of digits allowed in the value. the maximum precision is 38.
optional int32 scale = 3
declared scale of decimal literal
message
expressions.proto:218Used in:
optional key_type = 1
optional value_type = 2
repeated keys = 3
repeated values = 4
message
expressions.proto:225Used in:
optional struct_type = 1
repeated elements = 2
message
expressions.proto:121SortOrder is used to specify the data ordering, it is normally used in Sort and Window. It is an unevaluable expression and cannot be evaluated, so can not be used in Projection.
Used in: , ,
optional child = 1
(Required) The expression to be sorted.
SortOrder.SortDirection direction = 2
(Required) The sort direction, should be ASCENDING or DESCENDING.
SortOrder.NullOrdering null_ordering = 3
(Required) How to deal with NULLs, should be NULLS_FIRST or NULLS_LAST.
Used in:
SORT_NULLS_UNSPECIFIED = 0
SORT_NULLS_FIRST = 1
SORT_NULLS_LAST = 2
Used in:
SORT_DIRECTION_UNSPECIFIED = 0
SORT_DIRECTION_ASCENDING = 1
SORT_DIRECTION_DESCENDING = 2
message
expressions.proto:233An unresolved attribute that is not explicitly bound to a specific column, but the column is resolved during analysis by name.
Used in:
string unparsed_identifier = 1
(Required) An identifier that will be parsed by Catalyst parser. This should follow the Spark SQL identifier syntax.
optional int64 plan_id = 2
(Optional) The id of corresponding connect plan.
optional bool is_metadata_column = 3
(Optional) The requested column is a metadata column.
message
expressions.proto:294Extracts a value or values from an Expression
Used in:
optional child = 1
(Required) The expression to extract value from, can be Map, Array, Struct or array of Structs.
optional extraction = 2
(Required) The expression to describe the extraction, can be key of Map, index of Array, field name of Struct.
message
expressions.proto:247An unresolved function is not explicitly bound to one explicit function, but the function is resolved during analysis following Sparks name resolution rules.
Used in:
string function_name = 1
(Required) name (or unparsed name for user defined function) for the unresolved function.
repeated arguments = 2
(Optional) Function arguments. Empty arguments are allowed.
bool is_distinct = 3
(Required) Indicate if this function should be applied on distinct values.
bool is_user_defined_function = 4
(Required) Indicate if this is a user defined function. When it is not a user defined function, Connect will use the function name directly. When it is a user defined function, Connect will parse the function name first.
message
expressions.proto:342Used in: ,
repeated string name_parts = 1
(Required) a list of name parts for the variable. Must not be empty.
message
expressions.proto:285Represents all of the input attributes to a given relational operator, for example in "SELECT `(id)?+.+` FROM ...".
Used in:
string col_name = 1
(Required) The column name used to extract column with regex.
optional int64 plan_id = 2
(Optional) The id of corresponding connect plan.
message
expressions.proto:271UnresolvedStar is used to expand all the fields of a relation or struct.
Used in:
optional string unparsed_target = 1
(Optional) The target of the expansion. If set, it should end with '.*' and will be parsed by 'parseAttributeName' in the server side.
optional int64 plan_id = 2
(Optional) The id of corresponding connect plan.
message
expressions.proto:305Add, replace or drop a field of `StructType` expression by name.
Used in:
optional struct_expression = 1
(Required) The struct expression.
string field_name = 2
(Required) The field name.
optional value_expression = 3
(Optional) The expression to add or replace. When not set, it means this field will be dropped.
message
expressions.proto:63Expression for the OVER clause or WINDOW clause.
Used in:
optional window_function = 1
(Required) The window function.
repeated partition_spec = 2
(Optional) The way that input rows are partitioned.
repeated order_spec = 3
(Optional) Ordering of rows in a partition.
optional frame_spec = 4
(Optional) Window frame in a partition. If not set, it will be treated as 'UnspecifiedFrame'.
message
expressions.proto:80The window frame
Used in:
WindowFrame.FrameType frame_type = 1
(Required) The type of the frame.
optional lower = 2
(Required) The lower bound of the frame.
optional upper = 3
(Required) The upper bound of the frame.
message
expressions.proto:102Used in:
oneof boundary
bool current_row = 1
CURRENT ROW boundary
bool unbounded = 2
UNBOUNDED boundary. For lower bound, it will be converted to 'UnboundedPreceding'. for upper bound, it will be converted to 'UnboundedFollowing'.
value = 3
This is an expression for future proofing. We are expecting literals on the server side.
Used in:
FRAME_TYPE_UNDEFINED = 0
FRAME_TYPE_ROW = 1
RowFrame treats rows in a partition individually.
FRAME_TYPE_RANGE = 2
RangeFrame treats rows in a partition as groups of peers. All rows having the same 'ORDER BY' ordering are considered as peers.
message
expressions.proto:349Used in:
optional origin = 1
(Required) Keep the information of the origin for this expression such as stacktrace.
message
base.proto:1035Error defines the schema for the representing exception.
Used in:
repeated string error_type_hierarchy = 1
The fully qualified names of the exception class and its parent classes.
string message = 2
The detailed message of the exception.
repeated stack_trace = 3
The stackTrace of the exception. It will be set if the SQLConf spark.sql.connect.serverStacktrace.enabled is true.
optional int32 cause_idx = 4
The index of the cause error in errors.
optional spark_throwable = 5
The structured data of a SparkThrowable exception.
message
base.proto:984QueryContext defines the schema for the query context of a SparkThrowable. It helps users understand where the error occurs while executing queries.
Used in:
QueryContext.ContextType context_type = 10
string object_type = 1
The object type of the query which throws the exception. If the exception is directly from the main query, it should be an empty string. Otherwise, it should be the exact object type in upper case. For example, a "VIEW".
string object_name = 2
The object name of the query which throws the exception. If the exception is directly from the main query, it should be an empty string. Otherwise, it should be the object name. For example, a view name "V1".
int32 start_index = 3
The starting index in the query text which throws the exception. The index starts from 0.
int32 stop_index = 4
The stopping index in the query which throws the exception. The index starts from 0.
string fragment = 5
The corresponding fragment of the query which throws the exception.
string call_site = 6
The user code (call site of the API) that caused throwing the exception.
string summary = 7
Summary of the exception cause.
The type of this query context.
Used in:
SQL = 0
DATAFRAME = 1
message
base.proto:1019SparkThrowable defines the schema for SparkThrowable exceptions.
Used in:
optional string error_class = 1
Succinct, human-readable, unique, and consistent representation of the error category.
map<string, string> message_parameters = 2
The message parameters for the error framework.
repeated query_contexts = 3
The query context of a SparkThrowable.
optional string sql_state = 4
Portable error identifier across SQL engines If null, error class or SQLSTATE is not set.
message
base.proto:968Used in:
string declaring_class = 1
The fully qualified name of the class containing the execution point.
string method_name = 2
The name of the method containing the execution point.
optional string file_name = 3
The name of the file containing the execution point.
int32 line_number = 4
The line number of the source line containing the execution point.
message
relations.proto:219Relation that applies a boolean expression `condition` on each row of `input` to produce the output result.
Used in:
optional input = 1
(Required) Input relation for a Filter.
optional condition = 2
(Required) A Filter must have a condition expression.
message
catalog.proto:137See `spark.catalog.functionExists`
Used in:
string function_name = 1
(Required)
optional string db_name = 2
(Optional)
message
catalog.proto:101See `spark.catalog.getDatabase`
Used in:
string db_name = 1
(Required)
message
catalog.proto:115See `spark.catalog.getFunction`
Used in:
string function_name = 1
(Required)
optional string db_name = 2
(Optional)
message
commands.proto:475Command to get the output of 'SparkContext.resources'
Used in:
(message has no fields)
message
commands.proto:478Response for command 'GetResourcesCommand'.
Used in:
map<string, > resources = 1
message
catalog.proto:107See `spark.catalog.getTable`
Used in:
string table_name = 1
(Required)
optional string db_name = 2
(Optional)
message
relations.proto:940Used in:
optional input = 1
(Required) Input relation for Group Map API: apply, applyInPandas.
repeated grouping_expressions = 2
(Required) Expressions for grouping keys.
optional func = 3
(Required) Input user-defined function.
repeated sorting_expressions = 4
(Optional) Expressions for sorting. Only used by Scala Sorted Group Map API.
optional initial_input = 5
Below fields are only used by (Flat)MapGroupsWithState (Optional) Input relation for initial State.
repeated initial_grouping_expressions = 6
(Optional) Expressions for grouping keys of the initial state input relation.
optional bool is_map_groups_with_state = 7
(Optional) True if MapGroupsWithState, false if FlatMapGroupsWithState.
optional string output_mode = 8
(Optional) The output mode of the function.
optional string timeout_conf = 9
(Optional) Timeout configuration for groups that do not receive data for a while.
message
relations.proto:856Specify a hint over a relation. Hint should have a name and optional parameters.
Used in:
optional input = 1
(Required) The input relation.
string name = 2
(Required) Hint name. Supported Join hints include BROADCAST, MERGE, SHUFFLE_HASH, SHUFFLE_REPLICATE_NL. Supported partitioning hints include COALESCE, REPARTITION, REPARTITION_BY_RANGE.
repeated parameters = 3
(Optional) Hint parameters.
message
relations.proto:545Compose the string representing rows for output. It will invoke 'Dataset.htmlString' to compute the results.
Used in:
optional input = 1
(Required) The input relation.
int32 num_rows = 2
(Required) Number of rows to show.
int32 truncate = 3
(Required) If set to more than 0, truncates strings to `truncate` characters and all cells will be aligned right.
Used in:
INTERRUPT_TYPE_UNSPECIFIED = 0
INTERRUPT_TYPE_ALL = 1
Interrupt all running executions within the session with the provided session_id.
INTERRUPT_TYPE_TAG = 2
Interrupt all running executions within the session with the provided operation_tag.
INTERRUPT_TYPE_OPERATION_ID = 3
Interrupt the running execution within the session with the provided operation_id.
message
catalog.proto:195See `spark.catalog.isCached`
Used in:
string table_name = 1
(Required)
message
expressions.proto:395Used in:
string class_name = 1
(Required) Fully qualified name of Java class
optional output_type = 2
(Optional) Output type of the Java UDF
bool aggregate = 3
(Required) Indicate if the Java user-defined function is an aggregate function
message
relations.proto:230Relation of type [[Join]]. `left` and `right` must be present.
Used in:
optional left = 1
(Required) Left input relation for a Join.
optional right = 2
(Required) Right input relation for a Join.
optional join_condition = 3
(Optional) The join condition. Could be unset when `using_columns` is utilized. This field does not co-exist with using_columns.
Join.JoinType join_type = 4
(Required) The join type.
repeated string using_columns = 5
Optional. using_columns provides a list of columns that should present on both sides of the join inputs that this Join will join on. For example A JOIN B USING col_name is equivalent to A JOIN B on A.col_name = B.col_name. This field does not co-exist with join_condition.
optional join_data_type = 6
(Optional) Only used by joinWith. Set the left and right join data types.
message
relations.proto:266Used in:
bool is_left_struct = 1
If the left data type is a struct.
bool is_right_struct = 2
If the right data type is a struct.
Used in:
JOIN_TYPE_UNSPECIFIED = 0
JOIN_TYPE_INNER = 1
JOIN_TYPE_FULL_OUTER = 2
JOIN_TYPE_LEFT_OUTER = 3
JOIN_TYPE_RIGHT_OUTER = 4
JOIN_TYPE_LEFT_ANTI = 5
JOIN_TYPE_LEFT_SEMI = 6
JOIN_TYPE_CROSS = 7
message
base.proto:468The key-value pair for the config request and response.
Used in: , ,
string key = 1
(Required) The key.
optional string value = 2
(Optional) The value.
message
relations.proto:310Relation of type [[Limit]] that is used to `limit` rows from the input relation.
Used in:
optional input = 1
(Required) Input relation for a Limit.
int32 limit = 2
(Required) the limit.
message
catalog.proto:240See `spark.catalog.listCatalogs`
Used in:
optional string pattern = 1
(Optional) The pattern that the catalog name needs to match
message
catalog.proto:93See `spark.catalog.listColumns`
Used in:
string table_name = 1
(Required)
optional string db_name = 2
(Optional)
message
catalog.proto:71See `spark.catalog.listDatabases`
Used in:
optional string pattern = 1
(Optional) The pattern that the database name needs to match
message
catalog.proto:85See `spark.catalog.listFunctions`
Used in:
optional string db_name = 1
(Optional)
optional string pattern = 2
(Optional) The pattern that the function name needs to match
message
catalog.proto:77See `spark.catalog.listTables`
Used in:
optional string db_name = 1
(Optional)
optional string pattern = 2
(Optional) The pattern that the table name needs to match
message
relations.proto:430A relation that does not need to be qualified by name.
Used in:
optional bytes data = 1
(Optional) Local collection data serialized into Arrow IPC streaming format which contains the schema of the data.
optional string schema = 2
(Optional) The schema of local data. It should be either a DDL-formatted type string or a JSON string. The server side will update the column names and data types according to this schema. If the 'data' is not provided, then this schema will be required.
message
relations.proto:926Used in:
optional input = 1
(Required) Input relation for a mapPartitions-equivalent API: mapInPandas, mapInArrow.
optional func = 2
(Required) Input user-defined function.
optional bool is_barrier = 3
(Optional) Whether to use barrier mode execution or not.
optional int32 profile_id = 4
(Optional) ResourceProfile id used for the stage level scheduling.
message
expressions.proto:427Used in:
MergeAction.ActionType action_type = 1
(Required) The action type of the merge action.
optional condition = 2
(Optional) The condition expression of the merge action.
repeated assignments = 3
(Optional) The assignments of the merge action. Required for ActionTypes INSERT and UPDATE.
Used in:
ACTION_TYPE_INVALID = 0
ACTION_TYPE_DELETE = 1
ACTION_TYPE_INSERT = 2
ACTION_TYPE_INSERT_STAR = 3
ACTION_TYPE_UPDATE = 4
ACTION_TYPE_UPDATE_STAR = 5
message
expressions.proto:446Used in:
optional key = 1
(Required) The key of the assignment.
optional value = 2
(Required) The value of the assignment.
message
commands.proto:512Used in:
string target_table_name = 1
(Required) The name of the target table.
optional source_table_plan = 2
(Required) The relation of the source table.
optional merge_condition = 3
(Required) The condition to match the source and target.
repeated match_actions = 4
(Optional) The actions to be taken when the condition is matched.
repeated not_matched_actions = 5
(Optional) The actions to be taken when the condition is not matched.
repeated not_matched_by_source_actions = 6
(Optional) The actions to be taken when the condition is not matched by source.
bool with_schema_evolution = 7
(Required) Whether to enable schema evolution.
message
relations.proto:738Drop rows containing null values. It will invoke 'Dataset.na.drop' (same as 'DataFrameNaFunctions.drop') to compute the results.
Used in:
optional input = 1
(Required) The input relation.
repeated string cols = 2
(Optional) Optional list of column names to consider. When it is empty, all the columns in the input relation will be considered.
optional int32 min_non_nulls = 3
(Optional) The minimum number of non-null and non-NaN values required to keep. When not set, it is equivalent to the number of considered columns, which means a row will be kept only if all columns are non-null. 'how' options ('all', 'any') can be easily converted to this field: - 'all' -> set 'min_non_nulls' 1; - 'any' -> keep 'min_non_nulls' unset;
message
relations.proto:721Replaces null values. It will invoke 'Dataset.na.fill' (same as 'DataFrameNaFunctions.fill') to compute the results. Following 3 parameter combinations are supported: 1, 'values' only contains 1 item, 'cols' is empty: replaces null values in all type-compatible columns. 2, 'values' only contains 1 item, 'cols' is not empty: replaces null values in specified columns. 3, 'values' contains more than 1 items, then 'cols' is required to have the same length: replaces each specified column with corresponding value.
Used in:
optional input = 1
(Required) The input relation.
repeated string cols = 2
(Optional) Optional list of column names to consider.
repeated values = 3
(Required) Values to replace null values with. Should contain at least 1 item. Only 4 data types are supported now: bool, long, double, string
message
relations.proto:762Replaces old values with the corresponding values. It will invoke 'Dataset.na.replace' (same as 'DataFrameNaFunctions.replace') to compute the results.
Used in:
optional input = 1
(Required) The input relation.
repeated string cols = 2
(Optional) List of column names to consider. When it is empty, all the type-compatible columns in the input relation will be considered.
repeated replacements = 3
(Optional) The value replacement mapping.
message
relations.proto:774Used in:
optional old_value = 1
(Required) The old value. Only 4 data types are supported now: null, bool, double, string.
optional new_value = 2
(Required) The new value. Should be of the same data type with the old value.
message
expressions.proto:419Used in:
string key = 1
(Required) The key of the named argument.
optional value = 2
(Required) The value expression of the named argument.
message
relations.proto:320Relation of type [[Offset]] that is used to read rows staring from the `offset` on the input relation.
Used in:
optional input = 1
(Required) Input relation for an Offset.
int32 offset = 2
(Required) the limit.
message
common.proto:85Used in: ,
oneof function
(Required) Indicate the origin type.
python_origin = 1
message
relations.proto:1076Used in:
optional input = 1
(Required) Input relation to Parse. The input is expected to have single text column.
Parse.ParseFormat format = 2
(Required) The expected format of the text.
optional schema = 3
(Optional) DataType representing the schema. If not set, Spark will infer the schema.
map<string, string> options = 4
Options for the csv/json parser. The map key is case insensitive.
Used in:
PARSE_FORMAT_UNSPECIFIED = 0
PARSE_FORMAT_CSV = 1
PARSE_FORMAT_JSON = 2
message
base.proto:37A [[Plan]] is the structure that carries the runtime information for the execution from the client to the server. A [[Plan]] can either be of the type [[Relation]] which is a reference to the underlying logical plan or it can be of the [[Command]] type that is used to execute commands on the server.
Used in: , , , , , , , ,
oneof op_type
root = 1
command = 2
message
relations.proto:207Projection of a bag of expressions for a given input relation. The input relation must be specified. The projected expression can be an arbitrary expression.
Used in:
optional input = 1
(Optional) Input relation is optional for Project. For example, `SELECT ABS(-1)` is valid plan without an input plan.
repeated expressions = 3
(Required) A Project requires at least one expression.
message
relations.proto:1056Used in:
bytes command = 1
(Required) The encoded commands of the Python data source.
string python_ver = 2
(Required) Python version being used in the client.
message
common.proto:92Used in:
string fragment = 1
(Required) Name of the origin, for example, the name of the function
string call_site = 2
(Required) Callsite to show to end users, for example, stacktrace.
message
expressions.proto:369Used in: , ,
optional output_type = 1
(Required) Output type of the Python UDF
int32 eval_type = 2
(Required) EvalType of the Python UDF
bytes command = 3
(Required) The encoded commands of the Python UDF
string python_ver = 4
(Required) Python version being used in the client.
repeated string additional_includes = 5
(Optional) Additional includes for the Python UDF.
message
relations.proto:1032Used in:
optional return_type = 1
(Optional) Return type of the Python UDTF.
int32 eval_type = 2
(Required) EvalType of the Python UDTF.
bytes command = 3
(Required) The encoded commands of the Python UDTF.
string python_ver = 4
(Required) Python version being used in the client.
message
relations.proto:487Relation of type [[Range]] that generates a sequence of integers.
Used in:
optional int64 start = 1
(Optional) Default value = 0
int64 end = 2
(Required)
int64 step = 3
(Required)
optional int32 num_partitions = 4
Optional. Default value is assigned by 1) SQL conf "spark.sql.leafNodeDefaultParallelism" if it is set, or 2) spark default parallelism.
message
relations.proto:160Relation that reads from a file / table or other data source. Does not have additional inputs.
Used in:
oneof read_type
named_table = 1
data_source = 2
bool is_streaming = 3
(Optional) Indicates if this is a streaming read.
message
relations.proto:177Used in:
optional string format = 1
(Optional) Supported formats include: parquet, orc, text, json, parquet, csv, avro. If not set, the value from SQL conf 'spark.sql.sources.default' will be used.
optional string schema = 2
(Optional) If not set, Spark will infer the schema. This schema string should be either DDL-formatted or JSON-formatted.
map<string, string> options = 3
Options for the data source. The context of this map varies based on the data source format. This options could be empty for valid data source format. The map key is case insensitive.
repeated string paths = 4
(Optional) A list of path for file-system backed data sources.
repeated string predicates = 5
(Optional) Condition in the where clause for each partition. This is only supported by the JDBC data source.
message
relations.proto:169Used in:
string unparsed_identifier = 1
(Required) Unparsed identifier for the table.
map<string, string> options = 2
Options for the named table. The map key is case insensitive.
message
base.proto:785Used in:
bool reattachable = 1
If true, the request can be reattached to using ReattachExecute. ReattachExecute can be used either if the stream broke with a GRPC network error, or if the server closed the stream without sending a response with StreamStatus.complete=true. The server will keep a buffer of responses in case a response is lost, and ReattachExecute needs to back-track. If false, the execution response stream will will not be reattachable, and all responses are immediately released by the server after being sent.
message
catalog.proto:189See `spark.catalog.recoverPartitions`
Used in:
string table_name = 1
(Required)
message
catalog.proto:225See `spark.catalog.refreshByPath`
Used in:
string path = 1
(Required)
message
catalog.proto:219See `spark.catalog.refreshTable`
Used in:
string table_name = 1
(Required)
message
relations.proto:36The main [[Relation]] type. Fundamentally, a relation is a typed container that has exactly one explicit relation type set. When adding new relation types, they have to be registered here.
Used in: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ,
optional common = 1
oneof rel_type
read = 2
project = 3
filter = 4
join = 5
set_op = 6
sort = 7
limit = 8
aggregate = 9
sql = 10
local_relation = 11
sample = 12
offset = 13
deduplicate = 14
range = 15
subquery_alias = 16
repartition = 17
to_df = 18
with_columns_renamed = 19
show_string = 20
drop = 21
tail = 22
with_columns = 23
hint = 24
unpivot = 25
to_schema = 26
repartition_by_expression = 27
map_partitions = 28
collect_metrics = 29
parse = 30
group_map = 31
co_group_map = 32
with_watermark = 33
apply_in_pandas_with_state = 34
html_string = 35
cached_local_relation = 36
cached_remote_relation = 37
common_inline_user_defined_table_function = 38
as_of_join = 39
common_inline_user_defined_data_source = 40
with_relations = 41
transpose = 42
fill_na = 90
NA functions
drop_na = 91
replace = 92
summary = 100
stat functions
crosstab = 101
describe = 102
cov = 103
corr = 104
approx_quantile = 105
freq_items = 106
sample_by = 107
catalog = 200
Catalog API (experimental / unstable)
extension = 998
This field is used to mark extensions to the protocol. When plugins generate arbitrary relations they can add them here. During the planning the correct resolution is done.
unknown = 999
message
relations.proto:110Common metadata of all relations.
Used in:
string source_info = 1
(Required) Shared relation metadata.
optional int64 plan_id = 2
(Optional) A per-client globally unique id for a given connect plan.
optional origin = 3
(Optional) Keep the information of the origin for this expression such as stacktrace.
message
base.proto:868Release and close operation completely. This will also interrupt the query if it is running execution, and wait for it to be torn down.
Used in:
(message has no fields)
message
base.proto:875Release all responses from the operation response stream up to and including the response with the given by response_id. While server determines by itself how much of a buffer of responses to keep, client providing explicit release calls will help reduce resource consumption. Noop if response_id not found in cached responses.
Used in:
string response_id = 1
message
commands.proto:495Command to remove `CashedRemoteRelation`
Used in:
optional relation = 1
(Required) The remote to be related
message
relations.proto:515Relation repartition.
Used in:
optional input = 1
(Required) The input relation of Repartition.
int32 num_partitions = 2
(Required) Must be positive.
optional bool shuffle = 3
(Optional) Default value is false.
message
relations.proto:915Used in:
optional input = 1
(Required) The input relation.
repeated partition_exprs = 2
(Required) The partitioning expressions.
optional int32 num_partitions = 3
(Optional) number of partitions, must be positive.
message
common.proto:43ResourceInformation to hold information about a type of Resource. The corresponding class is 'org.apache.spark.resource.ResourceInformation'
Used in:
string name = 1
(Required) The name of the resource
repeated string addresses = 2
(Required) An array of strings describing the addresses of the resource.
message
common.proto:75Used in:
map<string, > executor_resources = 1
(Optional) Resource requests for executors. Mapped from the resource name (e.g., cores, memory, CPU) to its specific request.
map<string, > task_resources = 2
(Optional) Resource requests for tasks. Mapped from the resource name (e.g., cores, memory, CPU) to its specific request.
message
relations.proto:122Relation that uses a SQL query to generate the output.
Used in:
string query = 1
(Required) The SQL query.
map<string, > args = 2
(Optional) A map of parameter names to literal expressions.
repeated pos_args = 3
(Optional) A sequence of literal expressions for positional parameters in the SQL query text.
map<string, > named_arguments = 4
(Optional) A map of parameter names to expressions. It cannot coexist with `pos_arguments`.
repeated pos_arguments = 5
(Optional) A sequence of expressions for positional parameters in the SQL query text. It cannot coexist with `named_arguments`.
message
relations.proto:462Relation of type [[Sample]] that samples a fraction of the dataset.
Used in:
optional input = 1
(Required) Input relation for a Sample.
double lower_bound = 2
(Required) lower bound.
double upper_bound = 3
(Required) upper bound.
optional bool with_replacement = 4
(Optional) Whether to sample with replacement.
optional int64 seed = 5
(Required) The random seed. This field is required to avoid generating mutable dataframes (see SPARK-48184 for details), however, still keep it 'optional' here for backward compatibility.
bool deterministic_order = 6
(Required) Explicitly sort the underlying plan to make the ordering deterministic or cache it. This flag is true when invoking `dataframe.randomSplit` to randomly splits DataFrame with the provided weights. Otherwise, it is false.
message
expressions.proto:382Used in: , ,
bytes payload = 1
(Required) Serialized JVM object containing UDF definition, input encoders and output encoder
repeated inputTypes = 2
(Optional) Input type(s) of the UDF
optional outputType = 3
(Required) Output type of the UDF
bool nullable = 4
(Required) True if the UDF can return null value
bool aggregate = 5
(Required) Indicate if the UDF is an aggregate function
message
catalog.proto:234See `spark.catalog.setCurrentCatalog`
Used in:
string catalog_name = 1
(Required)
message
catalog.proto:65See `spark.catalog.setCurrentDatabase`
Used in:
string db_name = 1
(Required)
message
relations.proto:275Relation of type [[SetOperation]]
Used in:
optional left_input = 1
(Required) Left input relation for a Set operation.
optional right_input = 2
(Required) Right input relation for a Set operation.
SetOperation.SetOpType set_op_type = 3
(Required) The Set operation type.
optional bool is_all = 4
(Optional) If to remove duplicate rows. True to preserve all results. False to remove duplicate rows.
optional bool by_name = 5
(Optional) If to perform the Set operation based on name resolution. Only UNION supports this option.
optional bool allow_missing_columns = 6
(Optional) If to perform the Set operation and allow missing columns. Only UNION supports this option.
Used in:
SET_OP_TYPE_UNSPECIFIED = 0
SET_OP_TYPE_INTERSECT = 1
SET_OP_TYPE_UNION = 2
SET_OP_TYPE_EXCEPT = 3
message
relations.proto:528Compose the string representing rows for output. It will invoke 'Dataset.showString' to compute the results.
Used in:
optional input = 1
(Required) The input relation.
int32 num_rows = 2
(Required) Number of rows to show.
int32 truncate = 3
(Required) If set to more than 0, truncates strings to `truncate` characters and all cells will be aligned right.
bool vertical = 4
(Required) If set to true, prints output rows vertically (one line per column value).
message
relations.proto:384Relation of type [[Sort]].
Used in:
optional input = 1
(Required) Input relation for a Sort.
repeated order = 2
(Required) The ordering expressions
optional bool is_global = 3
(Optional) if this is a global sort.
message
commands.proto:65A SQL Command is used to trigger the eager evaluation of SQL commands in Spark. When the SQL provide as part of the message is a command it will be immediately evaluated and the result will be collected and returned as part of a LocalRelation. If the result is not a command, the operation will simply return a SQL Relation. This allows the client to be almost oblivious to the server-side behavior.
Used in:
string sql = 1
(Required) SQL Query.
map<string, > args = 2
(Optional) A map of parameter names to literal expressions.
repeated pos_args = 3
(Optional) A sequence of literal expressions for positional parameters in the SQL query text.
map<string, > named_arguments = 4
(Optional) A map of parameter names to expressions. It cannot coexist with `pos_arguments`.
repeated pos_arguments = 5
(Optional) A sequence of expressions for positional parameters in the SQL query text. It cannot coexist with `named_arguments`.
optional input = 6
(Optional) The relation that this SQL command will be built on.
message
relations.proto:645Calculates the approximate quantiles of numerical columns of a DataFrame. It will invoke 'Dataset.stat.approxQuantile' (same as 'StatFunctions.approxQuantile') to compute the results.
Used in:
optional input = 1
(Required) The input relation.
repeated string cols = 2
(Required) The names of the numerical columns.
repeated double probabilities = 3
(Required) A list of quantile probabilities. Each number must belong to [0, 1]. For example 0 is the minimum, 0.5 is the median, 1 is the maximum.
double relative_error = 4
(Required) The relative target precision to achieve (greater than or equal to 0). If set to zero, the exact quantiles are computed, which could be very expensive. Note that values greater than 1 are accepted but give the same result as 1.
message
relations.proto:626Calculates the correlation of two columns of a DataFrame. Currently only supports the Pearson Correlation Coefficient. It will invoke 'Dataset.stat.corr' (same as 'StatFunctions.pearsonCorrelation') to compute the results.
Used in:
optional input = 1
(Required) The input relation.
string col1 = 2
(Required) The name of the first column.
string col2 = 3
(Required) The name of the second column.
optional string method = 4
(Optional) Default value is 'pearson'. Currently only supports the Pearson Correlation Coefficient.
message
relations.proto:612Calculate the sample covariance of two numerical columns of a DataFrame. It will invoke 'Dataset.stat.cov' (same as 'StatFunctions.calculateCov') to compute the results.
Used in:
optional input = 1
(Required) The input relation.
string col1 = 2
(Required) The name of the first column.
string col2 = 3
(Required) The name of the second column.
message
relations.proto:595Computes a pair-wise frequency table of the given columns. Also known as a contingency table. It will invoke 'Dataset.stat.crosstab' (same as 'StatFunctions.crossTabulate') to compute the results.
Used in:
optional input = 1
(Required) The input relation.
string col1 = 2
(Required) The name of the first column. Distinct items will make the first item of each row.
string col2 = 3
(Required) The name of the second column. Distinct items will make the column names of the DataFrame.
message
relations.proto:584Computes basic statistics for numeric and string columns, including count, mean, stddev, min, and max. If no columns are given, this function computes statistics for all numerical or string columns.
Used in:
optional input = 1
(Required) The input relation.
repeated string cols = 2
(Optional) Columns to compute statistics on.
message
relations.proto:668Finding frequent items for columns, possibly with false positives. It will invoke 'Dataset.stat.freqItems' (same as 'StatFunctions.freqItems') to compute the results.
Used in:
optional input = 1
(Required) The input relation.
repeated string cols = 2
(Required) The names of the columns to search frequent items in.
optional double support = 3
(Optional) The minimum frequency for an item to be considered `frequent`. Should be greater than 1e-4.
message
relations.proto:685Returns a stratified sample without replacement based on the fraction given on each stratum. It will invoke 'Dataset.stat.freqItems' (same as 'StatFunctions.freqItems') to compute the results.
Used in:
optional input = 1
(Required) The input relation.
optional col = 2
(Required) The column that defines strata.
repeated fractions = 3
(Required) Sampling fraction for each stratum. If a stratum is not specified, we treat its fraction as zero.
optional int64 seed = 5
(Required) The random seed. This field is required to avoid generating mutable dataframes (see SPARK-48184 for details), however, still keep it 'optional' here for backward compatibility.
message
relations.proto:702Used in:
optional stratum = 1
(Required) The stratum.
double fraction = 2
(Required) The fraction value. Must be in [0, 1].
message
relations.proto:560Computes specified statistics for numeric and string columns. It will invoke 'Dataset.summary' (same as 'StatFunctions.summary') to compute the results.
Used in:
optional input = 1
(Required) The input relation.
repeated string statistics = 2
(Optional) Statistics from to be computed. Available statistics are: count mean stddev min max arbitrary approximate percentiles specified as a percentage (e.g. 75%) count_distinct approx_count_distinct If no statistics are given, this function computes 'count', 'mean', 'stddev', 'min', 'approximate quartiles' (percentiles at 25%, 50%, and 75%), and 'max'.
message
common.proto:27StorageLevel for persisting Datasets/Tables.
Used in: , ,
bool use_disk = 1
(Required) Whether the cache should use disk or not.
bool use_memory = 2
(Required) Whether the cache should use memory or not.
bool use_off_heap = 3
(Required) Whether the cache should use off-heap or not.
bool deserialized = 4
(Required) Whether the cached data is deserialized or not.
int32 replication = 5
(Required) The number of replicas.
message
commands.proto:248Used in:
oneof function
python_function = 1
scala_function = 2
message
commands.proto:287Commands for a streaming query.
Used in:
optional query_id = 1
(Required) Query instance. See `StreamingQueryInstanceId`.
oneof command
See documentation for the corresponding API method in StreamingQuery.
bool status = 2
status() API.
bool last_progress = 3
lastProgress() API.
bool recent_progress = 4
recentProgress() API.
bool stop = 5
stop() API. Stops the query.
bool process_all_available = 6
processAllAvailable() API. Waits till all the available data is processed
explain = 7
explain() API. Returns logical and physical plans.
bool exception = 8
exception() API. Returns the exception in the query if any.
await_termination = 9
awaitTermination() API. Waits for the termination of the query.
message
commands.proto:318Used in:
optional int64 timeout_ms = 2
message
commands.proto:312Used in:
bool extended = 1
TODO: Consider reusing Explain from AnalyzePlanRequest message. We can not do this right now since it base.proto imports this file.
message
commands.proto:324Response for commands on a streaming query.
Used in:
optional query_id = 1
(Required) Query instance id. See `StreamingQueryInstanceId`.
oneof result_type
status = 2
recent_progress = 3
explain = 4
exception = 5
await_termination = 6
message
commands.proto:364Used in:
bool terminated = 1
message
commands.proto:354Used in:
optional string exception_message = 1
(Optional) Exception message as string, maps to the return value of original StreamingQueryException's toString method
optional string error_class = 2
(Optional) Exception error class as string
optional string stack_trace = 3
(Optional) Exception stack trace as string
message
commands.proto:349Used in:
string result = 1
Logical and physical plans as string
message
commands.proto:344Used in:
repeated string recent_progress_json = 5
Progress reports as an array of json strings.
message
commands.proto:336Used in:
string status_message = 1
See documentation for these Scala 'StreamingQueryStatus' struct
bool is_data_available = 2
bool is_trigger_active = 3
bool is_active = 4
The enum used for client side streaming query listener event There is no QueryStartedEvent defined here, it is added as a field in WriteStreamOperationStartResult
Used in:
QUERY_PROGRESS_UNSPECIFIED = 0
QUERY_PROGRESS_EVENT = 1
QUERY_TERMINATED_EVENT = 2
QUERY_IDLE_EVENT = 3
message
commands.proto:273A tuple that uniquely identifies an instance of streaming query run. It consists of `id` that persists across the streaming runs and `run_id` that changes between each run of the streaming query that resumes from the checkpoint.
Used in: , , ,
string id = 1
(Required) The unique id of this query that persists across restarts from checkpoint data. That is, this id is generated when a query is started for the first time, and will be the same every time it is restarted from checkpoint data.
string run_id = 2
(Required) The unique id of this run of the query. That is, every start/restart of a query will generate a unique run_id. Therefore, every time a query is restarted from checkpoint, it will have the same `id` but different `run_id`s.
message
commands.proto:444The protocol for client-side StreamingQueryListener. This command will only be set when either the first listener is added to the client, or the last listener is removed from the client. The add_listener_bus_listener command will only be set true in the first case. The remove_listener_bus_listener command will only be set true in the second case.
Used in:
oneof command
bool add_listener_bus_listener = 1
bool remove_listener_bus_listener = 2
message
commands.proto:462The protocol for the returned events in the long-running response channel.
Used in:
string event_json = 1
(Required) The json serialized event, all StreamingQueryListener events have a json method
StreamingQueryEventType event_type = 2
(Required) Query event type used by client to decide how to deserialize the event_json
message
commands.proto:469Used in:
repeated events = 1
optional bool listener_bus_listener_added = 2
message
commands.proto:370Commands for the streaming query manager.
Used in:
oneof command
See documentation for the corresponding API method in StreamingQueryManager.
bool active = 1
active() API, returns a list of active queries.
string get_query = 2
get() API, returns the StreamingQuery identified by id.
await_any_termination = 3
awaitAnyTermination() API, wait until any query terminates or timeout.
bool reset_terminated = 4
resetTerminated() API.
add_listener = 5
addListener API.
remove_listener = 6
removeListener API.
bool list_listeners = 7
listListeners() API, returns a list of streaming query listeners.
message
commands.proto:390Used in:
optional int64 timeout_ms = 1
(Optional) The waiting time in milliseconds to wait for any query to terminate.
message
commands.proto:395Used in:
bytes listener_payload = 1
optional python_listener_payload = 2
string id = 3
message
commands.proto:403Response for commands on the streaming query manager.
Used in:
oneof result_type
active = 1
query = 2
await_any_termination = 3
bool reset_terminated = 4
bool add_listener = 5
bool remove_listener = 6
list_listeners = 7
message
commands.proto:414Used in:
repeated active_queries = 1
message
commands.proto:425Used in:
bool terminated = 1
message
commands.proto:433Used in:
repeated string listener_ids = 1
(Required) Reference IDs of listener instances.
message
commands.proto:418Used in: ,
optional id = 1
(Required) The id and runId of this query.
optional string name = 2
(Optional) The name of this query.
message
commands.proto:429bytes listener_payload = 1
message
relations.proto:503Relation alias.
Used in:
optional input = 1
(Required) The input relation of SubqueryAlias.
string alias = 2
(Required) The alias.
repeated string qualifier = 3
(Optional) Qualifier of the alias.
message
catalog.proto:129See `spark.catalog.tableExists`
Used in:
string table_name = 1
(Required)
optional string db_name = 2
(Optional)
message
relations.proto:329Relation of type [[Tail]] that is used to fetch `limit` rows from the last of the input relation.
Used in:
optional input = 1
(Required) Input relation for an Tail.
int32 limit = 2
(Required) the limit.
message
common.proto:66A task resource request.
Used in:
string resource_name = 1
(Required) resource name.
double amount = 2
(Required) resource amount requesting as a double to support fractional resource requests.
message
relations.proto:789Rename columns on the input relation by the same length of names.
Used in:
optional input = 1
(Required) The input relation of RenameColumnsBySameLengthNames.
repeated string column_names = 2
(Required) The number of columns of the input relation must be equal to the length of this field. If this is not true, an exception will be returned.
message
relations.proto:905Used in:
optional input = 1
(Required) The input relation.
optional schema = 2
(Required) The user provided schema. The Sever side will update the dataframe with this schema.
message
relations.proto:896Transpose a DataFrame, switching rows to columns. Transforms the DataFrame such that the values in the specified index column become the new columns of the DataFrame.
Used in:
optional input = 1
(Required) The input relation.
repeated index_columns = 2
(Optional) A list of columns that will be treated as the indices. Only single column is supported now.
message
expressions.proto:406Used in:
optional scalar_scala_udf = 1
(Required) The aggregate function object packed into bytes.
message
catalog.proto:210See `spark.catalog.uncacheTable`
Used in:
string table_name = 1
(Required)
message
relations.proto:107Used for testing purposes only.
Used in:
(message has no fields)
message
relations.proto:872Unpivot a DataFrame from wide format to long format, optionally leaving identifier columns set.
Used in:
optional input = 1
(Required) The input relation.
repeated ids = 2
(Required) Id columns.
optional values = 3
(Optional) Value columns to unpivot.
string variable_column_name = 4
(Required) Name of the variable column.
string value_column_name = 5
(Required) Name of the value column.
message
relations.proto:888Used in:
repeated values = 1
message
base.proto:48User Context is used to refer to one particular user session that is executing queries in the backend.
Used in: , , , , , , , , ,
string user_id = 1
string user_name = 2
repeated extensions = 999
To extend the existing user context message that is used to identify incoming requests, Spark Connect leverages the Any protobuf type that can be used to inject arbitrary other messages into this message. Extensions are stored as a `repeated` type to be able to handle multiple active extensions.
message
relations.proto:827Adding columns or replacing the existing columns that have the same names.
Used in:
optional input = 1
(Required) The input relation.
repeated aliases = 2
(Required) Given a column name, apply the corresponding expression on the column. If column name exists in the input relation, then replace the column. If the column name does not exist in the input relation, then adds it as a new column. Only one name part is expected from each Expression.Alias. An exception is thrown when duplicated names are present in the mapping.
message
relations.proto:802Rename columns on the input relation by a map with name to name mapping.
Used in:
optional input = 1
(Required) The input relation.
map<string, string> rename_columns_map = 2
(Optional) Renaming column names of input relation from A to B where A is the map key and B is the map value. This is a no-op if schema doesn't contain any A. It does not require that all input relation column names to present as keys. duplicated B are not allowed.
repeated renames = 3
message
relations.proto:817Used in:
string col_name = 1
(Required) The existing column name.
string new_col_name = 2
(Required) The new column name.
message
relations.proto:148Relation of type [[WithRelations]]. This relation contains a root plan, and one or more references that are used by the root plan. There are two ways of referencing a relation, by name (through a subquery alias), or by plan_id (using RelationCommon.plan_id). This relation can be used to implement CTEs, describe DAGs, or to reduce tree depth.
Used in:
optional root = 1
(Required) Plan at the root of the query tree. This plan is expected to contain one or more references. Those references get expanded later on by the engine.
repeated references = 2
(Required) Plans referenced by the root plan. Relations in this list are also allowed to contain references to other relations in this list, as long they do not form cycles.
message
relations.proto:843Used in:
optional input = 1
(Required) The input relation
string event_time = 2
(Required) Name of the column containing event time.
string delay_threshold = 3
(Required)
message
commands.proto:106As writes are not directly handled during analysis and planning, they are modeled as commands.
Used in:
optional input = 1
(Required) The output of the `input` relation will be persisted according to the options.
optional string source = 2
(Optional) Format value according to the Spark documentation. Examples are: text, parquet, delta.
oneof save_type
(Optional) The destination of the write operation can be either a path or a table. If the destination is neither a path nor a table, such as jdbc and noop, the `save_type` should not be set.
string path = 3
table = 4
WriteOperation.SaveMode mode = 5
(Required) the save mode.
repeated string sort_column_names = 6
(Optional) List of columns to sort the output by.
repeated string partitioning_columns = 7
(Optional) List of columns for partitioning.
optional bucket_by = 8
(Optional) Bucketing specification. Bucketing must set the number of buckets and the columns to bucket by.
map<string, string> options = 9
(Optional) A list of configuration options.
repeated string clustering_columns = 10
(Optional) Columns used for clustering the table.
message
commands.proto:155Used in:
repeated string bucket_column_names = 1
int32 num_buckets = 2
Used in:
SAVE_MODE_UNSPECIFIED = 0
SAVE_MODE_APPEND = 1
SAVE_MODE_OVERWRITE = 2
SAVE_MODE_ERROR_IF_EXISTS = 3
SAVE_MODE_IGNORE = 4
message
commands.proto:142Used in:
string table_name = 1
(Required) The table name.
SaveTable.TableSaveMethod save_method = 2
(Required) The method to be called to write to the table.
Used in:
TABLE_SAVE_METHOD_UNSPECIFIED = 0
TABLE_SAVE_METHOD_SAVE_AS_TABLE = 1
TABLE_SAVE_METHOD_INSERT_INTO = 2
message
commands.proto:170As writes are not directly handled during analysis and planning, they are modeled as commands.
Used in:
optional input = 1
(Required) The output of the `input` relation will be persisted according to the options.
string table_name = 2
(Required) The destination of the write operation must be either a path or a table.
optional string provider = 3
(Optional) A provider for the underlying output data source. Spark's default catalog supports "parquet", "json", etc.
repeated partitioning_columns = 4
(Optional) List of columns for partitioning for output table created by `create`, `createOrReplace`, or `replace`
map<string, string> options = 5
(Optional) A list of configuration options.
map<string, string> table_properties = 6
(Optional) A list of table properties.
WriteOperationV2.Mode mode = 7
(Required) Write mode.
optional overwrite_condition = 8
(Optional) A condition for overwrite saving mode
repeated string clustering_columns = 9
(Optional) Columns used for clustering the table.
Used in:
MODE_UNSPECIFIED = 0
MODE_CREATE = 1
MODE_OVERWRITE = 2
MODE_OVERWRITE_PARTITIONS = 3
MODE_APPEND = 4
MODE_REPLACE = 5
MODE_CREATE_OR_REPLACE = 6
message
commands.proto:213Starts write stream operation as streaming query. Query ID and Run ID of the streaming query are returned.
Used in:
optional input = 1
(Required) The output of the `input` streaming relation will be written.
string format = 2
map<string, string> options = 3
repeated string partitioning_column_names = 4
oneof trigger
string processing_time_interval = 5
bool available_now = 6
bool once = 7
string continuous_checkpoint_interval = 8
string output_mode = 9
string query_name = 10
oneof sink_destination
The destination is optional. When set, it can be a path or a table name.
string path = 11
string table_name = 12
optional foreach_writer = 13
optional foreach_batch = 14
repeated string clustering_column_names = 15
(Optional) Columns used for clustering the table.
message
commands.proto:255Used in:
optional query_id = 1
(Required) Query instance. See `StreamingQueryInstanceId`.
string name = 2
An optional query name.
optional string query_started_event_json = 3
Optional query started event if there is any listener registered on the client side.