package spark.connect
Get desktop application:
View/edit binary Protocol Buffers messages
Main interface for the SparkConnect service.
rpc AddArtifacts (stream , )
base.proto:1135Add artifacts to the session and returns a [[AddArtifactsResponse]] containing metadata about the added artifacts.
message
base.proto:617Request to transfer client-local artifacts.
string session_id = 1
(Required) The session_id specifies a spark session for a user id (which is specified by user_context.user_id). The session_id is set by the client to be able to collate streaming responses from different queries within the dedicated session. The id should be an UUID string of the format `00112233-4455-6677-8899-aabbccddeeff`
optional user_context = 2
User context
optional string client_observed_server_side_session_id = 7
(Optional) Server-side generated idempotency key from the previous responses (if any). Server can use this to validate that the server side session has not changed.
optional string client_type = 6
Provides optional information about the client sending the request. This field can be used for language or version specific information and is only intended for logging purposes and will not be interpreted by the server.
oneof payload
The payload is either a batch of artifacts or a partial chunk of a large artifact.
batch = 3
begin_chunk = 4
The metadata and the initial chunk of a large artifact chunked into multiple requests. The server side is notified about the total size of the large artifact as well as the number of chunks to expect.
chunk = 5
A chunk of an artifact excluding metadata. This can be any chunk of a large artifact excluding the first chunk (which is included in `BeginChunkedArtifact`).
message
base.proto:700Response to adding an artifact. Contains relevant metadata to verify successful transfer of artifact(s). Next ID: 4
string session_id = 2
Session id in which the AddArtifact was running.
string server_side_session_id = 3
Server-side generated idempotency key that the client can use to assert that the server side session has not changed.
repeated artifacts = 1
The list of artifact(s) seen by the server.
rpc AnalyzePlan (, )
base.proto:1128Analyzes a query and returns a [[AnalyzeResponse]] containing metadata about the query.
message
base.proto:83Request to perform plan analyze, optionally to explain the plan.
string session_id = 1
(Required) The session_id specifies a spark session for a user id (which is specified by user_context.user_id). The session_id is set by the client to be able to collate streaming responses from different queries within the dedicated session. The id should be an UUID string of the format `00112233-4455-6677-8899-aabbccddeeff`
optional string client_observed_server_side_session_id = 17
(Optional) Server-side generated idempotency key from the previous responses (if any). Server can use this to validate that the server side session has not changed.
optional user_context = 2
(Required) User context
optional string client_type = 3
Provides optional information about the client sending the request. This field can be used for language or version specific information and is only intended for logging purposes and will not be interpreted by the server.
oneof analyze
schema = 4
explain = 5
tree_string = 6
is_local = 7
is_streaming = 8
input_files = 9
spark_version = 10
ddl_parse = 11
same_semantics = 12
semantic_hash = 13
persist = 14
unpersist = 15
get_storage_level = 16
json_to_ddl = 18
message
base.proto:236Response to performing analysis of the query. Contains relevant metadata to be able to reason about the performance. Next ID: 16
string session_id = 1
string server_side_session_id = 15
Server-side generated idempotency key that the client can use to assert that the server side session has not changed.
oneof result
schema = 2
explain = 3
tree_string = 4
is_local = 5
is_streaming = 6
input_files = 7
spark_version = 8
ddl_parse = 9
same_semantics = 10
semantic_hash = 11
persist = 12
unpersist = 13
get_storage_level = 14
json_to_ddl = 16
rpc ArtifactStatus (, )
base.proto:1138Check statuses of artifacts in the session and returns them in a [[ArtifactStatusesResponse]]
message
base.proto:721Request to get current statuses of artifacts at the server side.
string session_id = 1
(Required) The session_id specifies a spark session for a user id (which is specified by user_context.user_id). The session_id is set by the client to be able to collate streaming responses from different queries within the dedicated session. The id should be an UUID string of the format `00112233-4455-6677-8899-aabbccddeeff`
optional string client_observed_server_side_session_id = 5
(Optional) Server-side generated idempotency key from the previous responses (if any). Server can use this to validate that the server side session has not changed.
optional user_context = 2
User context
optional string client_type = 3
Provides optional information about the client sending the request. This field can be used for language or version specific information and is only intended for logging purposes and will not be interpreted by the server.
repeated string names = 4
The name of the artifact is expected in the form of a "Relative Path" that is made up of a sequence of directories and the final file element. Examples of "Relative Path"s: "jars/test.jar", "classes/xyz.class", "abc.xyz", "a/b/X.jar". The server is expected to maintain the hierarchy of files as defined by their name. (i.e The relative path of the file on the server's filesystem will be the same as the name of the provided artifact)
message
base.proto:755Response to checking artifact statuses. Next ID: 4
string session_id = 2
Session id in which the ArtifactStatus was running.
string server_side_session_id = 3
Server-side generated idempotency key that the client can use to assert that the server side session has not changed.
map<string, > statuses = 1
A map of artifact names to their statuses.
rpc Config (, )
base.proto:1131Update or fetch the configurations and returns a [[ConfigResponse]] containing the result.
message
base.proto:519Request to update or fetch the configurations.
string session_id = 1
(Required) The session_id specifies a spark session for a user id (which is specified by user_context.user_id). The session_id is set by the client to be able to collate streaming responses from different queries within the dedicated session. The id should be an UUID string of the format `00112233-4455-6677-8899-aabbccddeeff`
optional string client_observed_server_side_session_id = 8
(Optional) Server-side generated idempotency key from the previous responses (if any). Server can use this to validate that the server side session has not changed.
optional user_context = 2
(Required) User context
optional operation = 3
(Required) The operation for the config.
optional string client_type = 4
Provides optional information about the client sending the request. This field can be used for language or version specific information and is only intended for logging purposes and will not be interpreted by the server.
message
base.proto:598Response to the config request. Next ID: 5
string session_id = 1
string server_side_session_id = 4
Server-side generated idempotency key that the client can use to assert that the server side session has not changed.
repeated pairs = 2
(Optional) The result key-value pairs. Available when the operation is 'Get', 'GetWithDefault', 'GetOption', 'GetAll'. Also available for the operation 'IsModifiable' with boolean string "true" and "false".
repeated string warnings = 3
(Optional) Warning messages for deprecated or unsupported configurations.
rpc ExecutePlan (, stream )
base.proto:1125Executes a request that contains the query and returns a stream of [[Response]]. It is guaranteed that there is at least one ARROW batch returned even if the result set is empty.
message
base.proto:315A request to be executed by the service.
string session_id = 1
(Required) The session_id specifies a spark session for a user id (which is specified by user_context.user_id). The session_id is set by the client to be able to collate streaming responses from different queries within the dedicated session. The id should be an UUID string of the format `00112233-4455-6677-8899-aabbccddeeff`
optional string client_observed_server_side_session_id = 8
(Optional) Server-side generated idempotency key from the previous responses (if any). Server can use this to validate that the server side session has not changed.
optional user_context = 2
(Required) User context user_context.user_id and session+id both identify a unique remote spark session on the server side.
optional string operation_id = 6
(Optional) Provide an id for this request. If not provided, it will be generated by the server. It is returned in every ExecutePlanResponse.operation_id of the ExecutePlan response stream. The id must be an UUID string of the format `00112233-4455-6677-8899-aabbccddeeff`
optional plan = 3
(Required) The logical plan to be executed / analyzed.
optional string client_type = 4
Provides optional information about the client sending the request. This field can be used for language or version specific information and is only intended for logging purposes and will not be interpreted by the server.
repeated request_options = 5
Repeated element for options that can be passed to the request. This element is currently unused but allows to pass in an extension value used for arbitrary options.
repeated string tags = 7
Tags to tag the given execution with. Tags cannot contain ',' character and cannot be empty strings. Used by Interrupt with interrupt.tag.
rpc FetchErrorDetails (, )
base.proto:1162FetchErrorDetails retrieves the matched exception with details based on a provided error id.
message
base.proto:987string session_id = 1
(Required) The session_id specifies a Spark session for a user identified by user_context.user_id. The id should be a UUID string of the format `00112233-4455-6677-8899-aabbccddeeff`.
optional string client_observed_server_side_session_id = 5
(Optional) Server-side generated idempotency key from the previous responses (if any). Server can use this to validate that the server side session has not changed.
optional user_context = 2
User context
string error_id = 3
(Required) The id of the error.
optional string client_type = 4
Provides optional information about the client sending the request. This field can be used for language or version specific information and is only intended for logging purposes and will not be interpreted by the server.
message
base.proto:1014Next ID: 5
string server_side_session_id = 3
Server-side generated idempotency key that the client can use to assert that the server side session has not changed.
string session_id = 4
optional int32 root_error_idx = 1
The index of the root error in errors. The field will not be set if the error is not found.
repeated errors = 2
A list of errors.
rpc Interrupt (, )
base.proto:1141Interrupts running executions
message
base.proto:770string session_id = 1
(Required) The session_id specifies a spark session for a user id (which is specified by user_context.user_id). The session_id is set by the client to be able to collate streaming responses from different queries within the dedicated session. The id should be an UUID string of the format `00112233-4455-6677-8899-aabbccddeeff`
optional string client_observed_server_side_session_id = 7
(Optional) Server-side generated idempotency key from the previous responses (if any). Server can use this to validate that the server side session has not changed.
optional user_context = 2
(Required) User context
optional string client_type = 3
Provides optional information about the client sending the request. This field can be used for language or version specific information and is only intended for logging purposes and will not be interpreted by the server.
InterruptRequest.InterruptType interrupt_type = 4
(Required) The type of interrupt to execute.
oneof interrupt
string operation_tag = 5
if interrupt_tag == INTERRUPT_TYPE_TAG, interrupt operation with this tag.
string operation_id = 6
if interrupt_tag == INTERRUPT_TYPE_OPERATION_ID, interrupt operation with this operation_id.
message
base.proto:819Next ID: 4
string session_id = 1
Session id in which the interrupt was running.
string server_side_session_id = 3
Server-side generated idempotency key that the client can use to assert that the server side session has not changed.
repeated string interrupted_ids = 2
Operation ids of the executions which were interrupted.
rpc ReattachExecute (, stream )
base.proto:1147Reattach to an existing reattachable execution. The ExecutePlan must have been started with ReattachOptions.reattachable=true. If the ExecutePlanResponse stream ends without a ResultComplete message, there is more to continue. If there is a ResultComplete, the client should use ReleaseExecute with
message
base.proto:843string session_id = 1
(Required) The session_id of the request to reattach to. This must be an id of existing session.
optional string client_observed_server_side_session_id = 6
(Optional) Server-side generated idempotency key from the previous responses (if any). Server can use this to validate that the server side session has not changed.
optional user_context = 2
(Required) User context user_context.user_id and session+id both identify a unique remote spark session on the server side.
string operation_id = 3
(Required) Provide an id of the request to reattach to. This must be an id of existing operation.
optional string client_type = 4
Provides optional information about the client sending the request. This field can be used for language or version specific information and is only intended for logging purposes and will not be interpreted by the server.
optional string last_response_id = 5
(Optional) Last already processed response id from the response stream. After reattach, server will resume the response stream after that response. If not specified, server will restart the stream from the start. Note: server controls the amount of responses that it buffers and it may drop responses, that are far behind the latest returned response, so this can't be used to arbitrarily scroll back the cursor. If the response is no longer available, this will result in an error.
rpc ReleaseExecute (, )
base.proto:1153Release an reattachable execution, or parts thereof. The ExecutePlan must have been started with ReattachOptions.reattachable=true. Non reattachable executions are released automatically and immediately after the ExecutePlan RPC and ReleaseExecute may not be used.
message
base.proto:883string session_id = 1
(Required) The session_id of the request to reattach to. This must be an id of existing session.
optional string client_observed_server_side_session_id = 7
(Optional) Server-side generated idempotency key from the previous responses (if any). Server can use this to validate that the server side session has not changed.
optional user_context = 2
(Required) User context user_context.user_id and session+id both identify a unique remote spark session on the server side.
string operation_id = 3
(Required) Provide an id of the request to reattach to. This must be an id of existing operation.
optional string client_type = 4
Provides optional information about the client sending the request. This field can be used for language or version specific information and is only intended for logging purposes and will not be interpreted by the server.
oneof release
release_all = 5
release_until = 6
message
base.proto:932Next ID: 4
string session_id = 1
Session id in which the release was running.
string server_side_session_id = 3
Server-side generated idempotency key that the client can use to assert that the server side session has not changed.
optional string operation_id = 2
Operation id of the operation on which the release executed. If the operation couldn't be found (because e.g. it was concurrently released), will be unset. Otherwise, it will be equal to the operation_id from request.
rpc ReleaseSession (, )
base.proto:1159Release a session. All the executions in the session will be released. Any further requests for the session with that session_id for the given user_id will fail. If the session didn't exist or was already released, this is a noop.
message
base.proto:945string session_id = 1
(Required) The session_id of the request to reattach to. This must be an id of existing session.
optional user_context = 2
(Required) User context user_context.user_id and session+id both identify a unique remote spark session on the server side.
optional string client_type = 3
Provides optional information about the client sending the request. This field can be used for language or version specific information and is only intended for logging purposes and will not be interpreted by the server.
bool allow_reconnect = 4
Signals the server to allow the client to reconnect to the session after it is released. By default, the server tombstones the session upon release, preventing reconnections and fully cleaning the session state. If this flag is set to true, the server may permit the client to reconnect to the session post-release, even if the session state has been cleaned. This can result in missing state, such as Temporary Views, Temporary UDFs, or the Current Catalog, in the reconnected session. Use this option sparingly and only when the client fully understands the implications of reconnecting to a released session. The client must ensure that any queries executed do not rely on the session state prior to its release.
message
base.proto:979Next ID: 3
string session_id = 1
Session id of the session on which the release executed.
string server_side_session_id = 2
Server-side generated idempotency key that the client can use to assert that the server side session has not changed.
message
base.proto:642A chunk of an Artifact.
Used in: , ,
bytes data = 1
Data chunk.
int64 crc = 2
CRC to allow server to verify integrity of the chunk.
message
base.proto:664A number of `SingleChunkArtifact` batched into a single RPC.
Used in:
repeated artifacts = 1
message
base.proto:671Signals the beginning/start of a chunked artifact. A large artifact is transferred through a payload of `BeginChunkedArtifact` followed by a sequence of `ArtifactChunk`s.
Used in:
string name = 1
Name of the artifact undergoing chunking. Follows the same conventions as the `name` in the `Artifact` message.
int64 total_bytes = 2
Total size of the artifact in bytes.
int64 num_chunks = 3
Number of chunks the artifact is split into. This includes the `initial_chunk`.
optional initial_chunk = 4
The first/initial chunk.
message
base.proto:651An artifact that is contained in a single `ArtifactChunk`. Generally, this message represents tiny artifacts such as REPL-generated class files.
Used in:
string name = 1
The name of the artifact is expected in the form of a "Relative Path" that is made up of a sequence of directories and the final file element. Examples of "Relative Path"s: "jars/test.jar", "classes/xyz.class", "abc.xyz", "a/b/X.jar". The server is expected to maintain the hierarchy of files as defined by their name. (i.e The relative path of the file on the server's filesystem will be the same as the name of the provided artifact)
optional data = 2
A single data chunk.
message
base.proto:711Metadata of an artifact.
Used in:
string name = 1
bool is_crc_successful = 2
Whether the CRC (Cyclic Redundancy Check) is successful on server verification. The server discards any artifact that fails the CRC. If false, the client may choose to resend the artifact specified by `name`.
message
relations.proto:414Relation of type [[Aggregate]].
Used in:
optional input = 1
(Required) Input relation for a RelationalGroupedDataset.
Aggregate.GroupType group_type = 2
(Required) How the RelationalGroupedDataset was built.
repeated grouping_expressions = 3
(Required) Expressions for grouping keys
repeated aggregate_expressions = 4
(Required) List of values that will be translated to columns in the output DataFrame.
optional pivot = 5
(Optional) Pivots a column of the current `DataFrame` and performs the specified aggregation.
repeated grouping_sets = 6
(Optional) List of values that will be translated to columns in the output DataFrame.
Used in:
GROUP_TYPE_UNSPECIFIED = 0
GROUP_TYPE_GROUPBY = 1
GROUP_TYPE_ROLLUP = 2
GROUP_TYPE_CUBE = 3
GROUP_TYPE_PIVOT = 4
GROUP_TYPE_GROUPING_SETS = 5
message
relations.proto:453Used in:
repeated grouping_set = 1
(Required) Individual grouping set
message
relations.proto:442Used in:
optional col = 1
(Required) The column to pivot
repeated values = 2
(Optional) List of values that will be translated to columns in the output DataFrame. Note that if it is empty, the server side will immediately trigger a job to collect the distinct values of the column.
message
base.proto:186Used in:
string ddl_string = 1
(Required) The DDL formatted string to be parsed.
message
base.proto:129Explains the input plan based on a configurable mode.
Used in:
optional plan = 1
(Required) The logical plan to be analyzed.
Explain.ExplainMode explain_mode = 2
(Required) For analyzePlan rpc calls, configure the mode to explain plan in strings.
Plan explanation mode.
Used in:
EXPLAIN_MODE_UNSPECIFIED = 0
EXPLAIN_MODE_SIMPLE = 1
Generates only physical plan.
EXPLAIN_MODE_EXTENDED = 2
Generates parsed logical plan, analyzed logical plan, optimized logical plan and physical plan. Parsed Logical plan is a unresolved plan that extracted from the query. Analyzed logical plans transforms which translates unresolvedAttribute and unresolvedRelation into fully typed objects. The optimized logical plan transforms through a set of optimization rules, resulting in the physical plan.
EXPLAIN_MODE_CODEGEN = 3
Generates code for the statement, if any and a physical plan.
EXPLAIN_MODE_COST = 4
If plan node statistics are available, generates a logical plan and also the statistics.
EXPLAIN_MODE_FORMATTED = 5
Generates a physical plan outline and also node details.
message
base.proto:222Used in:
optional relation = 1
(Required) The logical plan to get the storage level.
message
base.proto:179Used in:
optional plan = 1
(Required) The logical plan to be analyzed.
message
base.proto:169Used in:
optional plan = 1
(Required) The logical plan to be analyzed.
message
base.proto:174Used in:
optional plan = 1
(Required) The logical plan to be analyzed.
message
base.proto:227Used in:
string json_string = 1
(Required) The JSON formatted string to be converted to DDL.
message
base.proto:206Used in:
optional relation = 1
(Required) The logical plan to persist.
optional storage_level = 2
(Optional) The storage level.
message
base.proto:193Returns `true` when the logical query plans are equal and therefore return same results.
Used in:
optional target_plan = 1
(Required) The plan to be compared.
optional other_plan = 2
(Required) The other plan to be compared.
message
base.proto:123Used in:
optional plan = 1
(Required) The logical plan to be analyzed.
message
base.proto:201Used in:
optional plan = 1
(Required) The logical plan to get a hashCode.
message
base.proto:184Used in:
(message has no fields)
message
base.proto:161Used in:
optional plan = 1
(Required) The logical plan to be analyzed.
optional int32 level = 2
(Optional) Max level of the schema.
message
base.proto:214Used in:
optional relation = 1
(Required) The logical plan to unpersist.
optional bool blocking = 2
(Optional) Whether to block until all blocks are deleted.
message
base.proto:288Used in:
optional parsed = 1
message
base.proto:263Used in:
string explain_string = 1
message
base.proto:304Used in:
optional storage_level = 1
(Required) The StorageLevel as a result of get_storage_level request.
message
base.proto:279Used in:
repeated string files = 1
A best-effort snapshot of the files that compose this Dataset
message
base.proto:271Used in:
bool is_local = 1
message
base.proto:275Used in:
bool is_streaming = 1
message
base.proto:309Used in:
string ddl_string = 1
message
base.proto:300Used in:
(message has no fields)
message
base.proto:292Used in:
bool result = 1
message
base.proto:259Used in:
optional schema = 1
message
base.proto:296Used in:
int32 result = 1
message
base.proto:284Used in:
string version = 1
message
base.proto:267Used in:
string tree_string = 1
message
base.proto:302Used in:
(message has no fields)
message
relations.proto:1097Used in:
optional input = 1
(Required) Input relation for applyInPandasWithState.
repeated grouping_expressions = 2
(Required) Expressions for grouping keys.
optional func = 3
(Required) Input user-defined function.
string output_schema = 4
(Required) Schema for the output DataFrame.
string state_schema = 5
(Required) Schema for the state.
string output_mode = 6
(Required) The output mode of the function.
string timeout_conf = 7
(Required) Timeout configuration for groups that do not receive data for a while.
message
base.proto:764Used in:
bool exists = 1
Exists or not particular artifact at the server.
message
relations.proto:1201Relation of type [[AsOfJoin]]. `left` and `right` must be present.
Used in:
optional left = 1
(Required) Left input relation for a Join.
optional right = 2
(Required) Right input relation for a Join.
optional left_as_of = 3
(Required) Field to join on in left DataFrame
optional right_as_of = 4
(Required) Field to join on in right DataFrame
optional join_expr = 5
(Optional) The join condition. Could be unset when `using_columns` is utilized. This field does not co-exist with using_columns.
repeated string using_columns = 6
Optional. using_columns provides a list of columns that should present on both sides of the join inputs that this Join will join on. For example A JOIN B USING col_name is equivalent to A JOIN B on A.col_name = B.col_name. This field does not co-exist with join_condition.
string join_type = 7
(Required) The join type.
optional tolerance = 8
(Optional) The asof tolerance within this range.
bool allow_exact_matches = 9
(Required) Whether allow matching with the same value or not.
string direction = 10
(Required) Whether to search for prior, subsequent, or closest matches.
message
common.proto:172Used in:
repeated bool values = 1
message
catalog.proto:222See `spark.catalog.cacheTable`
Used in:
string table_name = 1
(Required)
optional storage_level = 2
(Optional)
message
relations.proto:520A local relation that has been cached already.
Used in:
string hash = 3
(Required) A sha-256 hash of the serialized local relation in proto, see LocalRelation.
message
relations.proto:532Represents a remote relation that has been cached on server.
Used in: , ,
string relation_id = 1
(Required) ID of the remote related (assigned by the service).
message
expressions.proto:453Used in:
string function_name = 1
(Required) Unparsed name of the SQL function.
repeated arguments = 2
(Optional) Function arguments. Empty arguments are allowed.
message
catalog.proto:51Catalog messages are marked as unstable.
Used in:
oneof cat_type
current_database = 1
set_current_database = 2
list_databases = 3
list_tables = 4
list_functions = 5
list_columns = 6
get_database = 7
get_table = 8
get_function = 9
database_exists = 10
table_exists = 11
function_exists = 12
create_external_table = 13
create_table = 14
drop_temp_view = 15
drop_global_temp_view = 16
recover_partitions = 17
is_cached = 18
cache_table = 19
uncache_table = 20
clear_cache = 21
refresh_table = 22
refresh_by_path = 23
current_catalog = 24
set_current_catalog = 25
list_catalogs = 26
message
commands.proto:526Used in:
optional relation = 1
(Required) The logical plan to checkpoint.
bool local = 2
(Required) Locally checkpoint using a local temporary directory in Spark Connect server (Spark Driver)
bool eager = 3
(Required) Whether to checkpoint this dataframe immediately.
optional storage_level = 4
(Optional) For local checkpoint, the storage level to use.
message
base.proto:1114Used in:
optional relation = 1
(Required) The logical plan checkpointed.
message
catalog.proto:237See `spark.catalog.clearCache`
Used in:
(message has no fields)
message
relations.proto:1074Used in:
optional input = 1
(Required) One input relation for CoGroup Map API - applyInPandas.
repeated input_grouping_expressions = 2
Expressions for grouping keys of the first input relation.
optional other = 3
(Required) The other input relation.
repeated other_grouping_expressions = 4
Expressions for grouping keys of the other input relation.
optional func = 5
(Required) Input user-defined function.
repeated input_sorting_expressions = 6
(Optional) Expressions for sorting. Only used by Scala Sorted CoGroup Map API.
repeated other_sorting_expressions = 7
(Optional) Expressions for sorting. Only used by Scala Sorted CoGroup Map API.
message
relations.proto:1169Collect arbitrary (named) metrics from a dataset.
Used in:
optional input = 1
(Required) The input relation.
string name = 2
(Required) Name of the metrics.
repeated metrics = 3
(Required) The metric sequence.
message
commands.proto:56A [[Command]] is an operation that is executed by the server that does not directly consume or produce a relational result.
Used in:
oneof command_type
register_function = 1
write_operation = 2
create_dataframe_view = 3
write_operation_v2 = 4
sql_command = 5
write_stream_operation_start = 6
streaming_query_command = 7
get_resources_command = 8
streaming_query_manager_command = 9
register_table_function = 10
streaming_query_listener_bus_command = 11
register_data_source = 12
create_resource_profile_command = 13
checkpoint_command = 14
remove_cached_remote_relation_command = 15
merge_into_table_command = 16
ml_command = 17
execute_external_command = 18
pipeline_command = 19
extension = 999
This field is used to mark extensions to the protocol. When plugins generate arbitrary Commands they can add them here. During the planning the correct resolution is done.
message
relations.proto:1150Used in: ,
string name = 1
(Required) Name of the data source.
oneof data_source
(Required) The data source type.
python_data_source = 2
message
expressions.proto:394Used in: , , , , ,
string function_name = 1
(Required) Name of the user-defined function.
bool deterministic = 2
(Optional) Indicate if the user-defined function is deterministic.
repeated arguments = 3
(Optional) Function arguments. Empty arguments are allowed.
oneof function
(Required) Indicate the function type of the user-defined function.
python_udf = 4
scalar_scala_udf = 5
java_udf = 6
bool is_distinct = 7
(Required) Indicate if this function should be applied on distinct values.
message
relations.proto:1120Used in: ,
string function_name = 1
(Required) Name of the user-defined table function.
bool deterministic = 2
(Optional) Whether the user-defined table function is deterministic.
repeated arguments = 3
(Optional) Function input arguments. Empty arguments are allowed.
oneof function
(Required) Type of the user-defined table function.
python_udtf = 4
message
base.proto:565Used in:
repeated string keys = 1
(Required) The config keys to get.
message
base.proto:580Used in:
optional string prefix = 1
(Optional) The prefix of the config key to get.
message
base.proto:575Used in:
repeated string keys = 1
(Required) The config keys to get optionally.
message
base.proto:570Used in:
repeated pairs = 1
(Required) The config key-value pairs to get. The value will be used as the default value.
message
base.proto:590Used in:
repeated string keys = 1
(Required) The config keys to check the config is modifiable.
message
base.proto:545Used in:
oneof op_type
set = 1
get = 2
get_with_default = 3
get_option = 4
get_all = 5
unset = 6
is_modifiable = 7
message
base.proto:557Used in:
repeated pairs = 1
(Required) The config key-value pairs to set.
optional bool silent = 2
(Optional) Whether to ignore failures.
message
base.proto:585Used in:
repeated string keys = 1
(Required) The config keys to unset.
message
commands.proto:114A command that can create DataFrame global temp view or local temp view.
Used in:
optional input = 1
(Required) The relation that this view will be built on.
string name = 2
(Required) View name.
bool is_global = 3
(Required) Whether this is global temp view or local temp view.
bool replace = 4
(Required) If true, and if the view already exists, updates it; if false, and if the view already exists, throws exception.
message
catalog.proto:166See `spark.catalog.createExternalTable`
Used in:
string table_name = 1
(Required)
optional string path = 2
(Optional)
optional string source = 3
(Optional)
optional schema = 4
(Optional)
map<string, string> options = 5
Options could be empty for valid data source format. The map key is case insensitive.
message
commands.proto:509Command to create ResourceProfile
Used in:
optional profile = 1
(Required) The ResourceProfile to be built on the server-side.
message
commands.proto:515Response for command 'CreateResourceProfileCommand'.
Used in:
int32 profile_id = 1
(Required) Server-side generated resource profile id.
message
catalog.proto:181See `spark.catalog.createTable`
Used in:
string table_name = 1
(Required)
optional string path = 2
(Optional)
optional string source = 3
(Optional)
optional string description = 4
(Optional)
optional schema = 5
(Optional)
map<string, string> options = 6
Options could be empty for valid data source format. The map key is case insensitive.
message
catalog.proto:252See `spark.catalog.currentCatalog`
Used in:
(message has no fields)
message
catalog.proto:83See `spark.catalog.currentDatabase`
Used in:
(message has no fields)
message
types.proto:49This message describes the logical [[DataType]] of something. It does not carry the value itself but only describes it.
Used in: , , , , , , , , , , , , , , , , , , , , , , , ,
oneof kind
null = 1
binary = 2
boolean = 3
byte = 4
Numeric types
short = 5
integer = 6
long = 7
float = 8
double = 9
decimal = 10
string = 11
String types
char = 12
var_char = 13
date = 14
Datatime types
timestamp = 15
timestamp_ntz = 16
calendar_interval = 17
Interval types
year_month_interval = 18
day_time_interval = 19
array = 20
Complex types
struct = 21
map = 22
variant = 25
udt = 23
UserDefinedType
unparsed = 24
UnparsedDataType
message
types.proto:193Used in:
optional element_type = 1
bool contains_null = 2
uint32 type_variation_reference = 3
message
types.proto:128Used in:
uint32 type_variation_reference = 1
message
types.proto:95Used in:
uint32 type_variation_reference = 1
message
types.proto:99Used in:
uint32 type_variation_reference = 1
message
types.proto:148Used in:
uint32 type_variation_reference = 1
message
types.proto:165Start compound types.
Used in:
int32 length = 1
uint32 type_variation_reference = 2
message
types.proto:140Used in:
uint32 type_variation_reference = 1
message
types.proto:158Used in:
optional int32 start_field = 1
optional int32 end_field = 2
uint32 type_variation_reference = 3
message
types.proto:175Used in:
optional int32 scale = 1
optional int32 precision = 2
uint32 type_variation_reference = 3
message
types.proto:119Used in:
uint32 type_variation_reference = 1
message
types.proto:115Used in:
uint32 type_variation_reference = 1
message
types.proto:107Used in:
uint32 type_variation_reference = 1
message
types.proto:111Used in:
uint32 type_variation_reference = 1
message
types.proto:199Used in:
optional key_type = 1
optional value_type = 2
bool value_contains_null = 3
uint32 type_variation_reference = 4
message
types.proto:132Used in:
uint32 type_variation_reference = 1
message
types.proto:103Used in:
uint32 type_variation_reference = 1
message
types.proto:123Used in:
uint32 type_variation_reference = 1
string collation = 2
message
types.proto:188Used in:
repeated fields = 1
uint32 type_variation_reference = 2
message
types.proto:181Used in:
string name = 1
optional data_type = 2
bool nullable = 3
optional string metadata = 4
message
types.proto:136Used in:
uint32 type_variation_reference = 1
message
types.proto:144Used in:
uint32 type_variation_reference = 1
message
types.proto:210Used in:
string type = 1
optional string jvm_class = 2
Required for Scala/Java UDT
optional string python_class = 3
Required for Python UDT
optional string serialized_python_class = 4
Required for Python UDT
optional sql_type = 5
Required for Python UDT
message
types.proto:222Used in:
string data_type_string = 1
(Required) The unparsed data type string
message
types.proto:170Used in:
int32 length = 1
uint32 type_variation_reference = 2
message
types.proto:206Used in:
uint32 type_variation_reference = 1
message
types.proto:152Used in:
optional int32 start_field = 1
optional int32 end_field = 2
uint32 type_variation_reference = 3
message
catalog.proto:144See `spark.catalog.databaseExists`
Used in:
string db_name = 1
(Required)
The type of dataset.
Used in:
DATASET_TYPE_UNSPECIFIED = 0
Safe default value. Should not be used.
MATERIALIZED_VIEW = 1
A materialized view dataset which is published to the catalog
TABLE = 2
A table which is published to the catalog
TEMPORARY_VIEW = 3
A view which is not published to the catalog
message
relations.proto:487Relation of type [[Deduplicate]] which have duplicate rows removed, could consider either only the subset of columns or all the columns.
Used in:
optional input = 1
(Required) Input relation for a Deduplicate.
repeated string column_names = 2
(Optional) Deduplicate based on a list of column names. This field does not co-use with `all_columns_as_keys`.
optional bool all_columns_as_keys = 3
(Optional) Deduplicate based on all the columns of the input relation. This field does not co-use with `column_names`.
optional bool within_watermark = 4
(Optional) Deduplicate within the time range of watermark.
message
pipelines.proto:142Parses the SQL file and registers all datasets and flows.
Used in:
optional string dataflow_graph_id = 1
The graph to attach this dataset to.
optional string sql_file_path = 2
The full path to the SQL file. Can be relative or absolute.
optional string sql_text = 3
The contents of the SQL file.
message
common.proto:188Used in:
repeated double values = 1
message
relations.proto:473Drop specified columns.
Used in:
optional input = 1
(Required) The input relation.
repeated columns = 2
(Optional) columns to drop.
repeated string column_names = 3
(Optional) names of columns to drop.
message
catalog.proto:204See `spark.catalog.dropGlobalTempView`
Used in:
string view_name = 1
(Required)
message
catalog.proto:198See `spark.catalog.dropTempView`
Used in:
string view_name = 1
(Required)
message
example_plugins.proto:61string custom_field = 1
message
example_plugins.proto:56optional child = 1
string custom_field = 2
message
example_plugins.proto:50optional input = 1
string custom_field = 2
message
commands.proto:565Execute an arbitrary string command inside an external execution engine
Used in:
string runner = 1
(Required) The class name of the runner that implements `ExternalCommandRunner`
string command = 2
(Required) The target command to be executed.
map<string, string> options = 3
(Optional) The options for the runner.
message
base.proto:354Used in:
oneof request_option
reattach_options = 1
extension = 999
Extension type for request options
message
base.proto:371The response of a query, can be one or more for each request. Responses belonging to the same input query, carry the same `session_id`. Next ID: 17
Used as response type in: SparkConnectService.ExecutePlan, SparkConnectService.ReattachExecute
string session_id = 1
string server_side_session_id = 15
Server-side generated idempotency key that the client can use to assert that the server side session has not changed.
string operation_id = 12
Identifies the ExecutePlan execution. If set by the client in ExecutePlanRequest.operationId, that value is returned. Otherwise generated by the server. It is an UUID string of the format `00112233-4455-6677-8899-aabbccddeeff`
string response_id = 13
Identified the response in the stream. The id is an UUID string of the format `00112233-4455-6677-8899-aabbccddeeff`
oneof response_type
Union type for the different response messages.
arrow_batch = 2
sql_command_result = 5
Special case for executing SQL commands.
write_stream_operation_start_result = 8
Response for a streaming query.
streaming_query_command_result = 9
Response for commands on a streaming query.
get_resources_command_result = 10
Response for 'SparkContext.resources'.
streaming_query_manager_command_result = 11
Response for commands on the streaming query manager.
streaming_query_listener_events_result = 16
Response for commands on the client side streaming query listener.
result_complete = 14
Response type informing if the stream is complete in reattachable execution.
create_resource_profile_command_result = 17
Response for command that creates ResourceProfile.
execution_progress = 18
(Optional) Intermediate query progress reports.
checkpoint_command_result = 19
Response for command that checkpoints a DataFrame.
ml_command_result = 20
ML command response
pipeline_event_result = 21
Response containing pipeline event that is streamed back to the client during a pipeline run
pipeline_command_result = 22
Pipeline command response
extension = 999
Support arbitrary result objects.
optional metrics = 4
Metrics for the query execution. Typically, this field is only present in the last batch of results and then represent the overall state of the query execution.
repeated observed_metrics = 6
The metrics observed during the execution of the query plan.
optional schema = 7
(Optional) The Spark schema. This field is available when `collect` is called.
message
base.proto:451Batch results of metrics.
Used in:
int64 row_count = 1
Count rows in `data`. Must match the number of rows inside `data`.
bytes data = 2
Serialized Arrow data.
optional int64 start_offset = 3
If set, row offset of the start of this ArrowBatch in execution results.
message
base.proto:493This message is used to communicate progress about the query progress during the execution.
Used in:
repeated stages = 1
Captures the progress of each individual stage.
int64 num_inflight_tasks = 2
Captures the currently in progress tasks.
message
base.proto:500Used in:
int64 stage_id = 1
int64 num_tasks = 2
int64 num_completed_tasks = 3
int64 input_bytes_read = 4
bool done = 5
message
base.proto:461Used in:
repeated metrics = 1
message
base.proto:465Used in:
string name = 1
int64 plan_id = 2
int64 parent = 3
map<string, > execution_metrics = 4
message
base.proto:472Used in:
string name = 1
int64 value = 2
string metric_type = 3
message
base.proto:479Used in:
string name = 1
repeated values = 2
repeated string keys = 3
int64 plan_id = 4
message
base.proto:486If present, in a reattachable execution this means that after server sends onComplete, the execution is complete. If the server sends onComplete without sending a ResultComplete, it means that there is more, and the client should use ReattachExecute RPC to continue.
Used in:
(message has no fields)
message
base.proto:446A SQL command returns an opaque Relation that can be directly used as input for the next call.
Used in:
optional relation = 1
message
common.proto:72An executor resource request.
Used in:
string resource_name = 1
(Required) resource name.
int64 amount = 2
(Required) resource amount requesting.
optional string discovery_script = 3
Optional script used to discover the resources.
optional string vendor = 4
Optional vendor, required for some cluster managers.
message
expressions.proto:53Expression used to refer to fields, functions and similar. This can be used everywhere expressions in SQL appear.
Used in: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ,
optional common = 18
oneof expr_type
literal = 1
unresolved_attribute = 2
unresolved_function = 3
expression_string = 4
unresolved_star = 5
alias = 6
cast = 7
unresolved_regex = 8
sort_order = 9
lambda_function = 10
window = 11
unresolved_extract_value = 12
update_fields = 13
unresolved_named_lambda_variable = 14
common_inline_user_defined_function = 15
call_function = 16
named_argument_expression = 17
merge_action = 19
typed_aggregate_expression = 20
subquery_expression = 21
extension = 999
This field is used to mark extensions to the protocol. When plugins generate arbitrary relations they can add them here. During the planning the correct resolution is done.
message
expressions.proto:358Used in: ,
optional expr = 1
(Required) The expression that alias will be added on.
repeated string name = 2
(Required) a list of name parts for the alias. Scalar columns only has one name that presents.
optional string metadata = 3
(Optional) Alias metadata expressed as a JSON map.
message
expressions.proto:166Used in:
optional expr = 1
(Required) the expression to be casted.
oneof cast_to_type
(Required) the data type that the expr to be casted to.
type = 2
string type_str = 3
If this is set, Server will use Catalyst parser to parse this string to DataType.
Cast.EvalMode eval_mode = 4
(Optional) The expression evaluation mode.
Used in:
EVAL_MODE_UNSPECIFIED = 0
EVAL_MODE_LEGACY = 1
EVAL_MODE_ANSI = 2
EVAL_MODE_TRY = 3
message
expressions.proto:305Expression as string.
Used in:
string expression = 1
(Required) A SQL expression that will be parsed by Catalyst parser.
message
expressions.proto:371Used in:
optional function = 1
(Required) The lambda function. The function body should use 'UnresolvedAttribute' as arguments, the sever side will replace 'UnresolvedAttribute' with 'UnresolvedNamedLambdaVariable'.
repeated arguments = 2
(Required) Function variables. Must contains 1 ~ 3 variables.
message
expressions.proto:188Used in: , , , , , , , , , , , , ,
oneof literal_type
null = 1
bytes binary = 2
bool boolean = 3
int32 byte = 4
int32 short = 5
int32 integer = 6
int64 long = 7
float float = 10
double double = 11
decimal = 12
string string = 13
int32 date = 16
Date in units of days since the UNIX epoch.
int64 timestamp = 17
Timestamp in units of microseconds since the UNIX epoch.
int64 timestamp_ntz = 18
Timestamp in units of microseconds since the UNIX epoch (without timezone information).
calendar_interval = 19
int32 year_month_interval = 20
int64 day_time_interval = 21
array = 22
map = 23
struct = 24
specialized_array = 25
message
expressions.proto:237Used in:
optional element_type = 1
repeated elements = 2
message
expressions.proto:231Used in:
int32 months = 1
int32 days = 2
int64 microseconds = 3
message
expressions.proto:221Used in:
string value = 1
the string representation.
optional int32 precision = 2
The maximum number of digits allowed in the value. the maximum precision is 38.
optional int32 scale = 3
declared scale of decimal literal
message
expressions.proto:242Used in:
optional key_type = 1
optional value_type = 2
repeated keys = 3
repeated values = 4
message
expressions.proto:254Used in:
oneof value_type
bools = 1
ints = 2
longs = 3
floats = 4
doubles = 5
strings = 6
message
expressions.proto:249Used in:
optional struct_type = 1
repeated elements = 2
message
expressions.proto:143SortOrder is used to specify the data ordering, it is normally used in Sort and Window. It is an unevaluable expression and cannot be evaluated, so can not be used in Projection.
Used in: , , ,
optional child = 1
(Required) The expression to be sorted.
SortOrder.SortDirection direction = 2
(Required) The sort direction, should be ASCENDING or DESCENDING.
SortOrder.NullOrdering null_ordering = 3
(Required) How to deal with NULLs, should be NULLS_FIRST or NULLS_LAST.
Used in:
SORT_NULLS_UNSPECIFIED = 0
SORT_NULLS_FIRST = 1
SORT_NULLS_LAST = 2
Used in:
SORT_DIRECTION_UNSPECIFIED = 0
SORT_DIRECTION_ASCENDING = 1
SORT_DIRECTION_DESCENDING = 2
message
expressions.proto:268An unresolved attribute that is not explicitly bound to a specific column, but the column is resolved during analysis by name.
Used in:
string unparsed_identifier = 1
(Required) An identifier that will be parsed by Catalyst parser. This should follow the Spark SQL identifier syntax.
optional int64 plan_id = 2
(Optional) The id of corresponding connect plan.
optional bool is_metadata_column = 3
(Optional) The requested column is a metadata column.
message
expressions.proto:334Extracts a value or values from an Expression
Used in:
optional child = 1
(Required) The expression to extract value from, can be Map, Array, Struct or array of Structs.
optional extraction = 2
(Required) The expression to describe the extraction, can be key of Map, index of Array, field name of Struct.
message
expressions.proto:282An unresolved function is not explicitly bound to one explicit function, but the function is resolved during analysis following Sparks name resolution rules.
Used in:
string function_name = 1
(Required) name (or unparsed name for user defined function) for the unresolved function.
repeated arguments = 2
(Optional) Function arguments. Empty arguments are allowed.
bool is_distinct = 3
(Required) Indicate if this function should be applied on distinct values.
bool is_user_defined_function = 4
(Required) Indicate if this is a user defined function. When it is not a user defined function, Connect will use the function name directly. When it is a user defined function, Connect will parse the function name first.
optional bool is_internal = 5
(Optional) Indicate if this function is defined in the internal function registry. If not set, the server will try to look up the function in the internal function registry and decide appropriately.
message
expressions.proto:382Used in: ,
repeated string name_parts = 1
(Required) a list of name parts for the variable. Must not be empty.
message
expressions.proto:325Represents all of the input attributes to a given relational operator, for example in "SELECT `(id)?+.+` FROM ...".
Used in:
string col_name = 1
(Required) The column name used to extract column with regex.
optional int64 plan_id = 2
(Optional) The id of corresponding connect plan.
message
expressions.proto:311UnresolvedStar is used to expand all the fields of a relation or struct.
Used in:
optional string unparsed_target = 1
(Optional) The target of the expansion. If set, it should end with '.*' and will be parsed by 'parseAttributeName' in the server side.
optional int64 plan_id = 2
(Optional) The id of corresponding connect plan.
message
expressions.proto:345Add, replace or drop a field of `StructType` expression by name.
Used in:
optional struct_expression = 1
(Required) The struct expression.
string field_name = 2
(Required) The field name.
optional value_expression = 3
(Optional) The expression to add or replace. When not set, it means this field will be dropped.
message
expressions.proto:85Expression for the OVER clause or WINDOW clause.
Used in:
optional window_function = 1
(Required) The window function.
repeated partition_spec = 2
(Optional) The way that input rows are partitioned.
repeated order_spec = 3
(Optional) Ordering of rows in a partition.
optional frame_spec = 4
(Optional) Window frame in a partition. If not set, it will be treated as 'UnspecifiedFrame'.
message
expressions.proto:102The window frame
Used in:
WindowFrame.FrameType frame_type = 1
(Required) The type of the frame.
optional lower = 2
(Required) The lower bound of the frame.
optional upper = 3
(Required) The upper bound of the frame.
message
expressions.proto:124Used in:
oneof boundary
bool current_row = 1
CURRENT ROW boundary
bool unbounded = 2
UNBOUNDED boundary. For lower bound, it will be converted to 'UnboundedPreceding'. for upper bound, it will be converted to 'UnboundedFollowing'.
value = 3
This is an expression for future proofing. We are expecting literals on the server side.
Used in:
FRAME_TYPE_UNDEFINED = 0
FRAME_TYPE_ROW = 1
RowFrame treats rows in a partition individually.
FRAME_TYPE_RANGE = 2
RangeFrame treats rows in a partition as groups of peers. All rows having the same 'ORDER BY' ordering are considered as peers.
message
expressions.proto:389Used in:
optional origin = 1
(Required) Keep the information of the origin for this expression such as stacktrace.
message
relations.proto:160Message for fetching attribute from object on the server side. Fetch can be represented as a Relation or a ML command Command: model.coefficients, model.summary.weightedPrecision which returns the final literal result Relation: model.summary.roc which returns a DataFrame (Relation)
Used in: ,
optional obj_ref = 1
(Required) reference to the object on the server side
repeated methods = 2
(Required) the calling method chains
message
relations.proto:167Represents a method with inclusion of method name and its arguments
Used in:
string method = 1
(Required) the method name
repeated args = 2
(Optional) the arguments of the method
message
relations.proto:173Used in:
oneof args_type
param = 1
input = 2
message
base.proto:1095Error defines the schema for the representing exception.
Used in:
repeated string error_type_hierarchy = 1
The fully qualified names of the exception class and its parent classes.
string message = 2
The detailed message of the exception.
repeated stack_trace = 3
The stackTrace of the exception. It will be set if the SQLConf spark.sql.connect.serverStacktrace.enabled is true.
optional int32 cause_idx = 4
The index of the cause error in errors.
optional spark_throwable = 5
The structured data of a SparkThrowable exception.
message
base.proto:1044QueryContext defines the schema for the query context of a SparkThrowable. It helps users understand where the error occurs while executing queries.
Used in:
QueryContext.ContextType context_type = 10
string object_type = 1
The object type of the query which throws the exception. If the exception is directly from the main query, it should be an empty string. Otherwise, it should be the exact object type in upper case. For example, a "VIEW".
string object_name = 2
The object name of the query which throws the exception. If the exception is directly from the main query, it should be an empty string. Otherwise, it should be the object name. For example, a view name "V1".
int32 start_index = 3
The starting index in the query text which throws the exception. The index starts from 0.
int32 stop_index = 4
The stopping index in the query which throws the exception. The index starts from 0.
string fragment = 5
The corresponding fragment of the query which throws the exception.
string call_site = 6
The user code (call site of the API) that caused throwing the exception.
string summary = 7
Summary of the exception cause.
The type of this query context.
Used in:
SQL = 0
DATAFRAME = 1
message
base.proto:1079SparkThrowable defines the schema for SparkThrowable exceptions.
Used in:
optional string error_class = 1
Succinct, human-readable, unique, and consistent representation of the error category.
map<string, string> message_parameters = 2
The message parameters for the error framework.
repeated query_contexts = 3
The query context of a SparkThrowable.
optional string sql_state = 4
Portable error identifier across SQL engines If null, error class or SQLSTATE is not set.
message
base.proto:1028Used in:
string declaring_class = 1
The fully qualified name of the class containing the execution point.
string method_name = 2
The name of the method containing the execution point.
optional string file_name = 3
The name of the file containing the execution point.
int32 line_number = 4
The line number of the source line containing the execution point.
message
relations.proto:295Relation that applies a boolean expression `condition` on each row of `input` to produce the output result.
Used in:
optional input = 1
(Required) Input relation for a Filter.
optional condition = 2
(Required) A Filter must have a condition expression.
message
common.proto:184Used in:
repeated float values = 1
message
catalog.proto:158See `spark.catalog.functionExists`
Used in:
string function_name = 1
(Required)
optional string db_name = 2
(Optional)
message
catalog.proto:122See `spark.catalog.getDatabase`
Used in:
string db_name = 1
(Required)
message
catalog.proto:136See `spark.catalog.getFunction`
Used in:
string function_name = 1
(Required)
optional string db_name = 2
(Optional)
message
commands.proto:501Command to get the output of 'SparkContext.resources'
Used in:
(message has no fields)
message
commands.proto:504Response for command 'GetResourcesCommand'.
Used in:
map<string, > resources = 1
message
catalog.proto:128See `spark.catalog.getTable`
Used in:
string table_name = 1
(Required)
optional string db_name = 2
(Optional)
message
relations.proto:1024Used in:
optional input = 1
(Required) Input relation for Group Map API: apply, applyInPandas.
repeated grouping_expressions = 2
(Required) Expressions for grouping keys.
optional func = 3
(Required) Input user-defined function.
repeated sorting_expressions = 4
(Optional) Expressions for sorting. Only used by Scala Sorted Group Map API.
optional initial_input = 5
Below fields are only used by (Flat)MapGroupsWithState (Optional) Input relation for initial State.
repeated initial_grouping_expressions = 6
(Optional) Expressions for grouping keys of the initial state input relation.
optional bool is_map_groups_with_state = 7
(Optional) True if MapGroupsWithState, false if FlatMapGroupsWithState.
optional string output_mode = 8
(Optional) The output mode of the function.
optional string timeout_conf = 9
(Optional) Timeout configuration for groups that do not receive data for a while.
optional state_schema = 10
(Optional) The schema for the grouped state.
optional transform_with_state_info = 11
Below fields are used by TransformWithState and TransformWithStateInPandas (Optional) TransformWithState related parameters.
message
relations.proto:932Specify a hint over a relation. Hint should have a name and optional parameters.
Used in:
optional input = 1
(Required) The input relation.
string name = 2
(Required) Hint name. Supported Join hints include BROADCAST, MERGE, SHUFFLE_HASH, SHUFFLE_REPLICATE_NL. Supported partitioning hints include COALESCE, REPARTITION, REPARTITION_BY_RANGE.
repeated parameters = 3
(Optional) Hint parameters.
message
relations.proto:621Compose the string representing rows for output. It will invoke 'Dataset.htmlString' to compute the results.
Used in:
optional input = 1
(Required) The input relation.
int32 num_rows = 2
(Required) Number of rows to show.
int32 truncate = 3
(Required) If set to more than 0, truncates strings to `truncate` characters and all cells will be aligned right.
Used in:
INTERRUPT_TYPE_UNSPECIFIED = 0
INTERRUPT_TYPE_ALL = 1
Interrupt all running executions within the session with the provided session_id.
INTERRUPT_TYPE_TAG = 2
Interrupt all running executions within the session with the provided operation_tag.
INTERRUPT_TYPE_OPERATION_ID = 3
Interrupt the running execution within the session with the provided operation_id.
message
common.proto:176Used in:
repeated int32 values = 1
message
catalog.proto:216See `spark.catalog.isCached`
Used in:
string table_name = 1
(Required)
message
expressions.proto:437Used in:
string class_name = 1
(Required) Fully qualified name of Java class
optional output_type = 2
(Optional) Output type of the Java UDF
bool aggregate = 3
(Required) Indicate if the Java user-defined function is an aggregate function
message
relations.proto:306Relation of type [[Join]]. `left` and `right` must be present.
Used in:
optional left = 1
(Required) Left input relation for a Join.
optional right = 2
(Required) Right input relation for a Join.
optional join_condition = 3
(Optional) The join condition. Could be unset when `using_columns` is utilized. This field does not co-exist with using_columns.
Join.JoinType join_type = 4
(Required) The join type.
repeated string using_columns = 5
Optional. using_columns provides a list of columns that should present on both sides of the join inputs that this Join will join on. For example A JOIN B USING col_name is equivalent to A JOIN B on A.col_name = B.col_name. This field does not co-exist with join_condition.
optional join_data_type = 6
(Optional) Only used by joinWith. Set the left and right join data types.
message
relations.proto:342Used in:
bool is_left_struct = 1
If the left data type is a struct.
bool is_right_struct = 2
If the right data type is a struct.
Used in: ,
JOIN_TYPE_UNSPECIFIED = 0
JOIN_TYPE_INNER = 1
JOIN_TYPE_FULL_OUTER = 2
JOIN_TYPE_LEFT_OUTER = 3
JOIN_TYPE_RIGHT_OUTER = 4
JOIN_TYPE_LEFT_ANTI = 5
JOIN_TYPE_LEFT_SEMI = 6
JOIN_TYPE_CROSS = 7
message
common.proto:122Used in:
optional int32 line = 1
(Optional) Line number in the source file.
optional int32 start_position = 2
(Optional) Start position in the source file.
optional int32 start_index = 3
(Optional) Start index in the source file.
optional int32 stop_index = 4
(Optional) Stop index in the source file.
optional string sql_text = 5
(Optional) SQL text.
optional string object_type = 6
(Optional) Object type.
optional string object_name = 7
(Optional) Object name.
repeated stack_trace = 8
(Optional) Stack trace.
message
base.proto:511The key-value pair for the config request and response.
Used in: , ,
string key = 1
(Required) The key.
optional string value = 2
(Optional) The value.
message
relations.proto:1242Relation of type [[LateralJoin]]. `left` and `right` must be present.
Used in:
optional left = 1
(Required) Left input relation for a Join.
optional right = 2
(Required) Right input relation for a Join.
optional join_condition = 3
(Optional) The join condition.
Join.JoinType join_type = 4
(Required) The join type.
message
relations.proto:386Relation of type [[Limit]] that is used to `limit` rows from the input relation.
Used in:
optional input = 1
(Required) Input relation for a Limit.
int32 limit = 2
(Required) the limit.
message
catalog.proto:261See `spark.catalog.listCatalogs`
Used in:
optional string pattern = 1
(Optional) The pattern that the catalog name needs to match
message
catalog.proto:114See `spark.catalog.listColumns`
Used in:
string table_name = 1
(Required)
optional string db_name = 2
(Optional)
message
catalog.proto:92See `spark.catalog.listDatabases`
Used in:
optional string pattern = 1
(Optional) The pattern that the database name needs to match
message
catalog.proto:106See `spark.catalog.listFunctions`
Used in:
optional string db_name = 1
(Optional)
optional string pattern = 2
(Optional) The pattern that the function name needs to match
message
catalog.proto:98See `spark.catalog.listTables`
Used in:
optional string db_name = 1
(Optional)
optional string pattern = 2
(Optional) The pattern that the table name needs to match
message
relations.proto:506A relation that does not need to be qualified by name.
Used in:
optional bytes data = 1
(Optional) Local collection data serialized into Arrow IPC streaming format which contains the schema of the data.
optional string schema = 2
(Optional) The schema of local data. It should be either a DDL-formatted type string or a JSON string. The server side will update the column names and data types according to this schema. If the 'data' is not provided, then this schema will be required.
message
common.proto:180Used in:
repeated int64 values = 1
message
relations.proto:1010Used in:
optional input = 1
(Required) Input relation for a mapPartitions-equivalent API: mapInPandas, mapInArrow.
optional func = 2
(Required) Input user-defined function.
optional bool is_barrier = 3
(Optional) Whether to use barrier mode execution or not.
optional int32 profile_id = 4
(Optional) ResourceProfile id used for the stage level scheduling.
message
expressions.proto:469Used in:
MergeAction.ActionType action_type = 1
(Required) The action type of the merge action.
optional condition = 2
(Optional) The condition expression of the merge action.
repeated assignments = 3
(Optional) The assignments of the merge action. Required for ActionTypes INSERT and UPDATE.
Used in:
ACTION_TYPE_INVALID = 0
ACTION_TYPE_DELETE = 1
ACTION_TYPE_INSERT = 2
ACTION_TYPE_INSERT_STAR = 3
ACTION_TYPE_UPDATE = 4
ACTION_TYPE_UPDATE_STAR = 5
message
expressions.proto:488Used in:
optional key = 1
(Required) The key of the assignment.
optional value = 2
(Required) The value of the assignment.
message
commands.proto:541Used in:
string target_table_name = 1
(Required) The name of the target table.
optional source_table_plan = 2
(Required) The relation of the source table.
optional merge_condition = 3
(Required) The condition to match the source and target.
repeated match_actions = 4
(Optional) The actions to be taken when the condition is matched.
repeated not_matched_actions = 5
(Optional) The actions to be taken when the condition is not matched.
repeated not_matched_by_source_actions = 6
(Optional) The actions to be taken when the condition is not matched by source.
bool with_schema_evolution = 7
(Required) Whether to enable schema evolution.
message
ml.proto:52Command for ML
Used in:
oneof command
fit = 1
fetch = 2
delete = 3
write = 4
read = 5
evaluate = 6
clean_cache = 7
get_cache_info = 8
message
ml.proto:81Force to clean up all the ML cached objects
Used in:
(message has no fields)
message
ml.proto:76Command to delete the cached objects which could be a model or summary evaluated by a model
Used in:
repeated obj_refs = 1
message
ml.proto:114Command for evaluator.evaluate(dataset)
Used in:
optional evaluator = 1
(Required) Evaluator information (its type should be OPERATOR_TYPE_EVALUATOR)
optional params = 2
(Optional) parameters of the Evaluator
optional dataset = 3
(Required) the evaluating dataset
message
ml.proto:65Command for estimator.fit(dataset)
Used in:
optional estimator = 1
(Required) Estimator information (its type should be OPERATOR_TYPE_ESTIMATOR)
optional params = 2
(Optional) parameters of the Estimator
optional dataset = 3
(Required) the training dataset
message
ml.proto:84Get the information of all the ML cached objects
Used in:
(message has no fields)
message
ml.proto:106Command to load ML operator.
Used in:
optional operator = 1
(Required) ML operator information
string path = 2
(Required) Load the ML instance from the input path
message
ml.proto:87Command to write ML operator
Used in:
oneof type
It could be an estimator/evaluator or the cached model
operator = 1
Estimator or evaluator
obj_ref = 2
The cached model
optional params = 3
(Optional) The parameters of operator which could be estimator/evaluator or a cached model
string path = 4
(Required) Save the ML instance to the path
optional bool should_overwrite = 5
(Optional) Overwrites if the output path already exists.
map<string, string> options = 6
(Optional) The options of the writer
message
ml.proto:125The result of MlCommand
Used in:
oneof result_type
param = 1
The result of the attribute
string summary = 2
Evaluate a Dataset in a model and return the cached ID of summary
operator_info = 3
Operator information
message
ml.proto:136Represents an operator info
Used in:
oneof type
obj_ref = 1
The cached object which could be a model or summary evaluated by a model
string name = 2
Operator name
optional string uid = 3
(Optional) the 'uid' of a ML object Note it is different from the 'id' of a cached object.
optional params = 4
(Optional) parameters
optional string warning_message = 5
(Optional) warning message generated during the ML command execution
message
ml_common.proto:56MLOperator represents the ML operators like (Estimator, Transformer or Evaluator)
Used in: , , , ,
string name = 1
(Required) The qualified name of the ML operator.
string uid = 2
(Required) Unique id of the ML operator
MlOperator.OperatorType type = 3
(Required) Represents what the ML operator is
Used in:
OPERATOR_TYPE_UNSPECIFIED = 0
OPERATOR_TYPE_ESTIMATOR = 1
ML estimator
OPERATOR_TYPE_TRANSFORMER = 2
ML transformer (non-model)
OPERATOR_TYPE_EVALUATOR = 3
ML evaluator
OPERATOR_TYPE_MODEL = 4
ML model
message
ml_common.proto:50MlParams stores param settings for ML Estimator / Transformer / Evaluator
Used in: , , , ,
map<string, > params = 1
User-supplied params
message
relations.proto:134Relation to represent ML world
Used in:
oneof ml_type
transform = 1
fetch = 2
message
relations.proto:141Relation to represent transform(input) of the operator which could be a cached model or a new transformer
Used in:
oneof operator
obj_ref = 1
Object reference
transformer = 2
Could be an ML transformer like VectorAssembler
optional input = 3
the input dataframe
optional params = 4
the operator specific parameters
message
relations.proto:814Drop rows containing null values. It will invoke 'Dataset.na.drop' (same as 'DataFrameNaFunctions.drop') to compute the results.
Used in:
optional input = 1
(Required) The input relation.
repeated string cols = 2
(Optional) Optional list of column names to consider. When it is empty, all the columns in the input relation will be considered.
optional int32 min_non_nulls = 3
(Optional) The minimum number of non-null and non-NaN values required to keep. When not set, it is equivalent to the number of considered columns, which means a row will be kept only if all columns are non-null. 'how' options ('all', 'any') can be easily converted to this field: - 'all' -> set 'min_non_nulls' 1; - 'any' -> keep 'min_non_nulls' unset;
message
relations.proto:797Replaces null values. It will invoke 'Dataset.na.fill' (same as 'DataFrameNaFunctions.fill') to compute the results. Following 3 parameter combinations are supported: 1, 'values' only contains 1 item, 'cols' is empty: replaces null values in all type-compatible columns. 2, 'values' only contains 1 item, 'cols' is not empty: replaces null values in specified columns. 3, 'values' contains more than 1 items, then 'cols' is required to have the same length: replaces each specified column with corresponding value.
Used in:
optional input = 1
(Required) The input relation.
repeated string cols = 2
(Optional) Optional list of column names to consider.
repeated values = 3
(Required) Values to replace null values with. Should contain at least 1 item. Only 4 data types are supported now: bool, long, double, string
message
relations.proto:838Replaces old values with the corresponding values. It will invoke 'Dataset.na.replace' (same as 'DataFrameNaFunctions.replace') to compute the results.
Used in:
optional input = 1
(Required) The input relation.
repeated string cols = 2
(Optional) List of column names to consider. When it is empty, all the type-compatible columns in the input relation will be considered.
repeated replacements = 3
(Optional) The value replacement mapping.
message
relations.proto:850Used in:
optional old_value = 1
(Required) The old value. Only 4 data types are supported now: null, bool, double, string.
optional new_value = 2
(Required) The new value. Should be of the same data type with the old value.
message
expressions.proto:461Used in:
string key = 1
(Required) The key of the named argument.
optional value = 2
(Required) The value expression of the named argument.
message
ml_common.proto:81Represents a reference to the cached object which could be a model or summary evaluated by a model
Used in: , , , ,
string id = 1
(Required) The ID is used to lookup the object on the server side. Note it is different from the 'uid' of a ML object.
message
relations.proto:396Relation of type [[Offset]] that is used to read rows staring from the `offset` on the input relation.
Used in:
optional input = 1
(Required) Input relation for an Offset.
int32 offset = 2
(Required) the limit.
message
common.proto:106Used in: ,
oneof function
(Required) Indicate the origin type.
python_origin = 1
jvm_origin = 2
message
relations.proto:1180Used in:
optional input = 1
(Required) Input relation to Parse. The input is expected to have single text column.
Parse.ParseFormat format = 2
(Required) The expected format of the text.
optional schema = 3
(Optional) DataType representing the schema. If not set, Spark will infer the schema.
map<string, string> options = 4
Options for the csv/json parser. The map key is case insensitive.
Used in:
PARSE_FORMAT_UNSPECIFIED = 0
PARSE_FORMAT_CSV = 1
PARSE_FORMAT_JSON = 2
message
pipelines.proto:50Dispatch object for pipelines commands. See each individual command for documentation.
Used in:
oneof command_type
create_dataflow_graph = 1
define_dataset = 2
define_flow = 3
drop_dataflow_graph = 4
start_run = 5
define_sql_graph_elements = 6
message
pipelines.proto:61Request to create a new dataflow graph.
Used in:
optional string default_catalog = 1
The default catalog.
optional string default_database = 2
The default database.
map<string, string> sql_conf = 5
SQL configurations for all flows in this graph.
message
pipelines.proto:71optional string dataflow_graph_id = 1
The ID of the created graph.
message
pipelines.proto:84Request to define a dataset: a table, a materialized view, or a temporary view.
Used in:
optional string dataflow_graph_id = 1
The graph to attach this dataset to.
optional string dataset_name = 2
Name of the dataset. Can be partially or fully qualified.
optional DatasetType dataset_type = 3
The type of the dataset.
optional string comment = 4
Optional comment for the dataset.
map<string, string> table_properties = 5
Optional table properties. Only applies to dataset_type == TABLE and dataset_type == MATERIALIZED_VIEW.
repeated string partition_cols = 6
Optional partition columns for the dataset. Only applies to dataset_type == TABLE and dataset_type == MATERIALIZED_VIEW.
optional schema = 7
Schema for the dataset. If unset, this will be inferred from incoming flows.
optional string format = 8
The output table format of the dataset. Only applies to dataset_type == TABLE and dataset_type == MATERIALIZED_VIEW.
message
pipelines.proto:113Request to define a flow targeting a dataset.
Used in:
optional string dataflow_graph_id = 1
The graph to attach this flow to.
optional string flow_name = 2
Name of the flow. For standalone flows, this must be a single-part name.
optional string target_dataset_name = 3
Name of the dataset this flow writes to. Can be partially or fully qualified.
optional plan = 4
An unresolved relation that defines the dataset's flow.
map<string, string> sql_conf = 5
SQL configurations set when running this flow.
optional bool once = 6
If true, this flow will only be run once per full refresh.
message
pipelines.proto:78Drops the graph and stops any running attached flows.
Used in:
optional string dataflow_graph_id = 1
The graph to drop.
message
pipelines.proto:135Resolves all datasets and flows and start a pipeline update. Should be called after all graph elements are registered.
Used in:
optional string dataflow_graph_id = 1
The graph to start.
message
pipelines.proto:154Dispatch object for pipelines command results.
Used in:
oneof result_type
create_dataflow_graph_result = 1
message
pipelines.proto:158Used in:
optional string dataflow_graph_id = 1
The ID of the created graph.
message
pipelines.proto:182An event emitted during the run of a graph.
Used in:
optional string timestamp = 1
The time of the event.
optional string message = 2
The message that should be displayed to users.
message
pipelines.proto:177A response containing an event emitted during the run of a pipeline.
Used in:
optional event = 1
message
base.proto:60A [[Plan]] is the structure that carries the runtime information for the execution from the client to the server. A [[Plan]] can either be of the type [[Relation]] which is a reference to the underlying logical plan or it can be of the [[Command]] type that is used to execute commands on the server.
Used in: , , , , , , , ,
oneof op_type
root = 1
command = 2
message
relations.proto:283Projection of a bag of expressions for a given input relation. The input relation must be specified. The projected expression can be an arbitrary expression.
Used in:
optional input = 1
(Optional) Input relation is optional for Project. For example, `SELECT ABS(-1)` is valid plan without an input plan.
repeated expressions = 3
(Required) A Project requires at least one expression.
message
relations.proto:1160Used in:
bytes command = 1
(Required) The encoded commands of the Python data source.
string python_ver = 2
(Required) Python version being used in the client.
message
common.proto:114Used in:
string fragment = 1
(Required) Name of the origin, for example, the name of the function
string call_site = 2
(Required) Callsite to show to end users, for example, stacktrace.
message
expressions.proto:411Used in: , ,
optional output_type = 1
(Required) Output type of the Python UDF
int32 eval_type = 2
(Required) EvalType of the Python UDF
bytes command = 3
(Required) The encoded commands of the Python UDF
string python_ver = 4
(Required) Python version being used in the client.
repeated string additional_includes = 5
(Optional) Additional includes for the Python UDF.
message
relations.proto:1136Used in:
optional return_type = 1
(Optional) Return type of the Python UDTF.
int32 eval_type = 2
(Required) EvalType of the Python UDTF.
bytes command = 3
(Required) The encoded commands of the Python UDTF.
string python_ver = 4
(Required) Python version being used in the client.
message
relations.proto:563Relation of type [[Range]] that generates a sequence of integers.
Used in:
optional int64 start = 1
(Optional) Default value = 0
int64 end = 2
(Required)
int64 step = 3
(Required)
optional int32 num_partitions = 4
Optional. Default value is assigned by 1) SQL conf "spark.sql.leafNodeDefaultParallelism" if it is set, or 2) spark default parallelism.
message
relations.proto:236Relation that reads from a file / table or other data source. Does not have additional inputs.
Used in:
oneof read_type
named_table = 1
data_source = 2
bool is_streaming = 3
(Optional) Indicates if this is a streaming read.
message
relations.proto:253Used in:
optional string format = 1
(Optional) Supported formats include: parquet, orc, text, json, parquet, csv, avro. If not set, the value from SQL conf 'spark.sql.sources.default' will be used.
optional string schema = 2
(Optional) If not set, Spark will infer the schema. This schema string should be either DDL-formatted or JSON-formatted.
map<string, string> options = 3
Options for the data source. The context of this map varies based on the data source format. This options could be empty for valid data source format. The map key is case insensitive.
repeated string paths = 4
(Optional) A list of path for file-system backed data sources.
repeated string predicates = 5
(Optional) Condition in the where clause for each partition. This is only supported by the JDBC data source.
message
relations.proto:245Used in:
string unparsed_identifier = 1
(Required) Unparsed identifier for the table.
map<string, string> options = 2
Options for the named table. The map key is case insensitive.
message
base.proto:831Used in:
bool reattachable = 1
If true, the request can be reattached to using ReattachExecute. ReattachExecute can be used either if the stream broke with a GRPC network error, or if the server closed the stream without sending a response with StreamStatus.complete=true. The server will keep a buffer of responses in case a response is lost, and ReattachExecute needs to back-track. If false, the execution response stream will will not be reattachable, and all responses are immediately released by the server after being sent.
message
catalog.proto:210See `spark.catalog.recoverPartitions`
Used in:
string table_name = 1
(Required)
message
catalog.proto:246See `spark.catalog.refreshByPath`
Used in:
string path = 1
(Required)
message
catalog.proto:240See `spark.catalog.refreshTable`
Used in:
string table_name = 1
(Required)
message
relations.proto:58The main [[Relation]] type. Fundamentally, a relation is a typed container that has exactly one explicit relation type set. When adding new relation types, they have to be registered here.
Used in: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ,
optional common = 1
oneof rel_type
read = 2
project = 3
filter = 4
join = 5
set_op = 6
sort = 7
limit = 8
aggregate = 9
sql = 10
local_relation = 11
sample = 12
offset = 13
deduplicate = 14
range = 15
subquery_alias = 16
repartition = 17
to_df = 18
with_columns_renamed = 19
show_string = 20
drop = 21
tail = 22
with_columns = 23
hint = 24
unpivot = 25
to_schema = 26
repartition_by_expression = 27
map_partitions = 28
collect_metrics = 29
parse = 30
group_map = 31
co_group_map = 32
with_watermark = 33
apply_in_pandas_with_state = 34
html_string = 35
cached_local_relation = 36
cached_remote_relation = 37
common_inline_user_defined_table_function = 38
as_of_join = 39
common_inline_user_defined_data_source = 40
with_relations = 41
transpose = 42
unresolved_table_valued_function = 43
lateral_join = 44
fill_na = 90
NA functions
drop_na = 91
replace = 92
summary = 100
stat functions
crosstab = 101
describe = 102
cov = 103
corr = 104
approx_quantile = 105
freq_items = 106
sample_by = 107
catalog = 200
Catalog API (experimental / unstable)
ml_relation = 300
ML relation
extension = 998
This field is used to mark extensions to the protocol. When plugins generate arbitrary relations they can add them here. During the planning the correct resolution is done.
unknown = 999
message
relations.proto:186Common metadata of all relations.
Used in:
string source_info = 1
(Required) Shared relation metadata.
optional int64 plan_id = 2
(Optional) A per-client globally unique id for a given connect plan.
optional origin = 3
(Optional) Keep the information of the origin for this expression such as stacktrace.
message
base.proto:914Release and close operation completely. This will also interrupt the query if it is running execution, and wait for it to be torn down.
Used in:
(message has no fields)
message
base.proto:921Release all responses from the operation response stream up to and including the response with the given by response_id. While server determines by itself how much of a buffer of responses to keep, client providing explicit release calls will help reduce resource consumption. Noop if response_id not found in cached responses.
Used in:
string response_id = 1
message
commands.proto:521Command to remove `CashedRemoteRelation`
Used in:
optional relation = 1
(Required) The remote to be related
message
relations.proto:591Relation repartition.
Used in:
optional input = 1
(Required) The input relation of Repartition.
int32 num_partitions = 2
(Required) Must be positive.
optional bool shuffle = 3
(Optional) Default value is false.
message
relations.proto:999Used in:
optional input = 1
(Required) The input relation.
repeated partition_exprs = 2
(Required) The partitioning expressions.
optional int32 num_partitions = 3
(Optional) number of partitions, must be positive.
message
common.proto:64ResourceInformation to hold information about a type of Resource. The corresponding class is 'org.apache.spark.resource.ResourceInformation'
Used in:
string name = 1
(Required) The name of the resource
repeated string addresses = 2
(Required) An array of strings describing the addresses of the resource.
message
common.proto:96Used in:
map<string, > executor_resources = 1
(Optional) Resource requests for executors. Mapped from the resource name (e.g., cores, memory, CPU) to its specific request.
map<string, > task_resources = 2
(Optional) Resource requests for tasks. Mapped from the resource name (e.g., cores, memory, CPU) to its specific request.
message
relations.proto:198Relation that uses a SQL query to generate the output.
Used in:
string query = 1
(Required) The SQL query.
map<string, > args = 2
(Optional) A map of parameter names to literal expressions.
repeated pos_args = 3
(Optional) A sequence of literal expressions for positional parameters in the SQL query text.
map<string, > named_arguments = 4
(Optional) A map of parameter names to expressions. It cannot coexist with `pos_arguments`.
repeated pos_arguments = 5
(Optional) A sequence of expressions for positional parameters in the SQL query text. It cannot coexist with `named_arguments`.
message
relations.proto:538Relation of type [[Sample]] that samples a fraction of the dataset.
Used in:
optional input = 1
(Required) Input relation for a Sample.
double lower_bound = 2
(Required) lower bound.
double upper_bound = 3
(Required) upper bound.
optional bool with_replacement = 4
(Optional) Whether to sample with replacement.
optional int64 seed = 5
(Required) The random seed. This field is required to avoid generating mutable dataframes (see SPARK-48184 for details), however, still keep it 'optional' here for backward compatibility.
bool deterministic_order = 6
(Required) Explicitly sort the underlying plan to make the ordering deterministic or cache it. This flag is true when invoking `dataframe.randomSplit` to randomly splits DataFrame with the provided weights. Otherwise, it is false.
message
expressions.proto:424Used in: , ,
bytes payload = 1
(Required) Serialized JVM object containing UDF definition, input encoders and output encoder
repeated inputTypes = 2
(Optional) Input type(s) of the UDF
optional outputType = 3
(Required) Output type of the UDF
bool nullable = 4
(Required) True if the UDF can return null value
bool aggregate = 5
(Required) Indicate if the UDF is an aggregate function
message
catalog.proto:255See `spark.catalog.setCurrentCatalog`
Used in:
string catalog_name = 1
(Required)
message
catalog.proto:86See `spark.catalog.setCurrentDatabase`
Used in:
string db_name = 1
(Required)
message
relations.proto:351Relation of type [[SetOperation]]
Used in:
optional left_input = 1
(Required) Left input relation for a Set operation.
optional right_input = 2
(Required) Right input relation for a Set operation.
SetOperation.SetOpType set_op_type = 3
(Required) The Set operation type.
optional bool is_all = 4
(Optional) If to remove duplicate rows. True to preserve all results. False to remove duplicate rows.
optional bool by_name = 5
(Optional) If to perform the Set operation based on name resolution. Only UNION supports this option.
optional bool allow_missing_columns = 6
(Optional) If to perform the Set operation and allow missing columns. Only UNION supports this option.
Used in:
SET_OP_TYPE_UNSPECIFIED = 0
SET_OP_TYPE_INTERSECT = 1
SET_OP_TYPE_UNION = 2
SET_OP_TYPE_EXCEPT = 3
message
relations.proto:604Compose the string representing rows for output. It will invoke 'Dataset.showString' to compute the results.
Used in:
optional input = 1
(Required) The input relation.
int32 num_rows = 2
(Required) Number of rows to show.
int32 truncate = 3
(Required) If set to more than 0, truncates strings to `truncate` characters and all cells will be aligned right.
bool vertical = 4
(Required) If set to true, prints output rows vertically (one line per column value).
message
relations.proto:460Relation of type [[Sort]].
Used in:
optional input = 1
(Required) Input relation for a Sort.
repeated order = 2
(Required) The ordering expressions
optional bool is_global = 3
(Optional) if this is a global sort.
message
commands.proto:91A SQL Command is used to trigger the eager evaluation of SQL commands in Spark. When the SQL provide as part of the message is a command it will be immediately evaluated and the result will be collected and returned as part of a LocalRelation. If the result is not a command, the operation will simply return a SQL Relation. This allows the client to be almost oblivious to the server-side behavior.
Used in:
string sql = 1
(Required) SQL Query.
map<string, > args = 2
(Optional) A map of parameter names to literal expressions.
repeated pos_args = 3
(Optional) A sequence of literal expressions for positional parameters in the SQL query text.
map<string, > named_arguments = 4
(Optional) A map of parameter names to expressions. It cannot coexist with `pos_arguments`.
repeated pos_arguments = 5
(Optional) A sequence of expressions for positional parameters in the SQL query text. It cannot coexist with `named_arguments`.
optional input = 6
(Optional) The relation that this SQL command will be built on.
message
common.proto:149A message to hold a [[java.lang.StackTraceElement]].
Used in:
optional string class_loader_name = 1
(Optional) Class loader name
optional string module_name = 2
(Optional) Module name
optional string module_version = 3
(Optional) Module version
string declaring_class = 4
(Required) Declaring class
string method_name = 5
(Required) Method name
optional string file_name = 6
(Optional) File name
int32 line_number = 7
(Required) Line number
message
relations.proto:721Calculates the approximate quantiles of numerical columns of a DataFrame. It will invoke 'Dataset.stat.approxQuantile' (same as 'StatFunctions.approxQuantile') to compute the results.
Used in:
optional input = 1
(Required) The input relation.
repeated string cols = 2
(Required) The names of the numerical columns.
repeated double probabilities = 3
(Required) A list of quantile probabilities. Each number must belong to [0, 1]. For example 0 is the minimum, 0.5 is the median, 1 is the maximum.
double relative_error = 4
(Required) The relative target precision to achieve (greater than or equal to 0). If set to zero, the exact quantiles are computed, which could be very expensive. Note that values greater than 1 are accepted but give the same result as 1.
message
relations.proto:702Calculates the correlation of two columns of a DataFrame. Currently only supports the Pearson Correlation Coefficient. It will invoke 'Dataset.stat.corr' (same as 'StatFunctions.pearsonCorrelation') to compute the results.
Used in:
optional input = 1
(Required) The input relation.
string col1 = 2
(Required) The name of the first column.
string col2 = 3
(Required) The name of the second column.
optional string method = 4
(Optional) Default value is 'pearson'. Currently only supports the Pearson Correlation Coefficient.
message
relations.proto:688Calculate the sample covariance of two numerical columns of a DataFrame. It will invoke 'Dataset.stat.cov' (same as 'StatFunctions.calculateCov') to compute the results.
Used in:
optional input = 1
(Required) The input relation.
string col1 = 2
(Required) The name of the first column.
string col2 = 3
(Required) The name of the second column.
message
relations.proto:671Computes a pair-wise frequency table of the given columns. Also known as a contingency table. It will invoke 'Dataset.stat.crosstab' (same as 'StatFunctions.crossTabulate') to compute the results.
Used in:
optional input = 1
(Required) The input relation.
string col1 = 2
(Required) The name of the first column. Distinct items will make the first item of each row.
string col2 = 3
(Required) The name of the second column. Distinct items will make the column names of the DataFrame.
message
relations.proto:660Computes basic statistics for numeric and string columns, including count, mean, stddev, min, and max. If no columns are given, this function computes statistics for all numerical or string columns.
Used in:
optional input = 1
(Required) The input relation.
repeated string cols = 2
(Optional) Columns to compute statistics on.
message
relations.proto:744Finding frequent items for columns, possibly with false positives. It will invoke 'Dataset.stat.freqItems' (same as 'StatFunctions.freqItems') to compute the results.
Used in:
optional input = 1
(Required) The input relation.
repeated string cols = 2
(Required) The names of the columns to search frequent items in.
optional double support = 3
(Optional) The minimum frequency for an item to be considered `frequent`. Should be greater than 1e-4.
message
relations.proto:761Returns a stratified sample without replacement based on the fraction given on each stratum. It will invoke 'Dataset.stat.freqItems' (same as 'StatFunctions.freqItems') to compute the results.
Used in:
optional input = 1
(Required) The input relation.
optional col = 2
(Required) The column that defines strata.
repeated fractions = 3
(Required) Sampling fraction for each stratum. If a stratum is not specified, we treat its fraction as zero.
optional int64 seed = 5
(Required) The random seed. This field is required to avoid generating mutable dataframes (see SPARK-48184 for details), however, still keep it 'optional' here for backward compatibility.
message
relations.proto:778Used in:
optional stratum = 1
(Required) The stratum.
double fraction = 2
(Required) The fraction value. Must be in [0, 1].
message
relations.proto:636Computes specified statistics for numeric and string columns. It will invoke 'Dataset.summary' (same as 'StatFunctions.summary') to compute the results.
Used in:
optional input = 1
(Required) The input relation.
repeated string statistics = 2
(Optional) Statistics from to be computed. Available statistics are: count mean stddev min max arbitrary approximate percentiles specified as a percentage (e.g. 75%) count_distinct approx_count_distinct If no statistics are given, this function computes 'count', 'mean', 'stddev', 'min', 'approximate quartiles' (percentiles at 25%, 50%, and 75%), and 'max'.
message
common.proto:48StorageLevel for persisting Datasets/Tables.
Used in: , , ,
bool use_disk = 1
(Required) Whether the cache should use disk or not.
bool use_memory = 2
(Required) Whether the cache should use memory or not.
bool use_off_heap = 3
(Required) Whether the cache should use off-heap or not.
bool deserialized = 4
(Required) Whether the cached data is deserialized or not.
int32 replication = 5
(Required) The number of replicas.
message
commands.proto:274Used in:
oneof function
python_function = 1
scala_function = 2
message
commands.proto:313Commands for a streaming query.
Used in:
optional query_id = 1
(Required) Query instance. See `StreamingQueryInstanceId`.
oneof command
See documentation for the corresponding API method in StreamingQuery.
bool status = 2
status() API.
bool last_progress = 3
lastProgress() API.
bool recent_progress = 4
recentProgress() API.
bool stop = 5
stop() API. Stops the query.
bool process_all_available = 6
processAllAvailable() API. Waits till all the available data is processed
explain = 7
explain() API. Returns logical and physical plans.
bool exception = 8
exception() API. Returns the exception in the query if any.
await_termination = 9
awaitTermination() API. Waits for the termination of the query.
message
commands.proto:344Used in:
optional int64 timeout_ms = 2
message
commands.proto:338Used in:
bool extended = 1
TODO: Consider reusing Explain from AnalyzePlanRequest message. We can not do this right now since it base.proto imports this file.
message
commands.proto:350Response for commands on a streaming query.
Used in:
optional query_id = 1
(Required) Query instance id. See `StreamingQueryInstanceId`.
oneof result_type
status = 2
recent_progress = 3
explain = 4
exception = 5
await_termination = 6
message
commands.proto:390Used in:
bool terminated = 1
message
commands.proto:380Used in:
optional string exception_message = 1
(Optional) Exception message as string, maps to the return value of original StreamingQueryException's toString method
optional string error_class = 2
(Optional) Exception error class as string
optional string stack_trace = 3
(Optional) Exception stack trace as string
message
commands.proto:375Used in:
string result = 1
Logical and physical plans as string
message
commands.proto:370Used in:
repeated string recent_progress_json = 5
Progress reports as an array of json strings.
message
commands.proto:362Used in:
string status_message = 1
See documentation for these Scala 'StreamingQueryStatus' struct
bool is_data_available = 2
bool is_trigger_active = 3
bool is_active = 4
The enum used for client side streaming query listener event There is no QueryStartedEvent defined here, it is added as a field in WriteStreamOperationStartResult
Used in:
QUERY_PROGRESS_UNSPECIFIED = 0
QUERY_PROGRESS_EVENT = 1
QUERY_TERMINATED_EVENT = 2
QUERY_IDLE_EVENT = 3
message
commands.proto:299A tuple that uniquely identifies an instance of streaming query run. It consists of `id` that persists across the streaming runs and `run_id` that changes between each run of the streaming query that resumes from the checkpoint.
Used in: , , ,
string id = 1
(Required) The unique id of this query that persists across restarts from checkpoint data. That is, this id is generated when a query is started for the first time, and will be the same every time it is restarted from checkpoint data.
string run_id = 2
(Required) The unique id of this run of the query. That is, every start/restart of a query will generate a unique run_id. Therefore, every time a query is restarted from checkpoint, it will have the same `id` but different `run_id`s.
message
commands.proto:470The protocol for client-side StreamingQueryListener. This command will only be set when either the first listener is added to the client, or the last listener is removed from the client. The add_listener_bus_listener command will only be set true in the first case. The remove_listener_bus_listener command will only be set true in the second case.
Used in:
oneof command
bool add_listener_bus_listener = 1
bool remove_listener_bus_listener = 2
message
commands.proto:488The protocol for the returned events in the long-running response channel.
Used in:
string event_json = 1
(Required) The json serialized event, all StreamingQueryListener events have a json method
StreamingQueryEventType event_type = 2
(Required) Query event type used by client to decide how to deserialize the event_json
message
commands.proto:495Used in:
repeated events = 1
optional bool listener_bus_listener_added = 2
message
commands.proto:396Commands for the streaming query manager.
Used in:
oneof command
See documentation for the corresponding API method in StreamingQueryManager.
bool active = 1
active() API, returns a list of active queries.
string get_query = 2
get() API, returns the StreamingQuery identified by id.
await_any_termination = 3
awaitAnyTermination() API, wait until any query terminates or timeout.
bool reset_terminated = 4
resetTerminated() API.
add_listener = 5
addListener API.
remove_listener = 6
removeListener API.
bool list_listeners = 7
listListeners() API, returns a list of streaming query listeners.
message
commands.proto:416Used in:
optional int64 timeout_ms = 1
(Optional) The waiting time in milliseconds to wait for any query to terminate.
message
commands.proto:421Used in:
bytes listener_payload = 1
optional python_listener_payload = 2
string id = 3
message
commands.proto:429Response for commands on the streaming query manager.
Used in:
oneof result_type
active = 1
query = 2
await_any_termination = 3
bool reset_terminated = 4
bool add_listener = 5
bool remove_listener = 6
list_listeners = 7
message
commands.proto:440Used in:
repeated active_queries = 1
message
commands.proto:451Used in:
bool terminated = 1
message
commands.proto:459Used in:
repeated string listener_ids = 1
(Required) Reference IDs of listener instances.
message
commands.proto:444Used in: ,
optional id = 1
(Required) The id and runId of this query.
optional string name = 2
(Optional) The name of this query.
message
commands.proto:455bytes listener_payload = 1
message
common.proto:192Used in:
repeated string values = 1
message
relations.proto:579Relation alias.
Used in:
optional input = 1
(Required) The input relation of SubqueryAlias.
string alias = 2
(Required) The alias.
repeated string qualifier = 3
(Optional) Qualifier of the alias.
message
expressions.proto:497Used in:
int64 plan_id = 1
(Required) The ID of the corresponding connect plan.
SubqueryExpression.SubqueryType subquery_type = 2
(Required) The type of the subquery.
optional table_arg_options = 3
(Optional) Options specific to table arguments.
repeated in_subquery_values = 4
(Optional) IN subquery values.
Used in:
SUBQUERY_TYPE_UNKNOWN = 0
SUBQUERY_TYPE_SCALAR = 1
SUBQUERY_TYPE_EXISTS = 2
SUBQUERY_TYPE_TABLE_ARG = 3
SUBQUERY_TYPE_IN = 4
message
expressions.proto:519Nested message for table argument options.
Used in:
repeated partition_spec = 1
(Optional) The way that input rows are partitioned.
repeated order_spec = 2
(Optional) Ordering of rows in a partition.
optional bool with_single_partition = 3
(Optional) Whether this is a single partition.
message
catalog.proto:150See `spark.catalog.tableExists`
Used in:
string table_name = 1
(Required)
optional string db_name = 2
(Optional)
message
relations.proto:405Relation of type [[Tail]] that is used to fetch `limit` rows from the last of the input relation.
Used in:
optional input = 1
(Required) Input relation for an Tail.
int32 limit = 2
(Required) the limit.
message
common.proto:87A task resource request.
Used in:
string resource_name = 1
(Required) resource name.
double amount = 2
(Required) resource amount requesting as a double to support fractional resource requests.
message
relations.proto:865Rename columns on the input relation by the same length of names.
Used in:
optional input = 1
(Required) The input relation of RenameColumnsBySameLengthNames.
repeated string column_names = 2
(Required) The number of columns of the input relation must be equal to the length of this field. If this is not true, an exception will be returned.
message
relations.proto:989Used in:
optional input = 1
(Required) The input relation.
optional schema = 2
(Required) The user provided schema. The Sever side will update the dataframe with this schema.
message
relations.proto:1062Additional input parameters used for TransformWithState operator.
Used in:
string time_mode = 1
(Required) Time mode string for transformWithState.
optional string event_time_column_name = 2
(Optional) Event time column name.
optional output_schema = 3
(Optional) Schema for the output DataFrame. Only required used for TransformWithStateInPandas.
message
relations.proto:972Transpose a DataFrame, switching rows to columns. Transforms the DataFrame such that the values in the specified index column become the new columns of the DataFrame.
Used in:
optional input = 1
(Required) The input relation.
repeated index_columns = 2
(Optional) A list of columns that will be treated as the indices. Only single column is supported now.
message
expressions.proto:448Used in:
optional scalar_scala_udf = 1
(Required) The aggregate function object packed into bytes.
message
catalog.proto:231See `spark.catalog.uncacheTable`
Used in:
string table_name = 1
(Required)
message
relations.proto:183Used for testing purposes only.
Used in:
(message has no fields)
message
relations.proto:948Unpivot a DataFrame from wide format to long format, optionally leaving identifier columns set.
Used in:
optional input = 1
(Required) The input relation.
repeated ids = 2
(Required) Id columns.
optional values = 3
(Optional) Value columns to unpivot.
string variable_column_name = 4
(Required) Name of the variable column.
string value_column_name = 5
(Required) Name of the value column.
message
relations.proto:964Used in:
repeated values = 1
message
relations.proto:981Used in:
string function_name = 1
(Required) name (or unparsed name for user defined function) for the unresolved function.
repeated arguments = 2
(Optional) Function arguments. Empty arguments are allowed.
message
base.proto:71User Context is used to refer to one particular user session that is executing queries in the backend.
Used in: , , , , , , , , ,
string user_id = 1
string user_name = 2
repeated extensions = 999
To extend the existing user context message that is used to identify incoming requests, Spark Connect leverages the Any protobuf type that can be used to inject arbitrary other messages into this message. Extensions are stored as a `repeated` type to be able to handle multiple active extensions.
message
relations.proto:903Adding columns or replacing the existing columns that have the same names.
Used in:
optional input = 1
(Required) The input relation.
repeated aliases = 2
(Required) Given a column name, apply the corresponding expression on the column. If column name exists in the input relation, then replace the column. If the column name does not exist in the input relation, then adds it as a new column. Only one name part is expected from each Expression.Alias. An exception is thrown when duplicated names are present in the mapping.
message
relations.proto:878Rename columns on the input relation by a map with name to name mapping.
Used in:
optional input = 1
(Required) The input relation.
map<string, string> rename_columns_map = 2
(Optional) Renaming column names of input relation from A to B where A is the map key and B is the map value. This is a no-op if schema doesn't contain any A. It does not require that all input relation column names to present as keys. duplicated B are not allowed.
repeated renames = 3
message
relations.proto:893Used in:
string col_name = 1
(Required) The existing column name.
string new_col_name = 2
(Required) The new column name.
message
relations.proto:224Relation of type [[WithRelations]]. This relation contains a root plan, and one or more references that are used by the root plan. There are two ways of referencing a relation, by name (through a subquery alias), or by plan_id (using RelationCommon.plan_id). This relation can be used to implement CTEs, describe DAGs, or to reduce tree depth.
Used in:
optional root = 1
(Required) Plan at the root of the query tree. This plan is expected to contain one or more references. Those references get expanded later on by the engine.
repeated references = 2
(Required) Plans referenced by the root plan. Relations in this list are also allowed to contain references to other relations in this list, as long they do not form cycles.
message
relations.proto:919Used in:
optional input = 1
(Required) The input relation
string event_time = 2
(Required) Name of the column containing event time.
string delay_threshold = 3
(Required)
message
commands.proto:132As writes are not directly handled during analysis and planning, they are modeled as commands.
Used in:
optional input = 1
(Required) The output of the `input` relation will be persisted according to the options.
optional string source = 2
(Optional) Format value according to the Spark documentation. Examples are: text, parquet, delta.
oneof save_type
(Optional) The destination of the write operation can be either a path or a table. If the destination is neither a path nor a table, such as jdbc and noop, the `save_type` should not be set.
string path = 3
table = 4
WriteOperation.SaveMode mode = 5
(Required) the save mode.
repeated string sort_column_names = 6
(Optional) List of columns to sort the output by.
repeated string partitioning_columns = 7
(Optional) List of columns for partitioning.
optional bucket_by = 8
(Optional) Bucketing specification. Bucketing must set the number of buckets and the columns to bucket by.
map<string, string> options = 9
(Optional) A list of configuration options.
repeated string clustering_columns = 10
(Optional) Columns used for clustering the table.
message
commands.proto:181Used in:
repeated string bucket_column_names = 1
int32 num_buckets = 2
Used in:
SAVE_MODE_UNSPECIFIED = 0
SAVE_MODE_APPEND = 1
SAVE_MODE_OVERWRITE = 2
SAVE_MODE_ERROR_IF_EXISTS = 3
SAVE_MODE_IGNORE = 4
message
commands.proto:168Used in:
string table_name = 1
(Required) The table name.
SaveTable.TableSaveMethod save_method = 2
(Required) The method to be called to write to the table.
Used in:
TABLE_SAVE_METHOD_UNSPECIFIED = 0
TABLE_SAVE_METHOD_SAVE_AS_TABLE = 1
TABLE_SAVE_METHOD_INSERT_INTO = 2
message
commands.proto:196As writes are not directly handled during analysis and planning, they are modeled as commands.
Used in:
optional input = 1
(Required) The output of the `input` relation will be persisted according to the options.
string table_name = 2
(Required) The destination of the write operation must be either a path or a table.
optional string provider = 3
(Optional) A provider for the underlying output data source. Spark's default catalog supports "parquet", "json", etc.
repeated partitioning_columns = 4
(Optional) List of columns for partitioning for output table created by `create`, `createOrReplace`, or `replace`
map<string, string> options = 5
(Optional) A list of configuration options.
map<string, string> table_properties = 6
(Optional) A list of table properties.
WriteOperationV2.Mode mode = 7
(Required) Write mode.
optional overwrite_condition = 8
(Optional) A condition for overwrite saving mode
repeated string clustering_columns = 9
(Optional) Columns used for clustering the table.
Used in:
MODE_UNSPECIFIED = 0
MODE_CREATE = 1
MODE_OVERWRITE = 2
MODE_OVERWRITE_PARTITIONS = 3
MODE_APPEND = 4
MODE_REPLACE = 5
MODE_CREATE_OR_REPLACE = 6
message
commands.proto:239Starts write stream operation as streaming query. Query ID and Run ID of the streaming query are returned.
Used in:
optional input = 1
(Required) The output of the `input` streaming relation will be written.
string format = 2
map<string, string> options = 3
repeated string partitioning_column_names = 4
oneof trigger
string processing_time_interval = 5
bool available_now = 6
bool once = 7
string continuous_checkpoint_interval = 8
string output_mode = 9
string query_name = 10
oneof sink_destination
The destination is optional. When set, it can be a path or a table name.
string path = 11
string table_name = 12
optional foreach_writer = 13
optional foreach_batch = 14
repeated string clustering_column_names = 15
(Optional) Columns used for clustering the table.
message
commands.proto:281Used in:
optional query_id = 1
(Required) Query instance. See `StreamingQueryInstanceId`.
string name = 2
An optional query name.
optional string query_started_event_json = 3
Optional query started event if there is any listener registered on the client side.