package spark.connect
Get desktop application:
View/edit binary Protocol Buffers messages
Main interface for the SparkConnect service.
rpc AddArtifacts (stream , )
base.proto:797Add artifacts to the session and returns a [[AddArtifactsResponse]] containing metadata about the added artifacts.
message
base.proto:505Request to transfer client-local artifacts.
string session_id = 1
(Required) The session_id specifies a spark session for a user id (which is specified by user_context.user_id). The session_id is set by the client to be able to collate streaming responses from different queries within the dedicated session. The id should be an UUID string of the format `00112233-4455-6677-8899-aabbccddeeff`
optional user_context = 2
User context
optional string client_type = 6
Provides optional information about the client sending the request. This field can be used for language or version specific information and is only intended for logging purposes and will not be interpreted by the server.
oneof payload
The payload is either a batch of artifacts or a partial chunk of a large artifact.
batch = 3
begin_chunk = 4
The metadata and the initial chunk of a large artifact chunked into multiple requests. The server side is notified about the total size of the large artifact as well as the number of chunks to expect.
chunk = 5
A chunk of an artifact excluding metadata. This can be any chunk of a large artifact excluding the first chunk (which is included in `BeginChunkedArtifact`).
message
base.proto:581Response to adding an artifact. Contains relevant metadata to verify successful transfer of artifact(s).
repeated artifacts = 1
The list of artifact(s) seen by the server.
rpc AnalyzePlan (, )
base.proto:790Analyzes a query and returns a [[AnalyzeResponse]] containing metadata about the query.
message
base.proto:60Request to perform plan analyze, optionally to explain the plan.
string session_id = 1
(Required) The session_id specifies a spark session for a user id (which is specified by user_context.user_id). The session_id is set by the client to be able to collate streaming responses from different queries within the dedicated session. The id should be an UUID string of the format `00112233-4455-6677-8899-aabbccddeeff`
optional user_context = 2
(Required) User context
optional string client_type = 3
Provides optional information about the client sending the request. This field can be used for language or version specific information and is only intended for logging purposes and will not be interpreted by the server.
oneof analyze
schema = 4
explain = 5
tree_string = 6
is_local = 7
is_streaming = 8
input_files = 9
spark_version = 10
ddl_parse = 11
same_semantics = 12
semantic_hash = 13
persist = 14
unpersist = 15
get_storage_level = 16
message
base.proto:200Response to performing analysis of the query. Contains relevant metadata to be able to reason about the performance.
string session_id = 1
oneof result
schema = 2
explain = 3
tree_string = 4
is_local = 5
is_streaming = 6
input_files = 7
spark_version = 8
ddl_parse = 9
same_semantics = 10
semantic_hash = 11
persist = 12
unpersist = 13
get_storage_level = 14
rpc ArtifactStatus (, )
base.proto:800Check statuses of artifacts in the session and returns them in a [[ArtifactStatusesResponse]]
message
base.proto:596Request to get current statuses of artifacts at the server side.
string session_id = 1
(Required) The session_id specifies a spark session for a user id (which is specified by user_context.user_id). The session_id is set by the client to be able to collate streaming responses from different queries within the dedicated session. The id should be an UUID string of the format `00112233-4455-6677-8899-aabbccddeeff`
optional user_context = 2
User context
optional string client_type = 3
Provides optional information about the client sending the request. This field can be used for language or version specific information and is only intended for logging purposes and will not be interpreted by the server.
repeated string names = 4
The name of the artifact is expected in the form of a "Relative Path" that is made up of a sequence of directories and the final file element. Examples of "Relative Path"s: "jars/test.jar", "classes/xyz.class", "abc.xyz", "a/b/X.jar". The server is expected to maintain the hierarchy of files as defined by their name. (i.e The relative path of the file on the server's filesystem will be the same as the name of the provided artifact)
message
base.proto:623Response to checking artifact statuses.
map<string, > statuses = 1
A map of artifact names to their statuses.
rpc Config (, )
base.proto:793Update or fetch the configurations and returns a [[ConfigResponse]] containing the result.
message
base.proto:420Request to update or fetch the configurations.
string session_id = 1
(Required) The session_id specifies a spark session for a user id (which is specified by user_context.user_id). The session_id is set by the client to be able to collate streaming responses from different queries within the dedicated session. The id should be an UUID string of the format `00112233-4455-6677-8899-aabbccddeeff`
optional user_context = 2
(Required) User context
optional operation = 3
(Required) The operation for the config.
optional string client_type = 4
Provides optional information about the client sending the request. This field can be used for language or version specific information and is only intended for logging purposes and will not be interpreted by the server.
message
base.proto:489Response to the config request.
string session_id = 1
repeated pairs = 2
(Optional) The result key-value pairs. Available when the operation is 'Get', 'GetWithDefault', 'GetOption', 'GetAll'. Also available for the operation 'IsModifiable' with boolean string "true" and "false".
repeated string warnings = 3
(Optional) Warning messages for deprecated or unsupported configurations.
rpc ExecutePlan (, stream )
base.proto:787Executes a request that contains the query and returns a stream of [[Response]]. It is guaranteed that there is at least one ARROW batch returned even if the result set is empty.
message
base.proto:271A request to be executed by the service.
string session_id = 1
(Required) The session_id specifies a spark session for a user id (which is specified by user_context.user_id). The session_id is set by the client to be able to collate streaming responses from different queries within the dedicated session. The id should be an UUID string of the format `00112233-4455-6677-8899-aabbccddeeff`
optional user_context = 2
(Required) User context user_context.user_id and session+id both identify a unique remote spark session on the server side.
optional string operation_id = 6
(Optional) Provide an id for this request. If not provided, it will be generated by the server. It is returned in every ExecutePlanResponse.operation_id of the ExecutePlan response stream. The id must be an UUID string of the format `00112233-4455-6677-8899-aabbccddeeff`
optional plan = 3
(Required) The logical plan to be executed / analyzed.
optional string client_type = 4
Provides optional information about the client sending the request. This field can be used for language or version specific information and is only intended for logging purposes and will not be interpreted by the server.
repeated request_options = 5
Repeated element for options that can be passed to the request. This element is currently unused but allows to pass in an extension value used for arbitrary options.
repeated string tags = 7
Tags to tag the given execution with. Tags cannot contain ',' character and cannot be empty strings. Used by Interrupt with interrupt.tag.
rpc Interrupt (, )
base.proto:803Interrupts running executions
message
base.proto:633string session_id = 1
(Required) The session_id specifies a spark session for a user id (which is specified by user_context.user_id). The session_id is set by the client to be able to collate streaming responses from different queries within the dedicated session. The id should be an UUID string of the format `00112233-4455-6677-8899-aabbccddeeff`
optional user_context = 2
(Required) User context
optional string client_type = 3
Provides optional information about the client sending the request. This field can be used for language or version specific information and is only intended for logging purposes and will not be interpreted by the server.
InterruptRequest.InterruptType interrupt_type = 4
(Required) The type of interrupt to execute.
oneof interrupt
string operation_tag = 5
if interrupt_tag == INTERRUPT_TYPE_TAG, interrupt operation with this tag.
string operation_id = 6
if interrupt_tag == INTERRUPT_TYPE_OPERATION_ID, interrupt operation with this operation_id.
message
base.proto:675string session_id = 1
Session id in which the interrupt was running.
repeated string interrupted_ids = 2
Operation ids of the executions which were interrupted.
rpc ReattachExecute (, stream )
base.proto:809Reattach to an existing reattachable execution. The ExecutePlan must have been started with ReattachOptions.reattachable=true. If the ExecutePlanResponse stream ends without a ResultComplete message, there is more to continue. If there is a ResultComplete, the client should use ReleaseExecute with
message
base.proto:695string session_id = 1
(Required) The session_id of the request to reattach to. This must be an id of existing session.
optional user_context = 2
(Required) User context user_context.user_id and session+id both identify a unique remote spark session on the server side.
string operation_id = 3
(Required) Provide an id of the request to reattach to. This must be an id of existing operation.
optional string client_type = 4
Provides optional information about the client sending the request. This field can be used for language or version specific information and is only intended for logging purposes and will not be interpreted by the server.
optional string last_response_id = 5
(Optional) Last already processed response id from the response stream. After reattach, server will resume the response stream after that response. If not specified, server will restart the stream from the start. Note: server controls the amount of responses that it buffers and it may drop responses, that are far behind the latest returned response, so this can't be used to arbitrarily scroll back the cursor. If the response is no longer available, this will result in an error.
rpc ReleaseExecute (, )
base.proto:815Release an reattachable execution, or parts thereof. The ExecutePlan must have been started with ReattachOptions.reattachable=true. Non reattachable executions are released automatically and immediately after the ExecutePlan RPC and ReleaseExecute may not be used.
message
base.proto:729string session_id = 1
(Required) The session_id of the request to reattach to. This must be an id of existing session.
optional user_context = 2
(Required) User context user_context.user_id and session+id both identify a unique remote spark session on the server side.
string operation_id = 3
(Required) Provide an id of the request to reattach to. This must be an id of existing operation.
optional string client_type = 4
Provides optional information about the client sending the request. This field can be used for language or version specific information and is only intended for logging purposes and will not be interpreted by the server.
oneof release
release_all = 5
release_until = 6
message
base.proto:771string session_id = 1
Session id in which the release was running.
optional string operation_id = 2
Operation id of the operation on which the release executed. If the operation couldn't be found (because e.g. it was concurrently released), will be unset. Otherwise, it will be equal to the operation_id from request.
message
base.proto:524A chunk of an Artifact.
Used in: , ,
bytes data = 1
Data chunk.
int64 crc = 2
CRC to allow server to verify integrity of the chunk.
message
base.proto:546A number of `SingleChunkArtifact` batched into a single RPC.
Used in:
repeated artifacts = 1
message
base.proto:553Signals the beginning/start of a chunked artifact. A large artifact is transferred through a payload of `BeginChunkedArtifact` followed by a sequence of `ArtifactChunk`s.
Used in:
string name = 1
Name of the artifact undergoing chunking. Follows the same conventions as the `name` in the `Artifact` message.
int64 total_bytes = 2
Total size of the artifact in bytes.
int64 num_chunks = 3
Number of chunks the artifact is split into. This includes the `initial_chunk`.
optional initial_chunk = 4
The first/initial chunk.
message
base.proto:533An artifact that is contained in a single `ArtifactChunk`. Generally, this message represents tiny artifacts such as REPL-generated class files.
Used in:
string name = 1
The name of the artifact is expected in the form of a "Relative Path" that is made up of a sequence of directories and the final file element. Examples of "Relative Path"s: "jars/test.jar", "classes/xyz.class", "abc.xyz", "a/b/X.jar". The server is expected to maintain the hierarchy of files as defined by their name. (i.e The relative path of the file on the server's filesystem will be the same as the name of the provided artifact)
optional data = 2
A single data chunk.
message
base.proto:583Metadata of an artifact.
Used in:
string name = 1
bool is_crc_successful = 2
Whether the CRC (Cyclic Redundancy Check) is successful on server verification. The server discards any artifact that fails the CRC. If false, the client may choose to resend the artifact specified by `name`.
message
relations.proto:305Relation of type [[Aggregate]].
Used in:
optional input = 1
(Required) Input relation for a RelationalGroupedDataset.
Aggregate.GroupType group_type = 2
(Required) How the RelationalGroupedDataset was built.
repeated grouping_expressions = 3
(Required) Expressions for grouping keys
repeated aggregate_expressions = 4
(Required) List of values that will be translated to columns in the output DataFrame.
optional pivot = 5
(Optional) Pivots a column of the current `DataFrame` and performs the specified aggregation.
Used in:
GROUP_TYPE_UNSPECIFIED = 0
GROUP_TYPE_GROUPBY = 1
GROUP_TYPE_ROLLUP = 2
GROUP_TYPE_CUBE = 3
GROUP_TYPE_PIVOT = 4
message
relations.proto:329Used in:
optional col = 1
(Required) The column to pivot
repeated values = 2
(Optional) List of values that will be translated to columns in the output DataFrame. Note that if it is empty, the server side will immediately trigger a job to collect the distinct values of the column.
message
base.proto:156Used in:
string ddl_string = 1
(Required) The DDL formatted string to be parsed.
message
base.proto:99Explains the input plan based on a configurable mode.
Used in:
optional plan = 1
(Required) The logical plan to be analyzed.
Explain.ExplainMode explain_mode = 2
(Required) For analyzePlan rpc calls, configure the mode to explain plan in strings.
Plan explanation mode.
Used in:
EXPLAIN_MODE_UNSPECIFIED = 0
EXPLAIN_MODE_SIMPLE = 1
Generates only physical plan.
EXPLAIN_MODE_EXTENDED = 2
Generates parsed logical plan, analyzed logical plan, optimized logical plan and physical plan. Parsed Logical plan is a unresolved plan that extracted from the query. Analyzed logical plans transforms which translates unresolvedAttribute and unresolvedRelation into fully typed objects. The optimized logical plan transforms through a set of optimization rules, resulting in the physical plan.
EXPLAIN_MODE_CODEGEN = 3
Generates code for the statement, if any and a physical plan.
EXPLAIN_MODE_COST = 4
If plan node statistics are available, generates a logical plan and also the statistics.
EXPLAIN_MODE_FORMATTED = 5
Generates a physical plan outline and also node details.
message
base.proto:192Used in:
optional relation = 1
(Required) The logical plan to get the storage level.
message
base.proto:149Used in:
optional plan = 1
(Required) The logical plan to be analyzed.
message
base.proto:139Used in:
optional plan = 1
(Required) The logical plan to be analyzed.
message
base.proto:144Used in:
optional plan = 1
(Required) The logical plan to be analyzed.
message
base.proto:176Used in:
optional relation = 1
(Required) The logical plan to persist.
optional storage_level = 2
(Optional) The storage level.
message
base.proto:163Returns `true` when the logical query plans are equal and therefore return same results.
Used in:
optional target_plan = 1
(Required) The plan to be compared.
optional other_plan = 2
(Required) The other plan to be compared.
message
base.proto:93Used in:
optional plan = 1
(Required) The logical plan to be analyzed.
message
base.proto:171Used in:
optional plan = 1
(Required) The logical plan to get a hashCode.
message
base.proto:154Used in:
(message has no fields)
message
base.proto:131Used in:
optional plan = 1
(Required) The logical plan to be analyzed.
optional int32 level = 2
(Optional) Max level of the schema.
message
base.proto:184Used in:
optional relation = 1
(Required) The logical plan to unpersist.
optional bool blocking = 2
(Optional) Whether to block until all blocks are deleted.
message
base.proto:248Used in:
optional parsed = 1
message
base.proto:223Used in:
string explain_string = 1
message
base.proto:264Used in:
optional storage_level = 1
(Required) The StorageLevel as a result of get_storage_level request.
message
base.proto:239Used in:
repeated string files = 1
A best-effort snapshot of the files that compose this Dataset
message
base.proto:231Used in:
bool is_local = 1
message
base.proto:235Used in:
bool is_streaming = 1
message
base.proto:260Used in:
(message has no fields)
message
base.proto:252Used in:
bool result = 1
message
base.proto:219Used in:
optional schema = 1
message
base.proto:256Used in:
int32 result = 1
message
base.proto:244Used in:
string version = 1
message
base.proto:227Used in:
string tree_string = 1
message
base.proto:262Used in:
(message has no fields)
message
relations.proto:922Used in:
optional input = 1
(Required) Input relation for applyInPandasWithState.
repeated grouping_expressions = 2
(Required) Expressions for grouping keys.
optional func = 3
(Required) Input user-defined function.
string output_schema = 4
(Required) Schema for the output DataFrame.
string state_schema = 5
(Required) Schema for the state.
string output_mode = 6
(Required) The output mode of the function.
string timeout_conf = 7
(Required) Timeout configuration for groups that do not receive data for a while.
message
base.proto:624Used in:
bool exists = 1
Exists or not particular artifact at the server.
message
catalog.proto:201See `spark.catalog.cacheTable`
Used in:
string table_name = 1
(Required)
optional storage_level = 2
(Optional)
message
relations.proto:402A local relation that has been cached already.
Used in:
string hash = 3
(Required) A sha-256 hash of the serialized local relation in proto, see LocalRelation.
message
relations.proto:414Represents a remote relation that has been cached on server.
Used in:
string relation_id = 1
(Required) ID of the remote related (assigned by the service).
message
expressions.proto:376Used in:
string function_name = 1
(Required) Unparsed name of the SQL function.
repeated arguments = 2
(Optional) Function arguments. Empty arguments are allowed.
message
catalog.proto:30Catalog messages are marked as unstable.
Used in:
oneof cat_type
current_database = 1
set_current_database = 2
list_databases = 3
list_tables = 4
list_functions = 5
list_columns = 6
get_database = 7
get_table = 8
get_function = 9
database_exists = 10
table_exists = 11
function_exists = 12
create_external_table = 13
create_table = 14
drop_temp_view = 15
drop_global_temp_view = 16
recover_partitions = 17
is_cached = 18
cache_table = 19
uncache_table = 20
clear_cache = 21
refresh_table = 22
refresh_by_path = 23
current_catalog = 24
set_current_catalog = 25
list_catalogs = 26
message
catalog.proto:216See `spark.catalog.clearCache`
Used in:
(message has no fields)
message
relations.proto:899Used in:
optional input = 1
(Required) One input relation for CoGroup Map API - applyInPandas.
repeated input_grouping_expressions = 2
Expressions for grouping keys of the first input relation.
optional other = 3
(Required) The other input relation.
repeated other_grouping_expressions = 4
Expressions for grouping keys of the other input relation.
optional func = 5
(Required) Input user-defined function.
repeated input_sorting_expressions = 6
(Optional) Expressions for sorting. Only used by Scala Sorted CoGroup Map API.
repeated other_sorting_expressions = 7
(Optional) Expressions for sorting. Only used by Scala Sorted CoGroup Map API.
message
relations.proto:976Collect arbitrary (named) metrics from a dataset.
Used in:
optional input = 1
(Required) The input relation.
string name = 2
(Required) Name of the metrics.
repeated metrics = 3
(Required) The metric sequence.
message
commands.proto:33A [[Command]] is an operation that is executed by the server that does not directly consume or produce a relational result.
Used in:
oneof command_type
register_function = 1
write_operation = 2
create_dataframe_view = 3
write_operation_v2 = 4
sql_command = 5
write_stream_operation_start = 6
streaming_query_command = 7
get_resources_command = 8
streaming_query_manager_command = 9
register_table_function = 10
extension = 999
This field is used to mark extensions to the protocol. When plugins generate arbitrary Commands they can add them here. During the planning the correct resolution is done.
message
expressions.proto:328Used in: , , , , ,
string function_name = 1
(Required) Name of the user-defined function.
bool deterministic = 2
(Optional) Indicate if the user-defined function is deterministic.
repeated arguments = 3
(Optional) Function arguments. Empty arguments are allowed.
oneof function
(Required) Indicate the function type of the user-defined function.
python_udf = 4
scalar_scala_udf = 5
java_udf = 6
message
relations.proto:945Used in: ,
string function_name = 1
(Required) Name of the user-defined table function.
bool deterministic = 2
(Optional) Whether the user-defined table function is deterministic.
repeated arguments = 3
(Optional) Function input arguments. Empty arguments are allowed.
oneof function
(Required) Type of the user-defined table function.
python_udtf = 4
message
base.proto:457Used in:
repeated string keys = 1
(Required) The config keys to get.
message
base.proto:472Used in:
optional string prefix = 1
(Optional) The prefix of the config key to get.
message
base.proto:467Used in:
repeated string keys = 1
(Required) The config keys to get optionally.
message
base.proto:462Used in:
repeated pairs = 1
(Required) The config key-value paris to get. The value will be used as the default value.
message
base.proto:482Used in:
repeated string keys = 1
(Required) The config keys to check the config is modifiable.
message
base.proto:440Used in:
oneof op_type
set = 1
get = 2
get_with_default = 3
get_option = 4
get_all = 5
unset = 6
is_modifiable = 7
message
base.proto:452Used in:
repeated pairs = 1
(Required) The config key-value pairs to set.
message
base.proto:477Used in:
repeated string keys = 1
(Required) The config keys to unset.
message
commands.proto:71A command that can create DataFrame global temp view or local temp view.
Used in:
optional input = 1
(Required) The relation that this view will be built on.
string name = 2
(Required) View name.
bool is_global = 3
(Required) Whether this is global temp view or local temp view.
bool replace = 4
(Required) If true, and if the view already exists, updates it; if false, and if the view already exists, throws exception.
message
catalog.proto:145See `spark.catalog.createExternalTable`
Used in:
string table_name = 1
(Required)
optional string path = 2
(Optional)
optional string source = 3
(Optional)
optional schema = 4
(Optional)
map<string, string> options = 5
Options could be empty for valid data source format. The map key is case insensitive.
message
catalog.proto:160See `spark.catalog.createTable`
Used in:
string table_name = 1
(Required)
optional string path = 2
(Optional)
optional string source = 3
(Optional)
optional string description = 4
(Optional)
optional schema = 5
(Optional)
map<string, string> options = 6
Options could be empty for valid data source format. The map key is case insensitive.
message
catalog.proto:231See `spark.catalog.currentCatalog`
Used in:
(message has no fields)
message
catalog.proto:62See `spark.catalog.currentDatabase`
Used in:
(message has no fields)
message
types.proto:28This message describes the logical [[DataType]] of something. It does not carry the value itself but only describes it.
Used in: , , , , , , , , , , , , , , , , , , ,
oneof kind
null = 1
binary = 2
boolean = 3
byte = 4
Numeric types
short = 5
integer = 6
long = 7
float = 8
double = 9
decimal = 10
string = 11
String types
char = 12
var_char = 13
date = 14
Datatime types
timestamp = 15
timestamp_ntz = 16
calendar_interval = 17
Interval types
year_month_interval = 18
day_time_interval = 19
array = 20
Complex types
struct = 21
map = 22
udt = 23
UserDefinedType
unparsed = 24
UnparsedDataType
message
types.proto:170Used in:
optional element_type = 1
bool contains_null = 2
uint32 type_variation_reference = 3
message
types.proto:105Used in:
uint32 type_variation_reference = 1
message
types.proto:73Used in:
uint32 type_variation_reference = 1
message
types.proto:77Used in:
uint32 type_variation_reference = 1
message
types.proto:125Used in:
uint32 type_variation_reference = 1
message
types.proto:142Start compound types.
Used in:
int32 length = 1
uint32 type_variation_reference = 2
message
types.proto:117Used in:
uint32 type_variation_reference = 1
message
types.proto:135Used in:
optional int32 start_field = 1
optional int32 end_field = 2
uint32 type_variation_reference = 3
message
types.proto:152Used in:
optional int32 scale = 1
optional int32 precision = 2
uint32 type_variation_reference = 3
message
types.proto:97Used in:
uint32 type_variation_reference = 1
message
types.proto:93Used in:
uint32 type_variation_reference = 1
message
types.proto:85Used in:
uint32 type_variation_reference = 1
message
types.proto:89Used in:
uint32 type_variation_reference = 1
message
types.proto:176Used in:
optional key_type = 1
optional value_type = 2
bool value_contains_null = 3
uint32 type_variation_reference = 4
message
types.proto:109Used in:
uint32 type_variation_reference = 1
message
types.proto:81Used in:
uint32 type_variation_reference = 1
message
types.proto:101Used in:
uint32 type_variation_reference = 1
message
types.proto:165Used in:
repeated fields = 1
uint32 type_variation_reference = 2
message
types.proto:158Used in:
string name = 1
optional data_type = 2
bool nullable = 3
optional string metadata = 4
message
types.proto:113Used in:
uint32 type_variation_reference = 1
message
types.proto:121Used in:
uint32 type_variation_reference = 1
message
types.proto:183Used in:
string type = 1
optional string jvm_class = 2
optional string python_class = 3
optional string serialized_python_class = 4
optional sql_type = 5
message
types.proto:191Used in:
string data_type_string = 1
(Required) The unparsed data type string
message
types.proto:147Used in:
int32 length = 1
uint32 type_variation_reference = 2
message
types.proto:129Used in:
optional int32 start_field = 1
optional int32 end_field = 2
uint32 type_variation_reference = 3
message
catalog.proto:123See `spark.catalog.databaseExists`
Used in:
string db_name = 1
(Required)
message
relations.proto:369Relation of type [[Deduplicate]] which have duplicate rows removed, could consider either only the subset of columns or all the columns.
Used in:
optional input = 1
(Required) Input relation for a Deduplicate.
repeated string column_names = 2
(Optional) Deduplicate based on a list of column names. This field does not co-use with `all_columns_as_keys`.
optional bool all_columns_as_keys = 3
(Optional) Deduplicate based on all the columns of the input relation. This field does not co-use with `column_names`.
optional bool within_watermark = 4
(Optional) Deduplicate within the time range of watermark.
message
relations.proto:355Drop specified columns.
Used in:
optional input = 1
(Required) The input relation.
repeated columns = 2
(Optional) columns to drop.
repeated string column_names = 3
(Optional) names of columns to drop.
message
catalog.proto:183See `spark.catalog.dropGlobalTempView`
Used in:
string view_name = 1
(Required)
message
catalog.proto:177See `spark.catalog.dropTempView`
Used in:
string view_name = 1
(Required)
message
base.proto:304Used in:
oneof request_option
reattach_options = 1
extension = 999
Extension type for request options
message
base.proto:320The response of a query, can be one or more for each request. Responses belonging to the same input query, carry the same `session_id`.
Used as response type in: SparkConnectService.ExecutePlan, SparkConnectService.ReattachExecute
string session_id = 1
string operation_id = 12
Identifies the ExecutePlan execution. If set by the client in ExecutePlanRequest.operationId, that value is returned. Otherwise generated by the server. It is an UUID string of the format `00112233-4455-6677-8899-aabbccddeeff`
string response_id = 13
Identified the response in the stream. The id is an UUID string of the format `00112233-4455-6677-8899-aabbccddeeff`
oneof response_type
Union type for the different response messages.
arrow_batch = 2
sql_command_result = 5
Special case for executing SQL commands.
write_stream_operation_start_result = 8
Response for a streaming query.
streaming_query_command_result = 9
Response for commands on a streaming query.
get_resources_command_result = 10
Response for 'SparkContext.resources'.
streaming_query_manager_command_result = 11
Response for commands on the streaming query manager.
result_complete = 14
Response type informing if the stream is complete in reattachable execution.
extension = 999
Support arbitrary result objects.
optional metrics = 4
Metrics for the query execution. Typically, this field is only present in the last batch of results and then represent the overall state of the query execution.
repeated observed_metrics = 6
The metrics observed during the execution of the query plan.
optional schema = 7
(Optional) The Spark schema. This field is available when `collect` is called.
message
base.proto:376Batch results of metrics.
Used in:
int64 row_count = 1
bytes data = 2
message
base.proto:381Used in:
repeated metrics = 1
message
base.proto:385Used in:
string name = 1
int64 plan_id = 2
int64 parent = 3
map<string, > execution_metrics = 4
message
base.proto:392Used in:
string name = 1
int64 value = 2
string metric_type = 3
message
base.proto:399Used in:
string name = 1
repeated values = 2
message
base.proto:404If present, in a reattachable execution this means that after server sends onComplete, the execution is complete. If the server sends onComplete without sending a ResultComplete, it means that there is more, and the client should use ReattachExecute RPC to continue.
Used in:
(message has no fields)
message
base.proto:371A SQL command returns an opaque Relation that can be directly used as input for the next call.
Used in:
optional relation = 1
message
expressions.proto:31Expression used to refer to fields, functions and similar. This can be used everywhere expressions in SQL appear.
Used in: , , , , , , , , , , , , , , , , , , , , , , , , , , ,
oneof expr_type
literal = 1
unresolved_attribute = 2
unresolved_function = 3
expression_string = 4
unresolved_star = 5
alias = 6
cast = 7
unresolved_regex = 8
sort_order = 9
lambda_function = 10
window = 11
unresolved_extract_value = 12
update_fields = 13
unresolved_named_lambda_variable = 14
common_inline_user_defined_function = 15
call_function = 16
extension = 999
This field is used to mark extensions to the protocol. When plugins generate arbitrary relations they can add them here. During the planning the correct resolution is done.
message
expressions.proto:297Used in: ,
optional expr = 1
(Required) The expression that alias will be added on.
repeated string name = 2
(Required) a list of name parts for the alias. Scalar columns only has one name that presents.
optional string metadata = 3
(Optional) Alias metadata expressed as a JSON map.
message
expressions.proto:139Used in:
optional expr = 1
(Required) the expression to be casted.
oneof cast_to_type
(Required) the data type that the expr to be casted to.
type = 2
string type_str = 3
If this is set, Server will use Catalyst parser to parse this string to DataType.
message
expressions.proto:247Expression as string.
Used in:
string expression = 1
(Required) A SQL expression that will be parsed by Catalyst parser.
message
expressions.proto:310Used in:
optional function = 1
(Required) The lambda function. The function body should use 'UnresolvedAttribute' as arguments, the sever side will replace 'UnresolvedAttribute' with 'UnresolvedNamedLambdaVariable'.
repeated arguments = 2
(Required) Function variables. Must contains 1 ~ 3 variables.
message
expressions.proto:151Used in: , , , , , , , , , ,
oneof literal_type
null = 1
bytes binary = 2
bool boolean = 3
int32 byte = 4
int32 short = 5
int32 integer = 6
int64 long = 7
float float = 10
double double = 11
decimal = 12
string string = 13
int32 date = 16
Date in units of days since the UNIX epoch.
int64 timestamp = 17
Timestamp in units of microseconds since the UNIX epoch.
int64 timestamp_ntz = 18
Timestamp in units of microseconds since the UNIX epoch (without timezone information).
calendar_interval = 19
int32 year_month_interval = 20
int64 day_time_interval = 21
array = 22
map = 23
struct = 24
message
expressions.proto:198Used in:
optional element_type = 1
repeated elements = 2
message
expressions.proto:192Used in:
int32 months = 1
int32 days = 2
int64 microseconds = 3
message
expressions.proto:182Used in:
string value = 1
the string representation.
optional int32 precision = 2
The maximum number of digits allowed in the value. the maximum precision is 38.
optional int32 scale = 3
declared scale of decimal literal
message
expressions.proto:203Used in:
optional key_type = 1
optional value_type = 2
repeated keys = 3
repeated values = 4
message
expressions.proto:210Used in:
optional struct_type = 1
repeated elements = 2
message
expressions.proto:116SortOrder is used to specify the data ordering, it is normally used in Sort and Window. It is an unevaluable expression and cannot be evaluated, so can not be used in Projection.
Used in: , ,
optional child = 1
(Required) The expression to be sorted.
SortOrder.SortDirection direction = 2
(Required) The sort direction, should be ASCENDING or DESCENDING.
SortOrder.NullOrdering null_ordering = 3
(Required) How to deal with NULLs, should be NULLS_FIRST or NULLS_LAST.
Used in:
SORT_NULLS_UNSPECIFIED = 0
SORT_NULLS_FIRST = 1
SORT_NULLS_LAST = 2
Used in:
SORT_DIRECTION_UNSPECIFIED = 0
SORT_DIRECTION_ASCENDING = 1
SORT_DIRECTION_DESCENDING = 2
message
expressions.proto:218An unresolved attribute that is not explicitly bound to a specific column, but the column is resolved during analysis by name.
Used in:
string unparsed_identifier = 1
(Required) An identifier that will be parsed by Catalyst parser. This should follow the Spark SQL identifier syntax.
optional int64 plan_id = 2
(Optional) The id of corresponding connect plan.
message
expressions.proto:273Extracts a value or values from an Expression
Used in:
optional child = 1
(Required) The expression to extract value from, can be Map, Array, Struct or array of Structs.
optional extraction = 2
(Required) The expression to describe the extraction, can be key of Map, index of Array, field name of Struct.
message
expressions.proto:229An unresolved function is not explicitly bound to one explicit function, but the function is resolved during analysis following Sparks name resolution rules.
Used in:
string function_name = 1
(Required) name (or unparsed name for user defined function) for the unresolved function.
repeated arguments = 2
(Optional) Function arguments. Empty arguments are allowed.
bool is_distinct = 3
(Required) Indicate if this function should be applied on distinct values.
bool is_user_defined_function = 4
(Required) Indicate if this is a user defined function. When it is not a user defined function, Connect will use the function name directly. When it is a user defined function, Connect will parse the function name first.
message
expressions.proto:321Used in: ,
repeated string name_parts = 1
(Required) a list of name parts for the variable. Must not be empty.
message
expressions.proto:264Represents all of the input attributes to a given relational operator, for example in "SELECT `(id)?+.+` FROM ...".
Used in:
string col_name = 1
(Required) The column name used to extract column with regex.
optional int64 plan_id = 2
(Optional) The id of corresponding connect plan.
message
expressions.proto:253UnresolvedStar is used to expand all the fields of a relation or struct.
Used in:
optional string unparsed_target = 1
(Optional) The target of the expansion. If set, it should end with '.*' and will be parsed by 'parseAttributeName' in the server side.
message
expressions.proto:284Add, replace or drop a field of `StructType` expression by name.
Used in:
optional struct_expression = 1
(Required) The struct expression.
string field_name = 2
(Required) The field name.
optional value_expression = 3
(Optional) The expression to add or replace. When not set, it means this field will be dropped.
message
expressions.proto:58Expression for the OVER clause or WINDOW clause.
Used in:
optional window_function = 1
(Required) The window function.
repeated partition_spec = 2
(Optional) The way that input rows are partitioned.
repeated order_spec = 3
(Optional) Ordering of rows in a partition.
optional frame_spec = 4
(Optional) Window frame in a partition. If not set, it will be treated as 'UnspecifiedFrame'.
message
expressions.proto:75The window frame
Used in:
WindowFrame.FrameType frame_type = 1
(Required) The type of the frame.
optional lower = 2
(Required) The lower bound of the frame.
optional upper = 3
(Required) The upper bound of the frame.
message
expressions.proto:97Used in:
oneof boundary
bool current_row = 1
CURRENT ROW boundary
bool unbounded = 2
UNBOUNDED boundary. For lower bound, it will be converted to 'UnboundedPreceding'. for upper bound, it will be converted to 'UnboundedFollowing'.
value = 3
This is an expression for future proofing. We are expecting literals on the server side.
Used in:
FRAME_TYPE_UNDEFINED = 0
FRAME_TYPE_ROW = 1
RowFrame treats rows in a partition individually.
FRAME_TYPE_RANGE = 2
RangeFrame treats rows in a partition as groups of peers. All rows having the same 'ORDER BY' ordering are considered as peers.
message
relations.proto:186Relation that applies a boolean expression `condition` on each row of `input` to produce the output result.
Used in:
optional input = 1
(Required) Input relation for a Filter.
optional condition = 2
(Required) A Filter must have a condition expression.
message
catalog.proto:137See `spark.catalog.functionExists`
Used in:
string function_name = 1
(Required)
optional string db_name = 2
(Optional)
message
catalog.proto:101See `spark.catalog.getDatabase`
Used in:
string db_name = 1
(Required)
message
catalog.proto:115See `spark.catalog.getFunction`
Used in:
string function_name = 1
(Required)
optional string db_name = 2
(Optional)
message
commands.proto:411Command to get the output of 'SparkContext.resources'
Used in:
(message has no fields)
message
commands.proto:414Response for command 'GetResourcesCommand'.
Used in:
map<string, > resources = 1
message
catalog.proto:107See `spark.catalog.getTable`
Used in:
string table_name = 1
(Required)
optional string db_name = 2
(Optional)
message
relations.proto:869Used in:
optional input = 1
(Required) Input relation for Group Map API: apply, applyInPandas.
repeated grouping_expressions = 2
(Required) Expressions for grouping keys.
optional func = 3
(Required) Input user-defined function.
repeated sorting_expressions = 4
(Optional) Expressions for sorting. Only used by Scala Sorted Group Map API.
optional initial_input = 5
Below fields are only used by (Flat)MapGroupsWithState (Optional) Input relation for initial State.
repeated initial_grouping_expressions = 6
(Optional) Expressions for grouping keys of the initial state input relation.
optional bool is_map_groups_with_state = 7
(Optional) True if MapGroupsWithState, false if FlatMapGroupsWithState.
optional string output_mode = 8
(Optional) The output mode of the function.
optional string timeout_conf = 9
(Optional) Timeout configuration for groups that do not receive data for a while.
message
relations.proto:800Specify a hint over a relation. Hint should have a name and optional parameters.
Used in:
optional input = 1
(Required) The input relation.
string name = 2
(Required) Hint name. Supported Join hints include BROADCAST, MERGE, SHUFFLE_HASH, SHUFFLE_REPLICATE_NL. Supported partitioning hints include COALESCE, REPARTITION, REPARTITION_BY_RANGE.
repeated parameters = 3
(Optional) Hint parameters.
message
relations.proto:501Compose the string representing rows for output. It will invoke 'Dataset.htmlString' to compute the results.
Used in:
optional input = 1
(Required) The input relation.
int32 num_rows = 2
(Required) Number of rows to show.
int32 truncate = 3
(Required) If set to more than 0, truncates strings to `truncate` characters and all cells will be aligned right.
Used in:
INTERRUPT_TYPE_UNSPECIFIED = 0
INTERRUPT_TYPE_ALL = 1
Interrupt all running executions within the session with the provided session_id.
INTERRUPT_TYPE_TAG = 2
Interrupt all running executions within the session with the provided operation_tag.
INTERRUPT_TYPE_OPERATION_ID = 3
Interrupt the running execution within the session with the provided operation_id.
message
catalog.proto:195See `spark.catalog.isCached`
Used in:
string table_name = 1
(Required)
message
expressions.proto:365Used in:
string class_name = 1
(Required) Fully qualified name of Java class
optional output_type = 2
(Optional) Output type of the Java UDF
bool aggregate = 3
(Required) Indicate if the Java user-defined function is an aggregate function
message
relations.proto:197Relation of type [[Join]]. `left` and `right` must be present.
Used in:
optional left = 1
(Required) Left input relation for a Join.
optional right = 2
(Required) Right input relation for a Join.
optional join_condition = 3
(Optional) The join condition. Could be unset when `using_columns` is utilized. This field does not co-exist with using_columns.
Join.JoinType join_type = 4
(Required) The join type.
repeated string using_columns = 5
Optional. using_columns provides a list of columns that should present on both sides of the join inputs that this Join will join on. For example A JOIN B USING col_name is equivalent to A JOIN B on A.col_name = B.col_name. This field does not co-exist with join_condition.
optional join_data_type = 6
(Optional) Only used by joinWith. Set the left and right join data types.
message
relations.proto:233Used in:
bool is_left_struct = 1
If the left data type is a struct.
bool is_right_struct = 2
If the right data type is a struct.
Used in:
JOIN_TYPE_UNSPECIFIED = 0
JOIN_TYPE_INNER = 1
JOIN_TYPE_FULL_OUTER = 2
JOIN_TYPE_LEFT_OUTER = 3
JOIN_TYPE_RIGHT_OUTER = 4
JOIN_TYPE_LEFT_ANTI = 5
JOIN_TYPE_LEFT_SEMI = 6
JOIN_TYPE_CROSS = 7
message
base.proto:412The key-value pair for the config request and response.
Used in: , ,
string key = 1
(Required) The key.
optional string value = 2
(Optional) The value.
message
relations.proto:277Relation of type [[Limit]] that is used to `limit` rows from the input relation.
Used in:
optional input = 1
(Required) Input relation for a Limit.
int32 limit = 2
(Required) the limit.
message
catalog.proto:240See `spark.catalog.listCatalogs`
Used in:
optional string pattern = 1
(Optional) The pattern that the catalog name needs to match
message
catalog.proto:93See `spark.catalog.listColumns`
Used in:
string table_name = 1
(Required)
optional string db_name = 2
(Optional)
message
catalog.proto:71See `spark.catalog.listDatabases`
Used in:
optional string pattern = 1
(Optional) The pattern that the database name needs to match
message
catalog.proto:85See `spark.catalog.listFunctions`
Used in:
optional string db_name = 1
(Optional)
optional string pattern = 2
(Optional) The pattern that the function name needs to match
message
catalog.proto:77See `spark.catalog.listTables`
Used in:
optional string db_name = 1
(Optional)
optional string pattern = 2
(Optional) The pattern that the table name needs to match
message
relations.proto:388A relation that does not need to be qualified by name.
Used in:
optional bytes data = 1
(Optional) Local collection data serialized into Arrow IPC streaming format which contains the schema of the data.
optional string schema = 2
(Optional) The schema of local data. It should be either a DDL-formatted type string or a JSON string. The server side will update the column names and data types according to this schema. If the 'data' is not provided, then this schema will be required.
message
relations.proto:858Used in:
optional input = 1
(Required) Input relation for a mapPartitions-equivalent API: mapInPandas, mapInArrow.
optional func = 2
(Required) Input user-defined function.
optional bool is_barrier = 3
(Optional) Whether to use barrier mode execution or not.
message
relations.proto:692Drop rows containing null values. It will invoke 'Dataset.na.drop' (same as 'DataFrameNaFunctions.drop') to compute the results.
Used in:
optional input = 1
(Required) The input relation.
repeated string cols = 2
(Optional) Optional list of column names to consider. When it is empty, all the columns in the input relation will be considered.
optional int32 min_non_nulls = 3
(Optional) The minimum number of non-null and non-NaN values required to keep. When not set, it is equivalent to the number of considered columns, which means a row will be kept only if all columns are non-null. 'how' options ('all', 'any') can be easily converted to this field: - 'all' -> set 'min_non_nulls' 1; - 'any' -> keep 'min_non_nulls' unset;
message
relations.proto:675Replaces null values. It will invoke 'Dataset.na.fill' (same as 'DataFrameNaFunctions.fill') to compute the results. Following 3 parameter combinations are supported: 1, 'values' only contains 1 item, 'cols' is empty: replaces null values in all type-compatible columns. 2, 'values' only contains 1 item, 'cols' is not empty: replaces null values in specified columns. 3, 'values' contains more than 1 items, then 'cols' is required to have the same length: replaces each specified column with corresponding value.
Used in:
optional input = 1
(Required) The input relation.
repeated string cols = 2
(Optional) Optional list of column names to consider.
repeated values = 3
(Required) Values to replace null values with. Should contain at least 1 item. Only 4 data types are supported now: bool, long, double, string
message
relations.proto:716Replaces old values with the corresponding values. It will invoke 'Dataset.na.replace' (same as 'DataFrameNaFunctions.replace') to compute the results.
Used in:
optional input = 1
(Required) The input relation.
repeated string cols = 2
(Optional) List of column names to consider. When it is empty, all the type-compatible columns in the input relation will be considered.
repeated replacements = 3
(Optional) The value replacement mapping.
message
relations.proto:728Used in:
optional old_value = 1
(Required) The old value. Only 4 data types are supported now: null, bool, double, string.
optional new_value = 2
(Required) The new value. Should be of the same data type with the old value.
message
relations.proto:287Relation of type [[Offset]] that is used to read rows staring from the `offset` on the input relation.
Used in:
optional input = 1
(Required) Input relation for an Offset.
int32 offset = 2
(Required) the limit.
message
relations.proto:987Used in:
optional input = 1
(Required) Input relation to Parse. The input is expected to have single text column.
Parse.ParseFormat format = 2
(Required) The expected format of the text.
optional schema = 3
(Optional) DataType representing the schema. If not set, Spark will infer the schema.
map<string, string> options = 4
Options for the csv/json parser. The map key is case insensitive.
Used in:
PARSE_FORMAT_UNSPECIFIED = 0
PARSE_FORMAT_CSV = 1
PARSE_FORMAT_JSON = 2
message
base.proto:37A [[Plan]] is the structure that carries the runtime information for the execution from the client to the server. A [[Plan]] can either be of the type [[Relation]] which is a reference to the underlying logical plan or it can be of the [[Command]] type that is used to execute commands on the server.
Used in: , , , , , , , ,
oneof op_type
root = 1
command = 2
message
relations.proto:174Projection of a bag of expressions for a given input relation. The input relation must be specified. The projected expression can be an arbitrary expression.
Used in:
optional input = 1
(Optional) Input relation is optional for Project. For example, `SELECT ABS(-1)` is valid plan without an input plan.
repeated expressions = 3
(Required) A Project requires at least one expression.
message
expressions.proto:343Used in: , ,
optional output_type = 1
(Required) Output type of the Python UDF
int32 eval_type = 2
(Required) EvalType of the Python UDF
bytes command = 3
(Required) The encoded commands of the Python UDF
string python_ver = 4
(Required) Python version being used in the client.
message
relations.proto:961Used in:
optional return_type = 1
(Optional) Return type of the Python UDTF.
int32 eval_type = 2
(Required) EvalType of the Python UDTF.
bytes command = 3
(Required) The encoded commands of the Python UDTF.
string python_ver = 4
(Required) Python version being used in the client.
message
relations.proto:443Relation of type [[Range]] that generates a sequence of integers.
Used in:
optional int64 start = 1
(Optional) Default value = 0
int64 end = 2
(Required)
int64 step = 3
(Required)
optional int32 num_partitions = 4
Optional. Default value is assigned by 1) SQL conf "spark.sql.leafNodeDefaultParallelism" if it is set, or 2) spark default parallelism.
message
relations.proto:127Relation that reads from a file / table or other data source. Does not have additional inputs.
Used in:
oneof read_type
named_table = 1
data_source = 2
bool is_streaming = 3
(Optional) Indicates if this is a streaming read.
message
relations.proto:144Used in:
optional string format = 1
(Optional) Supported formats include: parquet, orc, text, json, parquet, csv, avro. If not set, the value from SQL conf 'spark.sql.sources.default' will be used.
optional string schema = 2
(Optional) If not set, Spark will infer the schema. This schema string should be either DDL-formatted or JSON-formatted.
map<string, string> options = 3
Options for the data source. The context of this map varies based on the data source format. This options could be empty for valid data source format. The map key is case insensitive.
repeated string paths = 4
(Optional) A list of path for file-system backed data sources.
repeated string predicates = 5
(Optional) Condition in the where clause for each partition. This is only supported by the JDBC data source.
message
relations.proto:136Used in:
string unparsed_identifier = 1
(Required) Unparsed identifier for the table.
map<string, string> options = 2
Options for the named table. The map key is case insensitive.
message
base.proto:683Used in:
bool reattachable = 1
If true, the request can be reattached to using ReattachExecute. ReattachExecute can be used either if the stream broke with a GRPC network error, or if the server closed the stream without sending a response with StreamStatus.complete=true. The server will keep a buffer of responses in case a response is lost, and ReattachExecute needs to back-track. If false, the execution response stream will will not be reattachable, and all responses are immediately released by the server after being sent.
message
catalog.proto:189See `spark.catalog.recoverPartitions`
Used in:
string table_name = 1
(Required)
message
catalog.proto:225See `spark.catalog.refreshByPath`
Used in:
string path = 1
(Required)
message
catalog.proto:219See `spark.catalog.refreshTable`
Used in:
string table_name = 1
(Required)
message
relations.proto:35The main [[Relation]] type. Fundamentally, a relation is a typed container that has exactly one explicit relation type set. When adding new relation types, they have to be registered here.
Used in: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ,
optional common = 1
oneof rel_type
read = 2
project = 3
filter = 4
join = 5
set_op = 6
sort = 7
limit = 8
aggregate = 9
sql = 10
local_relation = 11
sample = 12
offset = 13
deduplicate = 14
range = 15
subquery_alias = 16
repartition = 17
to_df = 18
with_columns_renamed = 19
show_string = 20
drop = 21
tail = 22
with_columns = 23
hint = 24
unpivot = 25
to_schema = 26
repartition_by_expression = 27
map_partitions = 28
collect_metrics = 29
parse = 30
group_map = 31
co_group_map = 32
with_watermark = 33
apply_in_pandas_with_state = 34
html_string = 35
cached_local_relation = 36
cached_remote_relation = 37
common_inline_user_defined_table_function = 38
fill_na = 90
NA functions
drop_na = 91
replace = 92
summary = 100
stat functions
crosstab = 101
describe = 102
cov = 103
corr = 104
approx_quantile = 105
freq_items = 106
sample_by = 107
catalog = 200
Catalog API (experimental / unstable)
extension = 998
This field is used to mark extensions to the protocol. When plugins generate arbitrary relations they can add them here. During the planning the correct resolution is done.
unknown = 999
message
relations.proto:105Common metadata of all relations.
Used in:
string source_info = 1
(Required) Shared relation metadata.
optional int64 plan_id = 2
(Optional) A per-client globally unique id for a given connect plan.
message
base.proto:754Release and close operation completely. This will also interrupt the query if it is running execution, and wait for it to be torn down.
Used in:
(message has no fields)
message
base.proto:761Release all responses from the operation response stream up to and including the response with the given by response_id. While server determines by itself how much of a buffer of responses to keep, client providing explicit release calls will help reduce resource consumption. Noop if response_id not found in cached responses.
Used in:
string response_id = 1
message
relations.proto:471Relation repartition.
Used in:
optional input = 1
(Required) The input relation of Repartition.
int32 num_partitions = 2
(Required) Must be positive.
optional bool shuffle = 3
(Optional) Default value is false.
message
relations.proto:847Used in:
optional input = 1
(Required) The input relation.
repeated partition_exprs = 2
(Required) The partitioning expressions.
optional int32 num_partitions = 3
(Optional) number of partitions, must be positive.
message
common.proto:43ResourceInformation to hold information about a type of Resource. The corresponding class is 'org.apache.spark.resource.ResourceInformation'
Used in:
string name = 1
(Required) The name of the resource
repeated string addresses = 2
(Required) An array of strings describing the addresses of the resource.
message
relations.proto:114Relation that uses a SQL query to generate the output.
Used in:
string query = 1
(Required) The SQL query.
map<string, > args = 2
(Optional) A map of parameter names to literal expressions.
repeated pos_args = 3
(Optional) A sequence of literal expressions for positional parameters in the SQL query text.
message
relations.proto:420Relation of type [[Sample]] that samples a fraction of the dataset.
Used in:
optional input = 1
(Required) Input relation for a Sample.
double lower_bound = 2
(Required) lower bound.
double upper_bound = 3
(Required) upper bound.
optional bool with_replacement = 4
(Optional) Whether to sample with replacement.
optional int64 seed = 5
(Optional) The random seed.
bool deterministic_order = 6
(Required) Explicitly sort the underlying plan to make the ordering deterministic or cache it. This flag is true when invoking `dataframe.randomSplit` to randomly splits DataFrame with the provided weights. Otherwise, it is false.
message
expressions.proto:354Used in: ,
bytes payload = 1
(Required) Serialized JVM object containing UDF definition, input encoders and output encoder
repeated inputTypes = 2
(Optional) Input type(s) of the UDF
optional outputType = 3
(Required) Output type of the UDF
bool nullable = 4
(Required) True if the UDF can return null value
message
catalog.proto:234See `spark.catalog.setCurrentCatalog`
Used in:
string catalog_name = 1
(Required)
message
catalog.proto:65See `spark.catalog.setCurrentDatabase`
Used in:
string db_name = 1
(Required)
message
relations.proto:242Relation of type [[SetOperation]]
Used in:
optional left_input = 1
(Required) Left input relation for a Set operation.
optional right_input = 2
(Required) Right input relation for a Set operation.
SetOperation.SetOpType set_op_type = 3
(Required) The Set operation type.
optional bool is_all = 4
(Optional) If to remove duplicate rows. True to preserve all results. False to remove duplicate rows.
optional bool by_name = 5
(Optional) If to perform the Set operation based on name resolution. Only UNION supports this option.
optional bool allow_missing_columns = 6
(Optional) If to perform the Set operation and allow missing columns. Only UNION supports this option.
Used in:
SET_OP_TYPE_UNSPECIFIED = 0
SET_OP_TYPE_INTERSECT = 1
SET_OP_TYPE_UNION = 2
SET_OP_TYPE_EXCEPT = 3
message
relations.proto:484Compose the string representing rows for output. It will invoke 'Dataset.showString' to compute the results.
Used in:
optional input = 1
(Required) The input relation.
int32 num_rows = 2
(Required) Number of rows to show.
int32 truncate = 3
(Required) If set to more than 0, truncates strings to `truncate` characters and all cells will be aligned right.
bool vertical = 4
(Required) If set to true, prints output rows vertically (one line per column value).
message
relations.proto:342Relation of type [[Sort]].
Used in:
optional input = 1
(Required) Input relation for a Sort.
repeated order = 2
(Required) The ordering expressions
optional bool is_global = 3
(Optional) if this is a global sort.
message
commands.proto:59A SQL Command is used to trigger the eager evaluation of SQL commands in Spark. When the SQL provide as part of the message is a command it will be immediately evaluated and the result will be collected and returned as part of a LocalRelation. If the result is not a command, the operation will simply return a SQL Relation. This allows the client to be almost oblivious to the server-side behavior.
Used in:
string sql = 1
(Required) SQL Query.
map<string, > args = 2
(Optional) A map of parameter names to literal expressions.
repeated pos_args = 3
(Optional) A sequence of literal expressions for positional parameters in the SQL query text.
message
relations.proto:601Calculates the approximate quantiles of numerical columns of a DataFrame. It will invoke 'Dataset.stat.approxQuantile' (same as 'StatFunctions.approxQuantile') to compute the results.
Used in:
optional input = 1
(Required) The input relation.
repeated string cols = 2
(Required) The names of the numerical columns.
repeated double probabilities = 3
(Required) A list of quantile probabilities. Each number must belong to [0, 1]. For example 0 is the minimum, 0.5 is the median, 1 is the maximum.
double relative_error = 4
(Required) The relative target precision to achieve (greater than or equal to 0). If set to zero, the exact quantiles are computed, which could be very expensive. Note that values greater than 1 are accepted but give the same result as 1.
message
relations.proto:582Calculates the correlation of two columns of a DataFrame. Currently only supports the Pearson Correlation Coefficient. It will invoke 'Dataset.stat.corr' (same as 'StatFunctions.pearsonCorrelation') to compute the results.
Used in:
optional input = 1
(Required) The input relation.
string col1 = 2
(Required) The name of the first column.
string col2 = 3
(Required) The name of the second column.
optional string method = 4
(Optional) Default value is 'pearson'. Currently only supports the Pearson Correlation Coefficient.
message
relations.proto:568Calculate the sample covariance of two numerical columns of a DataFrame. It will invoke 'Dataset.stat.cov' (same as 'StatFunctions.calculateCov') to compute the results.
Used in:
optional input = 1
(Required) The input relation.
string col1 = 2
(Required) The name of the first column.
string col2 = 3
(Required) The name of the second column.
message
relations.proto:551Computes a pair-wise frequency table of the given columns. Also known as a contingency table. It will invoke 'Dataset.stat.crosstab' (same as 'StatFunctions.crossTabulate') to compute the results.
Used in:
optional input = 1
(Required) The input relation.
string col1 = 2
(Required) The name of the first column. Distinct items will make the first item of each row.
string col2 = 3
(Required) The name of the second column. Distinct items will make the column names of the DataFrame.
message
relations.proto:540Computes basic statistics for numeric and string columns, including count, mean, stddev, min, and max. If no columns are given, this function computes statistics for all numerical or string columns.
Used in:
optional input = 1
(Required) The input relation.
repeated string cols = 2
(Optional) Columns to compute statistics on.
message
relations.proto:624Finding frequent items for columns, possibly with false positives. It will invoke 'Dataset.stat.freqItems' (same as 'StatFunctions.freqItems') to compute the results.
Used in:
optional input = 1
(Required) The input relation.
repeated string cols = 2
(Required) The names of the columns to search frequent items in.
optional double support = 3
(Optional) The minimum frequency for an item to be considered `frequent`. Should be greater than 1e-4.
message
relations.proto:641Returns a stratified sample without replacement based on the fraction given on each stratum. It will invoke 'Dataset.stat.freqItems' (same as 'StatFunctions.freqItems') to compute the results.
Used in:
optional input = 1
(Required) The input relation.
optional col = 2
(Required) The column that defines strata.
repeated fractions = 3
(Required) Sampling fraction for each stratum. If a stratum is not specified, we treat its fraction as zero.
optional int64 seed = 5
(Optional) The random seed.
message
relations.proto:656Used in:
optional stratum = 1
(Required) The stratum.
double fraction = 2
(Required) The fraction value. Must be in [0, 1].
message
relations.proto:516Computes specified statistics for numeric and string columns. It will invoke 'Dataset.summary' (same as 'StatFunctions.summary') to compute the results.
Used in:
optional input = 1
(Required) The input relation.
repeated string statistics = 2
(Optional) Statistics from to be computed. Available statistics are: count mean stddev min max arbitrary approximate percentiles specified as a percentage (e.g. 75%) count_distinct approx_count_distinct If no statistics are given, this function computes 'count', 'mean', 'stddev', 'min', 'approximate quartiles' (percentiles at 25%, 50%, and 75%), and 'max'.
message
common.proto:27StorageLevel for persisting Datasets/Tables.
Used in: , ,
bool use_disk = 1
(Required) Whether the cache should use disk or not.
bool use_memory = 2
(Required) Whether the cache should use memory or not.
bool use_off_heap = 3
(Required) Whether the cache should use off-heap or not.
bool deserialized = 4
(Required) Whether the cached data is deserialized or not.
int32 replication = 5
(Required) The number of replicas.
message
commands.proto:222Used in:
oneof function
python_function = 1
scala_function = 2
message
commands.proto:258Commands for a streaming query.
Used in:
optional query_id = 1
(Required) Query instance. See `StreamingQueryInstanceId`.
oneof command
See documentation for the corresponding API method in StreamingQuery.
bool status = 2
status() API.
bool last_progress = 3
lastProgress() API.
bool recent_progress = 4
recentProgress() API.
bool stop = 5
stop() API. Stops the query.
bool process_all_available = 6
processAllAvailable() API. Waits till all the available data is processed
explain = 7
explain() API. Returns logical and physical plans.
bool exception = 8
exception() API. Returns the exception in the query if any.
await_termination = 9
awaitTermination() API. Waits for the termination of the query.
message
commands.proto:289Used in:
optional int64 timeout_ms = 2
message
commands.proto:283Used in:
bool extended = 1
TODO: Consider reusing Explain from AnalyzePlanRequest message. We can not do this right now since it base.proto imports this file.
message
commands.proto:295Response for commands on a streaming query.
Used in:
optional query_id = 1
(Required) Query instance id. See `StreamingQueryInstanceId`.
oneof result_type
status = 2
recent_progress = 3
explain = 4
exception = 5
await_termination = 6
message
commands.proto:335Used in:
bool terminated = 1
message
commands.proto:325Used in:
optional string exception_message = 1
(Optional) Exception message as string, maps to the return value of original StreamingQueryException's toString method
optional string error_class = 2
(Optional) Exception error class as string
optional string stack_trace = 3
(Optional) Exception stack trace as string
message
commands.proto:320Used in:
string result = 1
Logical and physical plans as string
message
commands.proto:315Used in:
repeated string recent_progress_json = 5
Progress reports as an array of json strings.
message
commands.proto:307Used in:
string status_message = 1
See documentation for these Scala 'StreamingQueryStatus' struct
bool is_data_available = 2
bool is_trigger_active = 3
bool is_active = 4
message
commands.proto:244A tuple that uniquely identifies an instance of streaming query run. It consists of `id` that persists across the streaming runs and `run_id` that changes between each run of the streaming query that resumes from the checkpoint.
Used in: , , ,
string id = 1
(Required) The unique id of this query that persists across restarts from checkpoint data. That is, this id is generated when a query is started for the first time, and will be the same every time it is restarted from checkpoint data.
string run_id = 2
(Required) The unique id of this run of the query. That is, every start/restart of a query will generate a unique run_id. Therefore, every time a query is restarted from checkpoint, it will have the same `id` but different `run_id`s.
message
commands.proto:341Commands for the streaming query manager.
Used in:
oneof command
See documentation for the corresponding API method in StreamingQueryManager.
bool active = 1
active() API, returns a list of active queries.
string get_query = 2
get() API, returns the StreamingQuery identified by id.
await_any_termination = 3
awaitAnyTermination() API, wait until any query terminates or timeout.
bool reset_terminated = 4
resetTerminated() API.
add_listener = 5
addListener API.
remove_listener = 6
removeListener API.
bool list_listeners = 7
listListeners() API, returns a list of streaming query listeners.
message
commands.proto:361Used in:
optional int64 timeout_ms = 1
(Optional) The waiting time in milliseconds to wait for any query to terminate.
message
commands.proto:366Used in:
bytes listener_payload = 1
optional python_listener_payload = 2
string id = 3
message
commands.proto:374Response for commands on the streaming query manager.
Used in:
oneof result_type
active = 1
query = 2
await_any_termination = 3
bool reset_terminated = 4
bool add_listener = 5
bool remove_listener = 6
list_listeners = 7
message
commands.proto:385Used in:
repeated active_queries = 1
message
commands.proto:396Used in:
bool terminated = 1
message
commands.proto:404Used in:
repeated string listener_ids = 1
(Required) Reference IDs of listener instances.
message
commands.proto:389Used in: ,
optional id = 1
(Required) The id and runId of this query.
optional string name = 2
(Optional) The name of this query.
message
commands.proto:400bytes listener_payload = 1
message
relations.proto:459Relation alias.
Used in:
optional input = 1
(Required) The input relation of SubqueryAlias.
string alias = 2
(Required) The alias.
repeated string qualifier = 3
(Optional) Qualifier of the alias.
message
catalog.proto:129See `spark.catalog.tableExists`
Used in:
string table_name = 1
(Required)
optional string db_name = 2
(Optional)
message
relations.proto:296Relation of type [[Tail]] that is used to fetch `limit` rows from the last of the input relation.
Used in:
optional input = 1
(Required) Input relation for an Tail.
int32 limit = 2
(Required) the limit.
message
relations.proto:743Rename columns on the input relation by the same length of names.
Used in:
optional input = 1
(Required) The input relation of RenameColumnsBySameLengthNames.
repeated string column_names = 2
(Required) The number of columns of the input relation must be equal to the length of this field. If this is not true, an exception will be returned.
message
relations.proto:837Used in:
optional input = 1
(Required) The input relation.
optional schema = 2
(Required) The user provided schema. The Sever side will update the dataframe with this schema.
message
catalog.proto:210See `spark.catalog.uncacheTable`
Used in:
string table_name = 1
(Required)
message
relations.proto:102Used for testing purposes only.
Used in:
(message has no fields)
message
relations.proto:816Unpivot a DataFrame from wide format to long format, optionally leaving identifier columns set.
Used in:
optional input = 1
(Required) The input relation.
repeated ids = 2
(Required) Id columns.
optional values = 3
(Optional) Value columns to unpivot.
string variable_column_name = 4
(Required) Name of the variable column.
string value_column_name = 5
(Required) Name of the value column.
message
relations.proto:832Used in:
repeated values = 1
message
base.proto:48User Context is used to refer to one particular user session that is executing queries in the backend.
Used in: , , , , , , ,
string user_id = 1
string user_name = 2
repeated extensions = 999
To extend the existing user context message that is used to identify incoming requests, Spark Connect leverages the Any protobuf type that can be used to inject arbitrary other messages into this message. Extensions are stored as a `repeated` type to be able to handle multiple active extensions.
message
relations.proto:771Adding columns or replacing the existing columns that have the same names.
Used in:
optional input = 1
(Required) The input relation.
repeated aliases = 2
(Required) Given a column name, apply the corresponding expression on the column. If column name exists in the input relation, then replace the column. If the column name does not exist in the input relation, then adds it as a new column. Only one name part is expected from each Expression.Alias. An exception is thrown when duplicated names are present in the mapping.
message
relations.proto:756Rename columns on the input relation by a map with name to name mapping.
Used in:
optional input = 1
(Required) The input relation.
map<string, string> rename_columns_map = 2
(Required) Renaming column names of input relation from A to B where A is the map key and B is the map value. This is a no-op if schema doesn't contain any A. It does not require that all input relation column names to present as keys. duplicated B are not allowed.
message
relations.proto:787Used in:
optional input = 1
(Required) The input relation
string event_time = 2
(Required) Name of the column containing event time.
string delay_threshold = 3
(Required)
message
commands.proto:89As writes are not directly handled during analysis and planning, they are modeled as commands.
Used in:
optional input = 1
(Required) The output of the `input` relation will be persisted according to the options.
optional string source = 2
(Optional) Format value according to the Spark documentation. Examples are: text, parquet, delta.
oneof save_type
(Optional) The destination of the write operation can be either a path or a table. If the destination is neither a path nor a table, such as jdbc and noop, the `save_type` should not be set.
string path = 3
table = 4
WriteOperation.SaveMode mode = 5
(Required) the save mode.
repeated string sort_column_names = 6
(Optional) List of columns to sort the output by.
repeated string partitioning_columns = 7
(Optional) List of columns for partitioning.
optional bucket_by = 8
(Optional) Bucketing specification. Bucketing must set the number of buckets and the columns to bucket by.
map<string, string> options = 9
(Optional) A list of configuration options.
message
commands.proto:135Used in:
repeated string bucket_column_names = 1
int32 num_buckets = 2
Used in:
SAVE_MODE_UNSPECIFIED = 0
SAVE_MODE_APPEND = 1
SAVE_MODE_OVERWRITE = 2
SAVE_MODE_ERROR_IF_EXISTS = 3
SAVE_MODE_IGNORE = 4
message
commands.proto:122Used in:
string table_name = 1
(Required) The table name.
SaveTable.TableSaveMethod save_method = 2
(Required) The method to be called to write to the table.
Used in:
TABLE_SAVE_METHOD_UNSPECIFIED = 0
TABLE_SAVE_METHOD_SAVE_AS_TABLE = 1
TABLE_SAVE_METHOD_INSERT_INTO = 2
message
commands.proto:150As writes are not directly handled during analysis and planning, they are modeled as commands.
Used in:
optional input = 1
(Required) The output of the `input` relation will be persisted according to the options.
string table_name = 2
(Required) The destination of the write operation must be either a path or a table.
optional string provider = 3
(Optional) A provider for the underlying output data source. Spark's default catalog supports "parquet", "json", etc.
repeated partitioning_columns = 4
(Optional) List of columns for partitioning for output table created by `create`, `createOrReplace`, or `replace`
map<string, string> options = 5
(Optional) A list of configuration options.
map<string, string> table_properties = 6
(Optional) A list of table properties.
WriteOperationV2.Mode mode = 7
(Required) Write mode.
optional overwrite_condition = 8
(Optional) A condition for overwrite saving mode
Used in:
MODE_UNSPECIFIED = 0
MODE_CREATE = 1
MODE_OVERWRITE = 2
MODE_OVERWRITE_PARTITIONS = 3
MODE_APPEND = 4
MODE_REPLACE = 5
MODE_CREATE_OR_REPLACE = 6
message
commands.proto:190Starts write stream operation as streaming query. Query ID and Run ID of the streaming query are returned.
Used in:
optional input = 1
(Required) The output of the `input` streaming relation will be written.
string format = 2
map<string, string> options = 3
repeated string partitioning_column_names = 4
oneof trigger
string processing_time_interval = 5
bool available_now = 6
bool once = 7
string continuous_checkpoint_interval = 8
string output_mode = 9
string query_name = 10
oneof sink_destination
The destination is optional. When set, it can be a path or a table name.
string path = 11
string table_name = 12
optional foreach_writer = 13
optional foreach_batch = 14
message
commands.proto:229Used in:
optional query_id = 1
(Required) Query instance. See `StreamingQueryInstanceId`.
string name = 2
An optional query name.