package delta.connect

Get desktop application:
View/edit binary Protocol Buffers messages

Command to add a supported feature to the table by modifying the protocol.

Used in: DeltaCommand

• optional DeltaTable table = 1

  (Required) The Delta table to add the supported feature to.

• string feature_name = 2

  (Required) The name of the supported feature to add.

Represents an assignment of a value to a field.

Used in: MergeIntoTable.Action.InsertAction, MergeIntoTable.Action.UpdateAction, UpdateTable

• optional spark.connect.Expression field = 1

  (Required) Expression identifying the (struct) field that is assigned a new value.

• optional spark.connect.Expression value = 2

  (Required) Expression that produces the value to assign to the field.

Command that creates a copy of a DeltaTable in the specified target location.

Used in: DeltaCommand

• optional DeltaTable table = 1

  (Required) The source Delta table to clone.

• string target = 2

  (Required) Path to the location where the cloned table should be stored.

• oneof version_or_timestamp

  (Optional) Optional parameter to clone a previous state of the source table. The current state of the table is cloned when it is not specified.

  • int32 version = 3

    Clones the source table as of the provided version.

  • string timestamp = 4

    Clones the source table as of the provided timestamp.

• bool is_shallow = 5

  (Required) Performs a clone when true, this field should always be set to true.

• bool replace = 6

  (Required) Overwrites the target location when true.

• map<string, string> properties = 7

  (Required) User-defined table properties that override properties with the same key in the source table.

Command that turns a Parquet table into a Delta table. This needs to be a Relation as it returns the identifier of the resulting table. We cannot simply reuse the input identifier, as it could be a path-based identifier, and in that case we need to replace "parquet.`...`" with "delta.`...`".

Used in: DeltaRelation

• string identifier = 1

  (Required) Parquet table identifier formatted as "parquet.`path`"

• oneof partition_schema

  (Optional) Partitioning schema of the input table

  • string partition_schema_string = 2

    Hive DDL formatted string

  • spark.connect.DataType partition_schema_struct = 3

    Struct with names and types of partitioning columns

Command that creates or replace a Delta table (depending on the mode).

Used in: DeltaCommand

• CreateDeltaTable.Mode mode = 1

  (Required) Mode that determines what to do when a table with the given name or location already exists.

• optional string table_name = 2

  (Optional) Qualified name of the table.

• optional string location = 3

  (Optional) Path to the directory where the table date is stored.

• optional string comment = 4

  (Optional) Comment describing the table.

• repeated CreateDeltaTable.Column columns = 5

  (Optional) Columns in the schema of the table.

• repeated string partitioning_columns = 6

  (Optional) Columns used for partitioning the table.

• map<string, string> properties = 7

  (Optional) Properties of the table.

• repeated string clustering_columns = 8

  (Optional) Columns used for clustering the table.

Column in the schema of the table.

Used in: CreateDeltaTable

• string name = 1

  (Required) Name of the column.

• optional spark.connect.DataType data_type = 2

  (Required) Data type of the column.

• bool nullable = 3

  (Required) Whether the column is nullable.

• optional string generated_always_as = 4

  (Optional) SQL Expression that is used to generate the values in the column.

• optional string comment = 5

  (Optional) Comment to describe the column.

• optional Column.IdentityInfo identity_info = 6

  (Optional) Identity information for the column.

Used in: Column

• int64 start = 1

  (Required) The start value of the identity column.

• int64 step = 2

  (Required) The increment value of the identity column.

• bool allow_explicit_insert = 3

  (Required) Whether the identity column is BY DEFAULT (true) or ALWAYS (false).

Used in: CreateDeltaTable

• MODE_UNSPECIFIED = 0

• MODE_CREATE = 1

  Create the table if it does not exist, and throw an error otherwise.

• MODE_CREATE_IF_NOT_EXISTS = 2

  Create the table if it does not exist, and do nothing otherwise.

• MODE_REPLACE = 3

  Replace the table if it already exists, and throw an error otherwise.

• MODE_CREATE_OR_REPLACE = 4

  Create the table if it does not exist, and replace it otherwise.

Command that deletes data from the target table that matches the given condition. Needs to be a Relation, as it returns a row containing the execution metrics.

Used in: DeltaRelation

• optional spark.connect.Relation target = 1

  (Required) Target table to delete data from. Must either be a DeltaRelation containing a Scan or a SubqueryAlias with a DeltaRelation containing a Scan as its input.

• optional spark.connect.Expression condition = 2

  (Optional) Expression returning a boolean.

Message to hold all command extensions in Delta Connect.

• oneof command_type

  • CloneTable clone_table = 1

  • VacuumTable vacuum_table = 2

  • UpgradeTableProtocol upgrade_table_protocol = 3

  • Generate generate = 4

  • CreateDeltaTable create_delta_table = 5

  • AddFeatureSupport add_feature_support = 6

  • DropFeatureSupport drop_feature_support = 7

Message to hold all relation extensions in Delta Connect.

• oneof relation_type

  • Scan scan = 1

  • DescribeHistory describe_history = 2

  • DescribeDetail describe_detail = 3

  • ConvertToDelta convert_to_delta = 4

  • RestoreTable restore_table = 5

  • IsDeltaTable is_delta_table = 6

  • DeleteFromTable delete_from_table = 7

  • UpdateTable update_table = 8

  • MergeIntoTable merge_into_table = 9

  • OptimizeTable optimize_table = 10

Information required to access a Delta table either by name or by path.

Used in: AddFeatureSupport, CloneTable, DescribeDetail, DescribeHistory, DropFeatureSupport, Generate, OptimizeTable, RestoreTable, Scan, UpgradeTableProtocol, VacuumTable

• oneof access_type

  (Required)

  • DeltaTable.Path path = 1

  • string table_or_view_name = 2

Used in: DeltaTable

• string path = 1

  (Required) Path to the Delta table.

• map<string, string> hadoop_conf = 2

  (Optional) Hadoop configuration used to access the file system.

Relation containing the details of a Delta table such as the format, name, and size.

Used in: DeltaRelation

• optional DeltaTable table = 1

  (Required) The Delta table to describe the details of.

Relation containing information of the latest commits on a Delta table. The information is in reverse chronological order.

Used in: DeltaRelation

• optional DeltaTable table = 1

  (Required) The Delta table to read the history of.

Command to drop a supported feature from the table by modifying the protocol.

Used in: DeltaCommand

• optional DeltaTable table = 1

  (Required) The Delta table to drop the supported feature from.

• string feature_name = 2

  (Required) The name of the supported feature to drop.

• optional bool truncate_history = 3

  (optional) Whether to truncate history. When not specified, history is not truncated.

Command that generates manifest files for a given Delta table.

Used in: DeltaCommand

• optional DeltaTable table = 1

  (Required) The Delta table to generate the manifest files for.

• string mode = 2

  (Required) The type of manifest file to be generated.

Relation containing a single row containing a single boolean that indicates whether the provided path contains a Delta table.

Used in: DeltaRelation

• string path = 1

  (Required) The path to check.

Command that merges a source query/table into a Delta table, Needs to be a Relation, as it returns a row containing the execution metrics.

Used in: DeltaRelation

• optional spark.connect.Relation target = 1

  (Required) Target table to merge into.

• optional spark.connect.Relation source = 2

  (Required) Source data to merge from.

• optional spark.connect.Expression condition = 3

  (Required) Condition for a source row to match with a target row.

• repeated MergeIntoTable.Action matched_actions = 4

  (Optional) Actions to apply when a source row matches a target row.

• repeated MergeIntoTable.Action not_matched_actions = 5

  (Optional) Actions to apply when a source row does not match a target row.

• repeated MergeIntoTable.Action not_matched_by_source_actions = 6

  (Optional) Actions to apply when a target row does not match a source row.

• optional bool with_schema_evolution = 7

  (Optional) Whether Schema Evolution is enabled for this command.

Rule that specifies how the target table should be modified.

Used in: MergeIntoTable

• optional spark.connect.Expression condition = 1

  (Optional) Condition for the action to be applied.

• oneof action_type

  (Required)

  • Action.DeleteAction delete_action = 2

  • Action.UpdateAction update_action = 3

  • Action.UpdateStarAction update_star_action = 4

  • Action.InsertAction insert_action = 5

  • Action.InsertStarAction insert_star_action = 6

Action that deletes the target row.

Used in: Action

(message has no fields)

Action that inserts the source row into the target using a set of assignments.

Used in: Action

• repeated Assignment assignments = 1

  (Optional) Set of assignments to apply.

Action that inserts the source row into the target by setting all columns.

Used in: Action

(message has no fields)

Action that updates the target row using a set of assignments.

Used in: Action

• repeated Assignment assignments = 1

  (Optional) Set of assignments to apply.

Action that updates the target row by overwriting all columns.

Used in: Action

(message has no fields)

Command that optimizes the layout of a Delta table by either compacting small files or by ordering the data. Allows specifying partition filters to limit the scope of the data reorganization. Needs to be a Relation, as it returns a row containing the execution metrics.

Used in: DeltaRelation

• optional DeltaTable table = 1

  (Required) The Delta table to optimize.

• repeated string partition_filters = 2

  (Optional) Partition filters that limit the operation to the files in the matched partitions.

• repeated string zorder_columns = 3

  (Optional) Columns to z-order by. Compaction is performed when no z-order columns are provided.

Command that restores the DeltaTable to an older version of the table specified by either a version number or a timestamp. Needs to be a Relation, as it returns a row containing the execution metrics.

Used in: DeltaRelation

• optional DeltaTable table = 1

  (Required) The Delta table to restore to an earlier version.

• oneof version_or_timestamp

  (Required) Version to restore to.

  • int64 version = 2

    The version number to restore to.

  • string timestamp = 3

    The timestamp to restore to.

Relation that reads from a Delta table.

Used in: DeltaRelation

• optional DeltaTable table = 1

  (Required) The Delta table to scan.

Command that updates data in the target table using the given assignments for rows that matches the given condition. Needs to be a Relation, as it returns a row containing the execution metrics.

Used in: DeltaRelation

• optional spark.connect.Relation target = 1

  (Required) Target table to delete data from. Must either be a DeltaRelation containing a Scan or a SubqueryAlias with a DeltaRelation containing a Scan as its input.

• optional spark.connect.Expression condition = 2

  (Optional) Condition that determines which rows must be updated. Must be an expression returning a boolean.

• repeated Assignment assignments = 3

  (Optional) Set of assignments to apply to the rows matching the condition.

Command to updates the protocol version of the table so that new features can be used.

Used in: DeltaCommand

• optional DeltaTable table = 1

  (Required) The Delta table to upgrade the protocol of.

• int32 reader_version = 2

  (Required) The minimum required reader protocol version.

• int32 writer_version = 3

  (Required) The minimum required writer protocol version.

Command that deletes files and directories in the table that are not needed by the table for maintaining older versions up to the given retention threshold.

Used in: DeltaCommand

• optional DeltaTable table = 1

  (Required) The Delta table to vacuum.

• optional double retention_hours = 2

  (Optional) Number of hours retain history for. If not specified, then the default retention period will be used.