package google.events.cloud.datastream.v1

Get desktop application:
View/edit binary Protocol Buffers messages

AVRO file format configuration.

Used in: GcsDestinationConfig

(message has no fields)

Used in: DestinationConfig

• oneof dataset_config

  Target dataset(s) configuration.

  • BigQueryDestinationConfig.SingleTargetDataset single_target_dataset = 201

    Single destination dataset.

  • BigQueryDestinationConfig.SourceHierarchyDatasets source_hierarchy_datasets = 202

    Source hierarchy datasets.

• optional protobuf.Duration data_freshness = 300

  The guaranteed data freshness (in seconds) when querying tables created by the stream. Editing this field will only affect new tables created in the future, but existing tables will not be impacted. Lower values mean that queries will return fresher data, but may result in higher cost.

A single target dataset to which all data will be streamed.

Used in: BigQueryDestinationConfig

• string dataset_id = 1

  The dataset ID of the target dataset.

Destination datasets are created so that hierarchy of the destination data objects matches the source hierarchy.

Used in: BigQueryDestinationConfig

• optional SourceHierarchyDatasets.DatasetTemplate dataset_template = 2

Dataset template used for dynamic dataset creation.

Used in: SourceHierarchyDatasets

• string location = 1

  Required. The geographic location where the dataset should reside. See https://cloud.google.com/bigquery/docs/locations for supported locations.

• string dataset_id_prefix = 2

  If supplied, every created dataset will have its name prefixed by the provided value. The prefix and name will be separated by an underscore. i.e. <prefix>_<dataset_name>.

• string kms_key_name = 3

  Describes the Cloud KMS encryption key that will be used to protect destination BigQuery table. The BigQuery Service Account associated with your project requires access to this encryption key. i.e. projects/{project}/locations/{location}/keyRings/{key_ring}/cryptoKeys/{cryptoKey}. See https://cloud.google.com/bigquery/docs/customer-managed-encryption for more information.

BigQuery warehouse profile.

Used in: ConnectionProfile

(message has no fields)

A set of reusable connection configurations to be used as a source or destination for a stream.

Used in: ConnectionProfileEventData

• string name = 1

  Output only. The resource's name.

• optional protobuf.Timestamp create_time = 2

  Output only. The create time of the resource.

• optional protobuf.Timestamp update_time = 3

  Output only. The update time of the resource.

• map<string, string> labels = 4

  Labels.

• string display_name = 5

  Required. Display name.

• oneof profile

  Connection configuration for the ConnectionProfile.

  • OracleProfile oracle_profile = 100

    Oracle ConnectionProfile configuration.

  • GcsProfile gcs_profile = 101

    Cloud Storage ConnectionProfile configuration.

  • MysqlProfile mysql_profile = 102

    MySQL ConnectionProfile configuration.

  • BigQueryProfile bigquery_profile = 103

    BigQuery Connection Profile configuration.

  • PostgresqlProfile postgresql_profile = 104

    PostgreSQL Connection Profile configuration.

• oneof connectivity

  Connectivity options used to establish a connection to the profile.

  • StaticServiceIpConnectivity static_service_ip_connectivity = 200

    Static Service IP connectivity.

  • ForwardSshTunnelConnectivity forward_ssh_connectivity = 201

    Forward SSH tunnel connectivity.

  • PrivateConnectivity private_connectivity = 202

    Private connectivity.

The CloudEvent raised when a ConnectionProfile is created.

• optional ConnectionProfileEventData data = 1

  The data associated with the event.

The CloudEvent raised when a ConnectionProfile is deleted.

• optional ConnectionProfileEventData data = 1

  The data associated with the event.

The data within all ConnectionProfile events.

Used in: ConnectionProfileCreatedEvent, ConnectionProfileDeletedEvent, ConnectionProfileUpdatedEvent

• optional ConnectionProfile payload = 1

  Optional. The ConnectionProfile event payload. Unset for deletion events.

The CloudEvent raised when a ConnectionProfile is updated.

• optional ConnectionProfileEventData data = 1

  The data associated with the event.

The configuration of the stream destination.

Used in: Stream

• string destination_connection_profile = 1

  Required. Destination connection profile resource. Format: `projects/{project}/locations/{location}/connectionProfiles/{name}`

• oneof destination_stream_config

  Stream configuration that is specific to the data destination type.

  • GcsDestinationConfig gcs_destination_config = 100

    A configuration for how data should be loaded to Cloud Storage.

  • BigQueryDestinationConfig bigquery_destination_config = 101

    BigQuery destination configuration.

Represent a user-facing Error.

Used in: PrivateConnection, Stream

• string reason = 1

  A title that explains the reason for the error.

• string error_uuid = 2

  A unique identifier for this specific error, allowing it to be traced throughout the system in logs and API responses.

• string message = 3

  A message containing more information about the error that occurred.

• optional protobuf.Timestamp error_time = 4

  The time when the error occurred.

• map<string, string> details = 5

  Additional information about the error.

Forward SSH Tunnel connectivity.

Used in: ConnectionProfile

• string hostname = 1

  Required. Hostname for the SSH tunnel.

• string username = 2

  Required. Username for the SSH tunnel.

• int32 port = 3

  Port for the SSH tunnel, default value is 22.

Google Cloud Storage destination configuration

Used in: DestinationConfig

• string path = 1

  Path inside the Cloud Storage bucket to write data to.

• int32 file_rotation_mb = 2

  The maximum file size to be saved in the bucket.

• optional protobuf.Duration file_rotation_interval = 3

  The maximum duration for which new events are added before a file is closed and a new file is created.

• oneof file_format

  File Format that the data should be written in.

  • AvroFileFormat avro_file_format = 100

    AVRO file format configuration.

  • JsonFileFormat json_file_format = 101

    JSON file format configuration.

Cloud Storage bucket profile.

Used in: ConnectionProfile

• string bucket = 1

  Required. The Cloud Storage bucket name.

• string root_path = 2

  The root path inside the Cloud Storage bucket.

JSON file format configuration.

Used in: GcsDestinationConfig

• JsonFileFormat.SchemaFileFormat schema_file_format = 1

  The schema file format along JSON data files.

• JsonFileFormat.JsonCompression compression = 2

  Compression of the loaded JSON file.

Json file compression.

Used in: JsonFileFormat

• JSON_COMPRESSION_UNSPECIFIED = 0

  Unspecified json file compression.

• NO_COMPRESSION = 1

  Do not compress JSON file.

• GZIP = 2

  Gzip compression.

Schema file format.

Used in: JsonFileFormat

• SCHEMA_FILE_FORMAT_UNSPECIFIED = 0

  Unspecified schema file format.

• NO_SCHEMA_FILE = 1

  Do not attach schema file.

• AVRO_SCHEMA_FILE = 2

  Avro schema format.

MySQL Column.

Used in: MysqlTable

• string column = 1

  Column name.

• string data_type = 2

  The MySQL data type. Full data types list can be found here: https://dev.mysql.com/doc/refman/8.0/en/data-types.html

• int32 length = 3

  Column length.

• string collation = 4

  Column collation.

• bool primary_key = 5

  Whether or not the column represents a primary key.

• bool nullable = 6

  Whether or not the column can accept a null value.

• int32 ordinal_position = 7

  The ordinal position of the column in the table.

MySQL database.

Used in: MysqlRdbms

• string database = 1

  Database name.

• repeated MysqlTable mysql_tables = 2

  Tables in the database.

MySQL database profile.

Used in: ConnectionProfile

• string hostname = 1

  Required. Hostname for the MySQL connection.

• int32 port = 2

  Port for the MySQL connection, default value is 3306.

• string username = 3

  Required. Username for the MySQL connection.

• optional MysqlSslConfig ssl_config = 5

  SSL configuration for the MySQL connection.

MySQL database structure

Used in: MysqlSourceConfig, Stream.BackfillAllStrategy

• repeated MysqlDatabase mysql_databases = 1

  Mysql databases on the server

MySQL source configuration

Used in: SourceConfig

• optional MysqlRdbms include_objects = 1

  MySQL objects to retrieve from the source.

• optional MysqlRdbms exclude_objects = 2

  MySQL objects to exclude from the stream.

• int32 max_concurrent_cdc_tasks = 3

  Maximum number of concurrent CDC tasks. The number should be non negative. If not set (or set to 0), the system's default value will be used.

MySQL SSL configuration information.

Used in: MysqlProfile

• bool client_key_set = 2

  Output only. Indicates whether the client_key field is set.

• bool client_certificate_set = 4

  Output only. Indicates whether the client_certificate field is set.

• bool ca_certificate_set = 6

  Output only. Indicates whether the ca_certificate field is set.

MySQL table.

Used in: MysqlDatabase

• string table = 1

  Table name.

• repeated MysqlColumn mysql_columns = 2

  MySQL columns in the database. When unspecified as part of include/exclude objects, includes/excludes everything.

Oracle Column.

Used in: OracleTable

• string column = 1

  Column name.

• string data_type = 2

  The Oracle data type.

• int32 length = 3

  Column length.

• int32 precision = 4

  Column precision.

• int32 scale = 5

  Column scale.

• string encoding = 6

  Column encoding.

• bool primary_key = 7

  Whether or not the column represents a primary key.

• bool nullable = 8

  Whether or not the column can accept a null value.

• int32 ordinal_position = 9

  The ordinal position of the column in the table.

Oracle database profile.

Used in: ConnectionProfile

• string hostname = 1

  Required. Hostname for the Oracle connection.

• int32 port = 2

  Port for the Oracle connection, default value is 1521.

• string username = 3

  Required. Username for the Oracle connection.

• string database_service = 5

  Required. Database for the Oracle connection.

• map<string, string> connection_attributes = 6

  Connection string attributes

Oracle database structure.

Used in: OracleSourceConfig, Stream.BackfillAllStrategy

• repeated OracleSchema oracle_schemas = 1

  Oracle schemas/databases in the database server.

Oracle schema.

Used in: OracleRdbms

• string schema = 1

  Schema name.

• repeated OracleTable oracle_tables = 2

  Tables in the schema.

Oracle data source configuration

Used in: SourceConfig

• optional OracleRdbms include_objects = 1

  Oracle objects to include in the stream.

• optional OracleRdbms exclude_objects = 2

  Oracle objects to exclude from the stream.

• int32 max_concurrent_cdc_tasks = 3

  Maximum number of concurrent CDC tasks. The number should be non negative. If not set (or set to 0), the system's default value will be used.

• oneof large_objects_handling

  The configuration for handle Oracle large objects.

  • OracleSourceConfig.DropLargeObjects drop_large_objects = 100

    Drop large object values.

  • OracleSourceConfig.StreamLargeObjects stream_large_objects = 102

    Stream large object values. NOTE: This feature is currently experimental.

Configuration to drop large object values.

Used in: OracleSourceConfig

(message has no fields)

Configuration to stream large object values.

Used in: OracleSourceConfig

(message has no fields)

Oracle table.

Used in: OracleSchema

• string table = 1

  Table name.

• repeated OracleColumn oracle_columns = 2

  Oracle columns in the schema. When unspecified as part of include/exclude objects, includes/excludes everything.

PostgreSQL Column.

Used in: PostgresqlTable

• string column = 1

  Column name.

• string data_type = 2

  The PostgreSQL data type.

• int32 length = 3

  Column length.

• int32 precision = 4

  Column precision.

• int32 scale = 5

  Column scale.

• bool primary_key = 7

  Whether or not the column represents a primary key.

• bool nullable = 8

  Whether or not the column can accept a null value.

• int32 ordinal_position = 9

  The ordinal position of the column in the table.

PostgreSQL database profile.

Used in: ConnectionProfile

• string hostname = 1

  Required. Hostname for the PostgreSQL connection.

• int32 port = 2

  Port for the PostgreSQL connection, default value is 5432.

• string username = 3

  Required. Username for the PostgreSQL connection.

• string database = 5

  Required. Database for the PostgreSQL connection.

PostgreSQL database structure.

Used in: PostgresqlSourceConfig, Stream.BackfillAllStrategy

• repeated PostgresqlSchema postgresql_schemas = 1

  PostgreSQL schemas in the database server.

PostgreSQL schema.

Used in: PostgresqlRdbms

• string schema = 1

  Schema name.

• repeated PostgresqlTable postgresql_tables = 2

  Tables in the schema.

PostgreSQL data source configuration

Used in: SourceConfig

• optional PostgresqlRdbms include_objects = 1

  PostgreSQL objects to include in the stream.

• optional PostgresqlRdbms exclude_objects = 2

  PostgreSQL objects to exclude from the stream.

• string replication_slot = 3

  Required. Immutable. The name of the logical replication slot that's configured with the pgoutput plugin.

• string publication = 4

  Required. The name of the publication that includes the set of all tables that are defined in the stream's include_objects.

PostgreSQL table.

Used in: PostgresqlSchema

• string table = 1

  Table name.

• repeated PostgresqlColumn postgresql_columns = 2

  PostgreSQL columns in the schema. When unspecified as part of include/exclude objects, includes/excludes everything.

The PrivateConnection resource is used to establish private connectivity between Datastream and a customer's network.

Used in: PrivateConnectionEventData

• string name = 1

  Output only. The resource's name.

• optional protobuf.Timestamp create_time = 2

  Output only. The create time of the resource.

• optional protobuf.Timestamp update_time = 3

  Output only. The update time of the resource.

• map<string, string> labels = 4

  Labels.

• string display_name = 5

  Required. Display name.

• PrivateConnection.State state = 6

  Output only. The state of the Private Connection.

• optional Error error = 7

  Output only. In case of error, the details of the error in a user-friendly format.

• optional VpcPeeringConfig vpc_peering_config = 100

  VPC Peering Config.

Private Connection state.

Used in: PrivateConnection

• STATE_UNSPECIFIED = 0

  Unspecified state.

• CREATING = 1

  The private connection is in creation state - creating resources.

• CREATED = 2

  The private connection has been created with all of its resources.

• FAILED = 3

  The private connection creation has failed.

• DELETING = 4

  The private connection is being deleted.

• FAILED_TO_DELETE = 5

  Delete request has failed, resource is in invalid state.

The CloudEvent raised when a PrivateConnection is created.

• optional PrivateConnectionEventData data = 1

  The data associated with the event.

The CloudEvent raised when a PrivateConnection is deleted.

• optional PrivateConnectionEventData data = 1

  The data associated with the event.

The data within all PrivateConnection events.

Used in: PrivateConnectionCreatedEvent, PrivateConnectionDeletedEvent

• optional PrivateConnection payload = 1

  Optional. The PrivateConnection event payload. Unset for deletion events.

Private Connectivity

Used in: ConnectionProfile

• string private_connection = 1

  Required. A reference to a private connection resource. Format: `projects/{project}/locations/{location}/privateConnections/{name}`

The route resource is the child of the private connection resource, used for defining a route for a private connection.

Used in: RouteEventData

• string name = 1

  Output only. The resource's name.

• optional protobuf.Timestamp create_time = 2

  Output only. The create time of the resource.

• optional protobuf.Timestamp update_time = 3

  Output only. The update time of the resource.

• map<string, string> labels = 4

  Labels.

• string display_name = 5

  Required. Display name.

• string destination_address = 6

  Required. Destination address for connection

• int32 destination_port = 7

  Destination port for connection

The CloudEvent raised when a Route is created.

• optional RouteEventData data = 1

  The data associated with the event.

The CloudEvent raised when a Route is deleted.

• optional RouteEventData data = 1

  The data associated with the event.

The data within all Route events.

Used in: RouteCreatedEvent, RouteDeletedEvent

• optional Route payload = 1

  Optional. The Route event payload. Unset for deletion events.

The configuration of the stream source.

Used in: Stream

• string source_connection_profile = 1

  Required. Source connection profile resoource. Format: `projects/{project}/locations/{location}/connectionProfiles/{name}`

• oneof source_stream_config

  Stream configuration that is specific to the data source type.

  • OracleSourceConfig oracle_source_config = 100

    Oracle data source configuration.

  • MysqlSourceConfig mysql_source_config = 101

    MySQL data source configuration.

  • PostgresqlSourceConfig postgresql_source_config = 102

    PostgreSQL data source configuration.

Static IP address connectivity.

Used in: ConnectionProfile

(message has no fields)

A resource representing streaming data from a source to a destination.

Used in: StreamEventData

• string name = 1

  Output only. The stream's name.

• optional protobuf.Timestamp create_time = 2

  Output only. The creation time of the stream.

• optional protobuf.Timestamp update_time = 3

  Output only. The last update time of the stream.

• map<string, string> labels = 4

  Labels.

• string display_name = 5

  Required. Display name.

• optional SourceConfig source_config = 6

  Required. Source connection profile configuration.

• optional DestinationConfig destination_config = 7

  Required. Destination connection profile configuration.

• Stream.State state = 8

  The state of the stream.

• oneof backfill_strategy

  Stream backfill strategy.

  • Stream.BackfillAllStrategy backfill_all = 101

    Automatically backfill objects included in the stream source configuration. Specific objects can be excluded.

  • Stream.BackfillNoneStrategy backfill_none = 102

    Do not automatically backfill any objects.

• repeated Error errors = 9

  Output only. Errors on the Stream.

• optional string customer_managed_encryption_key = 10

  Immutable. A reference to a KMS encryption key. If provided, it will be used to encrypt the data. If left blank, data will be encrypted using an internal Stream-specific encryption key provisioned through KMS.

Backfill strategy to automatically backfill the Stream's objects. Specific objects can be excluded.

Used in: Stream

• oneof excluded_objects

  List of objects to exclude.

  • OracleRdbms oracle_excluded_objects = 1

    Oracle data source objects to avoid backfilling.

  • MysqlRdbms mysql_excluded_objects = 2

    MySQL data source objects to avoid backfilling.

  • PostgresqlRdbms postgresql_excluded_objects = 3

    PostgreSQL data source objects to avoid backfilling.

Backfill strategy to disable automatic backfill for the Stream's objects.

Used in: Stream

(message has no fields)

Stream state.

Used in: Stream

• STATE_UNSPECIFIED = 0

  Unspecified stream state.

• NOT_STARTED = 1

  The stream has been created but has not yet started streaming data.

• RUNNING = 2

  The stream is running.

• PAUSED = 3

  The stream is paused.

• MAINTENANCE = 4

  The stream is in maintenance mode. Updates are rejected on the resource in this state.

• FAILED = 5

  The stream is experiencing an error that is preventing data from being streamed.

• FAILED_PERMANENTLY = 6

  The stream has experienced a terminal failure.

• STARTING = 7

  The stream is starting, but not yet running.

• DRAINING = 8

  The Stream is no longer reading new events, but still writing events in the buffer.

The CloudEvent raised when a Stream is created.

• optional StreamEventData data = 1

  The data associated with the event.

The CloudEvent raised when a Stream is deleted.

• optional StreamEventData data = 1

  The data associated with the event.

The data within all Stream events.

Used in: StreamCreatedEvent, StreamDeletedEvent, StreamUpdatedEvent

• optional Stream payload = 1

  Optional. The Stream event payload. Unset for deletion events.

The CloudEvent raised when a Stream is updated.

• optional StreamEventData data = 1

  The data associated with the event.

The VPC Peering configuration is used to create VPC peering between Datastream and the consumer's VPC.

Used in: PrivateConnection

• string vpc = 1

  Required. Fully qualified name of the VPC that Datastream will peer to. Format: `projects/{project}/global/{networks}/{name}`

• string subnet = 2

  Required. A free subnet for peering. (CIDR of /29)