package google.cloud.bigquery.datatransfer.v1
Get desktop application:
View/edit binary Protocol Buffers messages
The Google BigQuery Data Transfer API allows BigQuery users to configure transfer of their data from other Google Products into BigQuery. This service exposes methods that should be used by data source backend.
rpc UpdateTransferRun (, )
datasource.proto:45Update a transfer run. If successful, resets data_source.update_deadline_seconds timer.
message
datasource.proto:345A request to update a transfer run.
optional transfer_run = 1
Run name must be set and correspond to an already existing run. Only state, error_status, and data_version fields will be updated. All other fields will be ignored.
optional update_mask = 2
Required list of fields to be updated in this request.
Log messages for a transfer run. If successful (at least 1 message), resets data_source.update_deadline_seconds timer.
message
datasource.proto:356A request to add transfer status messages to the run.
string name = 1
Name of the resource in the form: "projects/{project_id}/locations/{location_id}/transferConfigs/{config_id}/runs/{run_id}"
repeated transfer_messages = 2
Messages to append.
rpc StartBigQueryJobs (, )
datasource.proto:66Notify the Data Transfer Service that data is ready for loading. The Data Transfer Service will start and monitor multiple BigQuery Load jobs for a transfer run. Monitored jobs will be automatically retried and produce log messages when starting and finishing a job. Can be called multiple times for the same transfer run.
message
datasource.proto:366A request to start and monitor a BigQuery load job.
string name = 1
Name of the resource in the form: "projects/{project_id}/locations/{location_id}/transferConfigs/{config_id}/runs/{run_id}"
repeated imported_data = 2
Import jobs which should be started and monitored.
bytes user_credentials = 3
User credentials which should be used to start/monitor BigQuery jobs. If not specified, then jobs are started using data source service account credentials. This may be OAuth token or JWT token.
int32 max_parallelism = 8
The number of BQ Jobs that can run in parallel.
rpc FinishRun (, )
datasource.proto:78Notify the Data Transfer Service that the data source is done processing the run. No more status updates or requests to start/monitor jobs will be accepted. The run will be finalized by the Data Transfer Service when all monitored jobs are completed. Does not need to be called if the run is set to FAILED.
message
datasource.proto:385A request to finish a run.
string name = 1
Name of the resource in the form: "projects/{project_id}/locations/{location_id}/transferConfigs/{config_id}/runs/{run_id}"
Creates a data source definition. Calling this method will automatically use your credentials to create the following Google Cloud resources in YOUR Google Cloud project. 1. OAuth client 2. Pub/Sub Topics and Subscriptions in each supported_location_ids. e.g., projects/{project_id}/{topics|subscriptions}/bigquerydatatransfer.{data_source_id}.{location_id}.run The field data_source.client_id should be left empty in the input request, as the API will create a new OAuth client on behalf of the caller. On the other hand data_source.scopes usually need to be set when there are OAuth scopes that need to be granted by end users. 3. We need a longer deadline due to the 60 seconds SLO from Pub/Sub admin Operations. This also applies to update and delete data source definition.
message
datasource.proto:392Represents the request of the CreateDataSourceDefinition method.
string parent = 1
The BigQuery project id for which data source definition is associated. Must be in the form: `projects/{project_id}/locations/{location_id}`
optional data_source_definition = 2
Data source definition.
Updates an existing data source definition. If changing supported_location_ids, triggers same effects as mentioned in "Create a data source definition."
message
datasource.proto:402Represents the request of the UpdateDataSourceDefinition method.
optional data_source_definition = 1
Data source definition.
optional update_mask = 2
Update field mask.
Deletes a data source definition, all of the transfer configs associated with this data source definition (if any) must be deleted first by the user in ALL regions, in order to delete the data source definition. This method is primarily meant for deleting data sources created during testing stage. If the data source is referenced by transfer configs in the region specified in the request URL, the method will fail immediately. If in the current region (e.g., US) it's not used by any transfer configs, but in another region (e.g., EU) it is, then although the method will succeed in region US, but it will fail when the deletion operation is replicated to region EU. And eventually, the system will replicate the data source definition back from EU to US, in order to bring all regions to consistency. The final effect is that the data source appears to be 'undeleted' in the US region.
message
datasource.proto:413Represents the request of the DeleteDataSourceDefinition method. All transfer configs associated with the data source must be deleted first, before the data source can be deleted.
string name = 1
The field will contain name of the resource requested, for example: `projects/{project_id}/locations/{location_id}/dataSourceDefinitions/{data_source_id}`
Retrieves an existing data source definition.
message
datasource.proto:420Represents the request of the GetDataSourceDefinition method.
string name = 1
The field will contain name of the resource requested.
Lists supported data source definitions.
message
datasource.proto:441Represents the request of the ListDataSourceDefinitions method.
string parent = 1
The BigQuery project id for which data sources should be returned. Must be in the form: `projects/{project_id}/locations/{location_id}`
string page_token = 2
Pagination token, which can be used to request a specific page of `ListDataSourceDefinitionsRequest` list results. For multiple-page results, `ListDataSourceDefinitionsResponse` outputs a `next_page` token, which can be used as the `page_token` value to request the next page of the list results.
int32 page_size = 3
Page size. The default page size is the maximum value of 1000 results.
message
datasource.proto:458Returns a list of supported data source definitions.
repeated data_source_definitions = 1
List of supported data source definitions.
string next_page_token = 2
Output only. The next-pagination token. For multiple-page list results, this token can be used as the `ListDataSourceDefinitionsRequest.page_token` to request the next page of the list results.
The Google BigQuery Data Transfer Service API enables BigQuery users to configure the transfer of their data from other Google Products into BigQuery. This service contains methods that are end user exposed. It backs up the frontend.
rpc GetDataSource (, )
datatransfer.proto:48Retrieves a supported data source and returns its settings, which can be used for UI rendering.
message
datatransfer.proto:418A request to get data source info.
string name = 1
Required. The field will contain name of the resource requested, for example: `projects/{project_id}/dataSources/{data_source_id}`
rpc ListDataSources (, )
datatransfer.proto:60Lists supported data sources and returns their settings, which can be used for UI rendering.
message
datatransfer.proto:430Request to list supported data sources and their data transfer settings.
string parent = 1
Required. The BigQuery project id for which data sources should be returned. Must be in the form: `projects/{project_id}`
string page_token = 3
Pagination token, which can be used to request a specific page of `ListDataSourcesRequest` list results. For multiple-page results, `ListDataSourcesResponse` outputs a `next_page` token, which can be used as the `page_token` value to request the next page of list results.
int32 page_size = 4
Page size. The default page size is the maximum value of 1000 results.
message
datatransfer.proto:452Returns list of supported data sources and their metadata.
repeated data_sources = 1
List of supported data sources and their transfer settings.
string next_page_token = 2
Output only. The next-pagination token. For multiple-page list results, this token can be used as the `ListDataSourcesRequest.page_token` to request the next page of list results.
Creates a new data transfer configuration.
message
datatransfer.proto:469A request to create a data transfer configuration. If new credentials are needed for this transfer configuration, an authorization code must be provided. If an authorization code is provided, the transfer configuration will be associated with the user id corresponding to the authorization code. Otherwise, the transfer configuration will be associated with the calling user.
string parent = 1
Required. The BigQuery project id where the transfer configuration should be created. Must be in the format projects/{project_id}/locations/{location_id} If specified location and location of the destination bigquery dataset do not match - the request will fail.
optional transfer_config = 2
Required. Data transfer configuration to create.
string authorization_code = 3
Optional OAuth2 authorization code to use with this transfer configuration. This is required if new credentials are needed, as indicated by `CheckValidCreds`. In order to obtain authorization_code, please make a request to https://www.gstatic.com/bigquerydatatransfer/oauthz/auth?client_id=<datatransferapiclientid>&scope=<data_source_scopes>&redirect_uri=<redirect_uri> * client_id should be OAuth client_id of BigQuery DTS API for the given data source returned by ListDataSources method. * data_source_scopes are the scopes returned by ListDataSources method. * redirect_uri is an optional parameter. If not specified, then authorization code is posted to the opener of authorization flow window. Otherwise it will be sent to the redirect uri. A special value of urn:ietf:wg:oauth:2.0:oob means that authorization code should be returned in the title bar of the browser, with the page text prompting the user to copy the code and paste it in the application.
string version_info = 5
Optional version info. If users want to find a very recent access token, that is, immediately after approving access, users have to set the version_info claim in the token request. To obtain the version_info, users must use the "none+gsession" response type. which be return a version_info back in the authorization response which be be put in a JWT claim in the token request.
Updates a data transfer configuration. All fields must be set, even if they are not updated.
message
datatransfer.proto:513A request to update a transfer configuration. To update the user id of the transfer configuration, an authorization code needs to be provided.
optional transfer_config = 1
Required. Data transfer configuration to create.
string authorization_code = 3
Optional OAuth2 authorization code to use with this transfer configuration. If it is provided, the transfer configuration will be associated with the authorizing user. In order to obtain authorization_code, please make a request to https://www.gstatic.com/bigquerydatatransfer/oauthz/auth?client_id=<datatransferapiclientid>&scope=<data_source_scopes>&redirect_uri=<redirect_uri> * client_id should be OAuth client_id of BigQuery DTS API for the given data source returned by ListDataSources method. * data_source_scopes are the scopes returned by ListDataSources method. * redirect_uri is an optional parameter. If not specified, then authorization code is posted to the opener of authorization flow window. Otherwise it will be sent to the redirect uri. A special value of urn:ietf:wg:oauth:2.0:oob means that authorization code should be returned in the title bar of the browser, with the page text prompting the user to copy the code and paste it in the application.
optional update_mask = 4
Required. Required list of fields to be updated in this request.
string version_info = 5
Optional version info. If users want to find a very recent access token, that is, immediately after approving access, users have to set the version_info claim in the token request. To obtain the version_info, users must use the "none+gsession" response type. which be return a version_info back in the authorization response which be be put in a JWT claim in the token request.
Deletes a data transfer configuration, including any associated transfer runs and logs.
message
datatransfer.proto:561A request to delete data transfer information. All associated transfer runs and log messages will be deleted as well.
string name = 1
Required. The field will contain name of the resource requested, for example: `projects/{project_id}/transferConfigs/{config_id}`
Returns information about a data transfer config.
message
datatransfer.proto:548A request to get data transfer information.
string name = 1
Required. The field will contain name of the resource requested, for example: `projects/{project_id}/transferConfigs/{config_id}`
Returns information about all data transfers in the project.
message
datatransfer.proto:597A request to list data transfers configured for a BigQuery project.
string parent = 1
Required. The BigQuery project id for which data sources should be returned: `projects/{project_id}`.
repeated string data_source_ids = 2
When specified, only configurations of requested data sources are returned.
string page_token = 3
Pagination token, which can be used to request a specific page of `ListTransfersRequest` list results. For multiple-page results, `ListTransfersResponse` outputs a `next_page` token, which can be used as the `page_token` value to request the next page of list results.
int32 page_size = 4
Page size. The default page size is the maximum value of 1000 results.
message
datatransfer.proto:622The returned list of pipelines in the project.
repeated transfer_configs = 1
Output only. The stored pipeline transfer configurations.
string next_page_token = 2
Output only. The next-pagination token. For multiple-page list results, this token can be used as the `ListTransferConfigsRequest.page_token` to request the next page of list results.
Creates transfer runs for a time range [start_time, end_time]. For each date - or whatever granularity the data source supports - in the range, one transfer run is created. Note that runs are created per UTC time in the time range. DEPRECATED: use StartManualTransferRuns instead.
message
datatransfer.proto:747A request to schedule transfer runs for a time range.
string parent = 1
Required. Transfer configuration name in the form: `projects/{project_id}/transferConfigs/{config_id}`.
optional start_time = 2
Required. Start time of the range of transfer runs. For example, `"2017-05-25T00:00:00+00:00"`.
optional end_time = 3
Required. End time of the range of transfer runs. For example, `"2017-05-30T00:00:00+00:00"`.
message
datatransfer.proto:767A response to schedule transfer runs for a time range.
repeated runs = 1
The transfer runs that were scheduled.
Start manual transfer runs to be executed now with schedule_time equal to current time. The transfer runs can be created for a time range where the run_time is between start_time (inclusive) and end_time (exclusive), or for a specific run_time.
message
datatransfer.proto:773A request to start manual transfer runs.
string parent = 1
Transfer configuration name in the form: `projects/{project_id}/transferConfigs/{config_id}`.
oneof time
The requested time specification - this can be a time range or a specific run_time.
requested_time_range = 3
Time range for the transfer runs that should be started.
requested_run_time = 4
Specific run_time for a transfer run to be started. The requested_run_time must not be in the future.
message
datatransfer.proto:809A response to start manual transfer runs.
repeated runs = 1
The transfer runs that were created.
rpc GetTransferRun (, )
datatransfer.proto:165Returns information about the particular transfer run.
message
datatransfer.proto:573A request to get data transfer run information.
string name = 1
Required. The field will contain name of the resource requested, for example: `projects/{project_id}/transferConfigs/{config_id}/runs/{run_id}`
Deletes the specified transfer run.
message
datatransfer.proto:585A request to delete data transfer run information.
string name = 1
Required. The field will contain name of the resource requested, for example: `projects/{project_id}/transferConfigs/{config_id}/runs/{run_id}`
Returns information about running and completed jobs.
message
datatransfer.proto:636A request to list data transfer runs. UI can use this method to show/filter specific data transfer runs. The data source can use this method to request all scheduled transfer runs.
string parent = 1
Required. Name of transfer configuration for which transfer runs should be retrieved. Format of transfer configuration resource name is: `projects/{project_id}/transferConfigs/{config_id}`.
repeated TransferState states = 2
When specified, only transfer runs with requested states are returned.
string page_token = 3
Pagination token, which can be used to request a specific page of `ListTransferRunsRequest` list results. For multiple-page results, `ListTransferRunsResponse` outputs a `next_page` token, which can be used as the `page_token` value to request the next page of list results.
int32 page_size = 4
Page size. The default page size is the maximum value of 1000 results.
ListTransferRunsRequest.RunAttempt run_attempt = 5
Indicates how run attempts are to be pulled.
message
datatransfer.proto:674The returned list of pipelines in the project.
repeated transfer_runs = 1
Output only. The stored pipeline transfer runs.
string next_page_token = 2
Output only. The next-pagination token. For multiple-page list results, this token can be used as the `ListTransferRunsRequest.page_token` to request the next page of list results.
Returns user facing log messages for the data transfer run.
message
datatransfer.proto:686A request to get user facing log messages associated with data transfer run.
string parent = 1
Required. Transfer run name in the form: `projects/{project_id}/transferConfigs/{config_Id}/runs/{run_id}`.
string page_token = 4
Pagination token, which can be used to request a specific page of `ListTransferLogsRequest` list results. For multiple-page results, `ListTransferLogsResponse` outputs a `next_page` token, which can be used as the `page_token` value to request the next page of list results.
int32 page_size = 5
Page size. The default page size is the maximum value of 1000 results.
repeated TransferMessage.MessageSeverity message_types = 6
Message types to return. If not populated - INFO, WARNING and ERROR messages are returned.
message
datatransfer.proto:712The returned list transfer run messages.
repeated transfer_messages = 1
Output only. The stored pipeline transfer messages.
string next_page_token = 2
Output only. The next-pagination token. For multiple-page list results, this token can be used as the `GetTransferRunLogRequest.page_token` to request the next page of list results.
Returns true if valid credentials exist for the given data source and requesting user. Some data sources doesn't support service account, so we need to talk to them on behalf of the end user. This API just checks whether we have OAuth token for the particular user, which is a pre-requisite before user can create a transfer config.
message
datatransfer.proto:729A request to determine whether the user has valid credentials. This method is used to limit the number of OAuth popups in the user interface. The user id is inferred from the API call context. If the data source has the Google+ authorization type, this method returns false, as it cannot be determined whether the credentials are already valid merely based on the user id.
string name = 1
Required. The data source in the form: `projects/{project_id}/dataSources/{data_source_id}`
message
datatransfer.proto:741A response indicating whether the credentials exist and are valid.
bool has_valid_creds = 1
If set to `true`, the credentials exist and are valid.
message
datatransfer.proto:312Represents data source metadata. Metadata is sufficient to render UI and request proper OAuth tokens.
Used as response type in: DataTransferService.GetDataSource
Used as field type in: ,
string name = 1
Output only. Data source resource name.
string data_source_id = 2
Data source id.
string display_name = 3
User friendly data source name.
string description = 4
User friendly data source description string.
string client_id = 5
Data source client id which should be used to receive refresh token.
repeated string scopes = 6
Api auth scopes for which refresh token needs to be obtained. These are scopes needed by a data source to prepare data and ingest them into BigQuery, e.g., https://www.googleapis.com/auth/bigquery
TransferType transfer_type = 7
Deprecated. This field has no effect.
bool supports_multiple_transfers = 8
Deprecated. This field has no effect.
int32 update_deadline_seconds = 9
The number of seconds to wait for an update from the data source before the Data Transfer Service marks the transfer as FAILED.
string default_schedule = 10
Default data transfer schedule. Examples of valid schedules include: `1st,3rd monday of month 15:30`, `every wed,fri of jan,jun 13:15`, and `first sunday of quarter 00:00`.
bool supports_custom_schedule = 11
Specifies whether the data source supports a user defined schedule, or operates on the default schedule. When set to `true`, user can override default schedule.
repeated parameters = 12
Data source parameters.
string help_url = 13
Url for the help document for this data source.
DataSource.AuthorizationType authorization_type = 14
Indicates the type of authorization.
DataSource.DataRefreshType data_refresh_type = 15
Specifies whether the data source supports automatic data refresh for the past few days, and how it's supported. For some data sources, data might not be complete until a few days later, so it's useful to refresh data automatically.
int32 default_data_refresh_window_days = 16
Default data refresh window on days. Only meaningful when `data_refresh_type` = `SLIDING_WINDOW`.
bool manual_runs_disabled = 17
Disables backfilling and manual run scheduling for the data source.
optional minimum_schedule_interval = 18
The minimum interval for scheduler to schedule runs.
The type of authorization needed for this data source.
Used in:
AUTHORIZATION_TYPE_UNSPECIFIED = 0
Type unspecified.
AUTHORIZATION_CODE = 1
Use OAuth 2 authorization codes that can be exchanged for a refresh token on the backend.
GOOGLE_PLUS_AUTHORIZATION_CODE = 2
Return an authorization code for a given Google+ page that can then be exchanged for a refresh token on the backend.
Represents how the data source supports data auto refresh.
Used in:
DATA_REFRESH_TYPE_UNSPECIFIED = 0
The data source won't support data auto refresh, which is default value.
SLIDING_WINDOW = 1
The data source supports data auto refresh, and runs will be scheduled for the past few days. Does not allow custom values to be set for each transfer config.
CUSTOM_SLIDING_WINDOW = 2
The data source supports data auto refresh, and runs will be scheduled for the past few days. Allows custom values to be set for each transfer config.
message
datasource.proto:470Represents the data source definition.
Used as response type in: DataSourceService.CreateDataSourceDefinition, DataSourceService.GetDataSourceDefinition, DataSourceService.UpdateDataSourceDefinition
Used as field type in: , ,
string name = 21
The resource name of the data source definition. Data source definition names have the form `projects/{project_id}/locations/{location}/dataSourceDefinitions/{data_source_id}`.
optional data_source = 1
Data source metadata.
string transfer_run_pubsub_topic = 13
The Pub/Sub topic to be used for broadcasting a message when a transfer run is created. Both this topic and transfer_config_pubsub_topic can be set to a custom topic. By default, both topics are auto-generated if none of them is provided when creating the definition. However, if one topic is manually set, the other topic has to be manually set as well. The only difference is that transfer_run_pubsub_topic must be a non-empty Pub/Sub topic, but transfer_config_pubsub_topic can be set to empty. The comments about "{location}" for transfer_config_pubsub_topic apply here too.
optional run_time_offset = 16
Duration which should be added to schedule_time to calculate run_time when job is scheduled. Only applicable for automatically scheduled transfer runs. Used to start a run early on a data source that supports continuous data refresh to compensate for unknown timezone offsets. Use a negative number to start a run late for data sources not supporting continuous data refresh.
string support_email = 22
Support e-mail address of the OAuth client's Brand, which contains the consent screen data.
string service_account = 2
When service account is specified, BigQuery will share created dataset with the given service account. Also, this service account will be eligible to perform status updates and message logging for data transfer runs for the corresponding data_source_id.
bool disabled = 5
Is data source disabled? If true, data_source is not visible. API will also stop returning any data transfer configs and/or runs associated with the data source. This setting has higher priority than whitelisted_project_ids.
string transfer_config_pubsub_topic = 12
The Pub/Sub topic to use for broadcasting a message for transfer config. If empty, a message will not be broadcasted. Both this topic and transfer_run_pubsub_topic are auto-generated if none of them is provided when creating the definition. It is recommended to provide transfer_config_pubsub_topic if a user-owned transfer_run_pubsub_topic is provided. Otherwise, it will be set to empty. If "{location}" is found in the value, then that means, data source wants to handle message separately for datasets in different regions. We will replace {location} with the actual dataset location, as the actual topic name. For example, projects/connector/topics/scheduler-{location} could become projects/connector/topics/scheduler-us. If "{location}" is not found, then we will use the input value as topic name.
repeated string supported_location_ids = 23
Supported location_ids used for deciding in which locations Pub/Sub topics need to be created. If custom Pub/Sub topics are used and they contains '{location}', the location_ids will be used for validating the topics by replacing the '{location}' with the individual location in the list. The valid values are the "location_id" field of the response of `GET https://bigquerydatatransfer.googleapis.com/v1/{name=projects/*}/locations` In addition, if the data source needs to support all available regions, supported_location_ids can be set to "global" (a single string element). When "global" is specified: 1) the data source implementation is supposed to stage the data in proper region of the destination dataset; 2) Data source developer should be aware of the implications (e.g., network traffic latency, potential charge associated with cross-region traffic, etc.) of supporting the "global" region;
message
datatransfer.proto:233Represents a data source parameter with validation rules, so that parameters can be rendered in the UI. These parameters are given to us by supported data sources, and include all needed information for rendering and validation. Thus, whoever uses this api can decide to generate either generic ui, or custom data source specific forms.
Used in:
string param_id = 1
Parameter identifier.
string display_name = 2
Parameter display name in the user interface.
string description = 3
Parameter description.
DataSourceParameter.Type type = 4
Parameter type.
bool required = 5
Is parameter required.
bool repeated = 6
Deprecated. This field has no effect.
string validation_regex = 7
Regular expression which can be used for parameter validation.
repeated string allowed_values = 8
All possible values for the parameter.
optional min_value = 9
For integer and double values specifies minimum allowed value.
optional max_value = 10
For integer and double values specifies maxminum allowed value.
repeated fields = 11
Deprecated. This field has no effect.
string validation_description = 12
Description of the requirements for this field, in case the user input does not fulfill the regex pattern or min/max values.
string validation_help_url = 13
URL to a help document to further explain the naming requirements.
bool immutable = 14
Cannot be changed after initial creation.
bool recurse = 15
Deprecated. This field has no effect.
bool deprecated = 20
If true, it should not be used in new transfers, and it should not be visible to users.
Parameter type.
Used in:
TYPE_UNSPECIFIED = 0
Type unspecified.
STRING = 1
String parameter.
INTEGER = 2
Integer parameter (64-bits). Will be serialized to json as string.
DOUBLE = 3
Double precision floating point parameter.
BOOLEAN = 4
Boolean parameter.
RECORD = 5
Deprecated. This field has no effect.
PLUS_PAGE = 6
Page ID for a Google+ Page.
message
datasource.proto:150Describes data which should be imported.
Used in:
string sql = 1
SQL query to run. When empty, API checks that there is only one table_def specified and loads this table. Only Standard SQL queries are accepted. Legacy SQL is not allowed.
string destination_table_id = 2
Table where results should be written.
string destination_table_description = 10
The description of a destination table. This can be several sentences or paragraphs describing the table contents in detail.
repeated table_defs = 3
When used WITHOUT the "sql" parameter, describes the schema of the destination table. When used WITH the "sql" parameter, describes tables with data stored outside of BigQuery.
repeated string user_defined_functions = 4
Inline code for User-defined function resources. Ignored when "sql" parameter is empty.
WriteDisposition write_disposition = 6
Specifies the action if the destination table already exists.
Encoding of input data in CSV/JSON format.
Used in:
ENCODING_UNSPECIFIED = 0
Default encoding (UTF8).
ISO_8859_1 = 1
ISO_8859_1 encoding.
UTF8 = 2
UTF8 encoding.
message
datasource.proto:152Defines schema of a field in the imported data.
Used in:
string field_name = 1
Field name. Matches: [A-Za-z_][A-Za-z_0-9]{0,127}
FieldSchema.Type type = 2
Field type
bool is_repeated = 3
Is field repeated.
string description = 4
Description for this field.
optional schema = 5
Present iff type == RECORD.
LINT.IfChange Field type.
Used in:
TYPE_UNSPECIFIED = 0
Illegal value.
STRING = 1
64K, UTF8.
INTEGER = 2
64-bit signed.
FLOAT = 3
64-bit IEEE floating point.
RECORD = 4
Aggregate type.
BYTES = 5
64K, Binary.
BOOLEAN = 6
2-valued.
TIMESTAMP = 7
64-bit signed usec since UTC epoch.
DATE = 8
Civil date - Year, Month, Day.
TIME = 9
Civil time - Hour, Minute, Second, Microseconds.
DATETIME = 10
Combination of civil date and civil time.
NUMERIC = 11
Numeric type with 38 decimal digits of precision and 9 decimal digits of scale.
GEOGRAPHY = 12
Geography object (go/googlesql_geography).
Data format.
Used in:
FORMAT_UNSPECIFIED = 0
Unspecified format. In this case, we have to infer the format from the data source.
CSV = 1
CSV format.
JSON = 2
Newline-delimited JSON.
AVRO = 3
Avro format. See http://avro.apache.org .
RECORDIO = 4
RecordIO.
COLUMNIO = 5
ColumnIO.
CAPACITOR = 6
Capacitor.
PARQUET = 7
Parquet format. See https://parquet.apache.org .
ORC = 8
ORC format. See https://orc.apache.org .
message
datasource.proto:214Describes schema of the data to be ingested.
Used in: ,
repeated fields = 1
One field per column in the record.
message
datasource.proto:221External table definition. These tables can be referenced with 'name' in the query and can be read just like any other table.
Used in:
string table_id = 1
BigQuery table_id (required). This will be used to reference this table in the query.
repeated string source_uris = 2
URIs for the data to be imported. All URIs must be from the same storage system.
Format format = 3
Describes the format of the data in source_uri.
int32 max_bad_records = 4
Specify the maximum number of bad records that can be ignored. If bad records exceed this threshold the query is aborted.
Encoding encoding = 5
Character encoding of the input when applicable (CSV, JSON). Defaults to UTF8.
optional csv_options = 6
CSV specific options.
optional schema = 7
Optional schema for the data. When not specified for JSON and CSV formats we will try to detect it automatically.
optional ignore_unknown_values = 10
Indicates if extra values that are not represented in the table schema is allowed.
message
datasource.proto:223CSV specific options.
Used in:
optional field_delimiter = 1
The delimiter. We currently restrict this to U+0001 to U+00FF and apply additional constraints during validation.
optional allow_quoted_newlines = 2
Whether CSV files are allowed to have quoted newlines. If quoted newlines are allowed, we can't split CSV files.
optional quote_char = 3
The quote character. We currently restrict this to U+0000 to U+00FF and apply additional constraints during validation. Set to '\0' to indicate no quote is used.
optional skip_leading_rows = 4
Number of leading rows to skip.
optional allow_jagged_rows = 5
Accept rows that are missing trailing optional columns.
Represents which runs should be pulled.
Used in:
RUN_ATTEMPT_UNSPECIFIED = 0
All runs should be returned.
LATEST = 1
Only latest run per day should be returned.
message
transfer.proto:72Options customizing the data transfer schedule.
Used in:
bool disable_auto_scheduling = 3
If true, automatic scheduling of data transfer runs for this configuration will be disabled. The runs can be started on ad-hoc basis using StartManualTransferRuns API. When automatic scheduling is disabled, the TransferConfig.schedule field will be ignored.
optional start_time = 1
Specifies time to start scheduling transfer runs. The first run will be scheduled at or after the start time according to a recurrence pattern defined in the schedule string. The start time can be changed at any moment. The time when a data transfer can be trigerred manually is not limited by this option.
optional end_time = 2
Defines time to stop scheduling transfer runs. A transfer run cannot be scheduled at or after the end time. The end time can be changed at any moment. The time when a data transfer can be trigerred manually is not limited by this option.
message
datatransfer.proto:776A specification for a time range, this will request transfer runs with run_time between start_time (inclusive) and end_time (exclusive).
Used in:
optional start_time = 1
Start time of the range of transfer runs. For example, `"2017-05-25T00:00:00+00:00"`. The start_time must be strictly less than the end_time. Creates transfer runs where run_time is in the range betwen start_time (inclusive) and end_time (exlusive).
optional end_time = 2
End time of the range of transfer runs. For example, `"2017-05-30T00:00:00+00:00"`. The end_time must not be in the future. Creates transfer runs where run_time is in the range betwen start_time (inclusive) and end_time (exlusive).
message
transfer.proto:99Represents a data transfer configuration. A transfer configuration contains all metadata needed to perform a data transfer. For example, `destination_dataset_id` specifies where data should be stored. When a new transfer configuration is created, the specified `destination_dataset_id` is created when needed and shared with the appropriate data source service account.
Used as response type in: DataTransferService.CreateTransferConfig, DataTransferService.GetTransferConfig, DataTransferService.UpdateTransferConfig
Used as field type in: , ,
string name = 1
The resource name of the transfer config. Transfer config names have the form of `projects/{project_id}/locations/{region}/transferConfigs/{config_id}`. The name is automatically generated based on the config_id specified in CreateTransferConfigRequest along with project_id and region. If config_id is not provided, usually a uuid, even though it is not guaranteed or required, will be generated for config_id.
oneof destination
The desination of the transfer config.
string destination_dataset_id = 2
The BigQuery target dataset id.
string display_name = 3
User specified display name for the data transfer.
string data_source_id = 5
Data source id. Cannot be changed once data transfer is created.
optional params = 9
Data transfer specific parameters.
string schedule = 7
Data transfer schedule. If the data source does not support a custom schedule, this should be empty. If it is empty, the default value for the data source will be used. The specified times are in UTC. Examples of valid format: `1st,3rd monday of month 15:30`, `every wed,fri of jan,jun 13:15`, and `first sunday of quarter 00:00`. See more explanation about the format here: https://cloud.google.com/appengine/docs/flexible/python/scheduling-jobs-with-cron-yaml#the_schedule_format NOTE: the granularity should be at least 8 hours, or less frequent.
optional schedule_options = 24
Options customizing the data transfer schedule.
int32 data_refresh_window_days = 12
The number of days to look back to automatically refresh the data. For example, if `data_refresh_window_days = 10`, then every day BigQuery reingests data for [today-10, today-1], rather than ingesting data for just [today-1]. Only valid if the data source supports the feature. Set the value to 0 to use the default value.
bool disabled = 13
Is this config disabled. When set to true, no runs are scheduled for a given transfer.
optional update_time = 4
Output only. Data transfer modification time. Ignored by server on input.
optional next_run_time = 8
Output only. Next time when data transfer will run.
TransferState state = 10
Output only. State of the most recently updated transfer run.
int64 user_id = 11
Deprecated. Unique ID of the user on whose behalf transfer is done.
string dataset_region = 14
Output only. Region in which BigQuery dataset is located.
message
transfer.proto:235Represents a user facing message for a particular data transfer run.
Used in: ,
optional message_time = 1
Time when message was logged.
TransferMessage.MessageSeverity severity = 2
Message severity.
string message_text = 3
Message text.
Represents data transfer user facing message severity.
Used in: ,
MESSAGE_SEVERITY_UNSPECIFIED = 0
No severity specified.
INFO = 1
Informational message.
WARNING = 2
Warning message.
ERROR = 3
Error message.
message
transfer.proto:175Represents a data transfer run.
Used as response type in: DataSourceService.UpdateTransferRun, DataTransferService.GetTransferRun
Used as field type in: , , ,
string name = 1
The resource name of the transfer run. Transfer run names have the form `projects/{project_id}/locations/{location}/transferConfigs/{config_id}/runs/{run_id}`. The name is ignored when creating a transfer run.
optional schedule_time = 3
Minimum time after which a transfer run can be started.
optional run_time = 10
For batch transfer runs, specifies the date and time of the data should be ingested.
optional error_status = 21
Status of the transfer run.
optional start_time = 4
Output only. Time when transfer run was started. Parameter ignored by server for input requests.
optional end_time = 5
Output only. Time when transfer run ended. Parameter ignored by server for input requests.
optional update_time = 6
Output only. Last time the data transfer run state was updated.
optional params = 9
Output only. Data transfer specific parameters.
oneof destination
Data transfer destination.
string destination_dataset_id = 2
Output only. The BigQuery target dataset id.
string data_source_id = 7
Output only. Data source id.
TransferState state = 8
Data transfer run state. Ignored for input requests.
int64 user_id = 11
Deprecated. Unique ID of the user on whose behalf transfer is done.
string schedule = 12
Output only. Describes the schedule of this transfer run if it was created as part of a regular schedule. For batch transfer runs that are scheduled manually, this is empty. NOTE: the system might choose to delay the schedule depending on the current load, so `schedule_time` doesn't always match this.
Represents data transfer run state.
Used in: , ,
TRANSFER_STATE_UNSPECIFIED = 0
State placeholder.
PENDING = 2
Data transfer is scheduled and is waiting to be picked up by data transfer backend.
RUNNING = 3
Data transfer is in progress.
SUCCEEDED = 4
Data transfer completed successfully.
FAILED = 5
Data transfer failed.
CANCELLED = 6
Data transfer is cancelled.
DEPRECATED. Represents data transfer type.
Used in:
TRANSFER_TYPE_UNSPECIFIED = 0
Invalid or Unknown transfer type placeholder.
BATCH = 1
Batch data transfer.
STREAMING = 2
Streaming data transfer. Streaming data source currently doesn't support multiple transfer configs per project.
Options for writing to the table. The WRITE_EMPTY option is intentionally excluded from the enum and is not supported by the data transfer service.
Used in:
WRITE_DISPOSITION_UNSPECIFIED = 0
The default writeDispostion
WRITE_TRUNCATE = 1
overwrites the table data.
WRITE_APPEND = 2
the data is appended to the table. Note duplication might happen if this mode is used.