package google.cloud.bigquery.datatransfer.v1
Get desktop application:
View/edit binary Protocol Buffers messages
The Google BigQuery Data Transfer Service API enables BigQuery users to configure the transfer of their data from other Google Products into BigQuery. This service contains methods that are end user exposed. It backs up the frontend.
Returns true if valid credentials exist for the given data source and requesting user. Some data sources doesn't support service account, so we need to talk to them on behalf of the end user. This API just checks whether we have OAuth token for the particular user, which is a pre-requisite before user can create a transfer config.
message
datatransfer.proto:622A request to determine whether the user has valid credentials. This method is used to limit the number of OAuth popups in the user interface. The user id is inferred from the API call context. If the data source has the Google+ authorization type, this method returns false, as it cannot be determined whether the credentials are already valid merely based on the user id.
string name = 1
The data source in the form: `projects/{project_id}/dataSources/{data_source_id}`
message
datatransfer.proto:629A response indicating whether the credentials exist and are valid.
bool has_valid_creds = 1
If set to `true`, the credentials exist and are valid.
Creates a new data transfer configuration.
message
datatransfer.proto:418A request to create a data transfer configuration. If new credentials are needed for this transfer configuration, an authorization code must be provided. If an authorization code is provided, the transfer configuration will be associated with the user id corresponding to the authorization code. Otherwise, the transfer configuration will be associated with the calling user.
string parent = 1
The BigQuery project id where the transfer configuration should be created. Must be in the format /projects/{project_id}/locations/{location_id} If specified location and location of the destination bigquery dataset do not match - the request will fail.
optional transfer_config = 2
Data transfer configuration to create.
string authorization_code = 3
Optional OAuth2 authorization code to use with this transfer configuration. This is required if new credentials are needed, as indicated by `CheckValidCreds`. In order to obtain authorization_code, please make a request to https://www.gstatic.com/bigquerydatatransfer/oauthz/auth?client_id=<datatransferapiclientid>&scope=<data_source_scopes>&redirect_uri=<redirect_uri> * client_id should be OAuth client_id of BigQuery DTS API for the given data source returned by ListDataSources method. * data_source_scopes are the scopes returned by ListDataSources method. * redirect_uri is an optional parameter. If not specified, then authorization code is posted to the opener of authorization flow window. Otherwise it will be sent to the redirect uri. A special value of urn:ietf:wg:oauth:2.0:oob means that authorization code should be returned in the title bar of the browser, with the page text prompting the user to copy the code and paste it in the application.
Deletes a data transfer configuration, including any associated transfer runs and logs.
message
datatransfer.proto:484A request to delete data transfer information. All associated transfer runs and log messages will be deleted as well.
string name = 1
The field will contain name of the resource requested, for example: `projects/{project_id}/transferConfigs/{config_id}`
Deletes the specified transfer run.
message
datatransfer.proto:498A request to delete data transfer run information.
string name = 1
The field will contain name of the resource requested, for example: `projects/{project_id}/transferConfigs/{config_id}/runs/{run_id}`
rpc GetDataSource (, )
datatransfer.proto:42Retrieves a supported data source and returns its settings, which can be used for UI rendering.
message
datatransfer.proto:377A request to get data source info.
string name = 1
The field will contain name of the resource requested, for example: `projects/{project_id}/dataSources/{data_source_id}`
Returns information about a data transfer config.
message
datatransfer.proto:476A request to get data transfer information.
string name = 1
The field will contain name of the resource requested, for example: `projects/{project_id}/transferConfigs/{config_id}`
rpc GetTransferRun (, )
datatransfer.proto:134Returns information about the particular transfer run.
message
datatransfer.proto:491A request to get data transfer run information.
string name = 1
The field will contain name of the resource requested, for example: `projects/{project_id}/transferConfigs/{config_id}/runs/{run_id}`
rpc ListDataSources (, )
datatransfer.proto:53Lists supported data sources and returns their settings, which can be used for UI rendering.
message
datatransfer.proto:384Request to list supported data sources and their data transfer settings.
string parent = 1
The BigQuery project id for which data sources should be returned. Must be in the form: `projects/{project_id}`
string page_token = 3
Pagination token, which can be used to request a specific page of `ListDataSourcesRequest` list results. For multiple-page results, `ListDataSourcesResponse` outputs a `next_page` token, which can be used as the `page_token` value to request the next page of list results.
int32 page_size = 4
Page size. The default page size is the maximum value of 1000 results.
message
datatransfer.proto:401Returns list of supported data sources and their metadata.
repeated data_sources = 1
List of supported data sources and their transfer settings.
string next_page_token = 2
Output only. The next-pagination token. For multiple-page list results, this token can be used as the `ListDataSourcesRequest.page_token` to request the next page of list results.
Returns information about all data transfers in the project.
message
datatransfer.proto:505A request to list data transfers configured for a BigQuery project.
string parent = 1
The BigQuery project id for which data sources should be returned: `projects/{project_id}`.
repeated string data_source_ids = 2
When specified, only configurations of requested data sources are returned.
string page_token = 3
Pagination token, which can be used to request a specific page of `ListTransfersRequest` list results. For multiple-page results, `ListTransfersResponse` outputs a `next_page` token, which can be used as the `page_token` value to request the next page of list results.
int32 page_size = 4
Page size. The default page size is the maximum value of 1000 results.
message
datatransfer.proto:525The returned list of pipelines in the project.
repeated transfer_configs = 1
Output only. The stored pipeline transfer configurations.
string next_page_token = 2
Output only. The next-pagination token. For multiple-page list results, this token can be used as the `ListTransferConfigsRequest.page_token` to request the next page of list results.
Returns user facing log messages for the data transfer run.
message
datatransfer.proto:584A request to get user facing log messages associated with data transfer run.
string parent = 1
Transfer run name in the form: `projects/{project_id}/transferConfigs/{config_Id}/runs/{run_id}`.
string page_token = 4
Pagination token, which can be used to request a specific page of `ListTransferLogsRequest` list results. For multiple-page results, `ListTransferLogsResponse` outputs a `next_page` token, which can be used as the `page_token` value to request the next page of list results.
int32 page_size = 5
Page size. The default page size is the maximum value of 1000 results.
repeated TransferMessage.MessageSeverity message_types = 6
Message types to return. If not populated - INFO, WARNING and ERROR messages are returned.
message
datatransfer.proto:605The returned list transfer run messages.
repeated transfer_messages = 1
Output only. The stored pipeline transfer messages.
string next_page_token = 2
Output only. The next-pagination token. For multiple-page list results, this token can be used as the `GetTransferRunLogRequest.page_token` to request the next page of list results.
Returns information about running and completed jobs.
message
datatransfer.proto:539A request to list data transfer runs. UI can use this method to show/filter specific data transfer runs. The data source can use this method to request all scheduled transfer runs.
string parent = 1
Name of transfer configuration for which transfer runs should be retrieved. Format of transfer configuration resource name is: `projects/{project_id}/transferConfigs/{config_id}`.
repeated TransferState states = 2
When specified, only transfer runs with requested states are returned.
string page_token = 3
Pagination token, which can be used to request a specific page of `ListTransferRunsRequest` list results. For multiple-page results, `ListTransferRunsResponse` outputs a `next_page` token, which can be used as the `page_token` value to request the next page of list results.
int32 page_size = 4
Page size. The default page size is the maximum value of 1000 results.
ListTransferRunsRequest.RunAttempt run_attempt = 5
Indicates how run attempts are to be pulled.
message
datatransfer.proto:572The returned list of pipelines in the project.
repeated transfer_runs = 1
Output only. The stored pipeline transfer runs.
string next_page_token = 2
Output only. The next-pagination token. For multiple-page list results, this token can be used as the `ListTransferRunsRequest.page_token` to request the next page of list results.
Creates transfer runs for a time range [start_time, end_time]. For each date - or whatever granularity the data source supports - in the range, one transfer run is created. Note that runs are created per UTC time in the time range.
message
datatransfer.proto:635A request to schedule transfer runs for a time range.
string parent = 1
Transfer configuration name in the form: `projects/{project_id}/transferConfigs/{config_id}`.
optional start_time = 2
Start time of the range of transfer runs. For example, `"2017-05-25T00:00:00+00:00"`.
optional end_time = 3
End time of the range of transfer runs. For example, `"2017-05-30T00:00:00+00:00"`.
message
datatransfer.proto:650A response to schedule transfer runs for a time range.
repeated runs = 1
The transfer runs that were scheduled.
Updates a data transfer configuration. All fields must be set, even if they are not updated.
message
datatransfer.proto:449A request to update a transfer configuration. To update the user id of the transfer configuration, an authorization code needs to be provided.
optional transfer_config = 1
Data transfer configuration to create.
string authorization_code = 3
Optional OAuth2 authorization code to use with this transfer configuration. If it is provided, the transfer configuration will be associated with the authorizing user. In order to obtain authorization_code, please make a request to https://www.gstatic.com/bigquerydatatransfer/oauthz/auth?client_id=<datatransferapiclientid>&scope=<data_source_scopes>&redirect_uri=<redirect_uri> * client_id should be OAuth client_id of BigQuery DTS API for the given data source returned by ListDataSources method. * data_source_scopes are the scopes returned by ListDataSources method. * redirect_uri is an optional parameter. If not specified, then authorization code is posted to the opener of authorization flow window. Otherwise it will be sent to the redirect uri. A special value of urn:ietf:wg:oauth:2.0:oob means that authorization code should be returned in the title bar of the browser, with the page text prompting the user to copy the code and paste it in the application.
optional update_mask = 4
Required list of fields to be updated in this request.
message
datatransfer.proto:273Represents data source metadata. Metadata is sufficient to render UI and request proper OAuth tokens.
Used as response type in: DataTransferService.GetDataSource
Used as field type in:
string name = 1
Output only. Data source resource name.
string data_source_id = 2
Data source id.
string display_name = 3
User friendly data source name.
string description = 4
User friendly data source description string.
string client_id = 5
Data source client id which should be used to receive refresh token. When not supplied, no offline credentials are populated for data transfer.
repeated string scopes = 6
Api auth scopes for which refresh token needs to be obtained. Only valid when `client_id` is specified. Ignored otherwise. These are scopes needed by a data source to prepare data and ingest them into BigQuery, e.g., https://www.googleapis.com/auth/bigquery
TransferType transfer_type = 7
Deprecated. This field has no effect.
bool supports_multiple_transfers = 8
Indicates whether the data source supports multiple transfers to different BigQuery targets.
int32 update_deadline_seconds = 9
The number of seconds to wait for an update from the data source before BigQuery marks the transfer as failed.
string default_schedule = 10
Default data transfer schedule. Examples of valid schedules include: `1st,3rd monday of month 15:30`, `every wed,fri of jan,jun 13:15`, and `first sunday of quarter 00:00`.
bool supports_custom_schedule = 11
Specifies whether the data source supports a user defined schedule, or operates on the default schedule. When set to `true`, user can override default schedule.
repeated parameters = 12
Data source parameters.
string help_url = 13
Url for the help document for this data source.
DataSource.AuthorizationType authorization_type = 14
Indicates the type of authorization.
DataSource.DataRefreshType data_refresh_type = 15
Specifies whether the data source supports automatic data refresh for the past few days, and how it's supported. For some data sources, data might not be complete until a few days later, so it's useful to refresh data automatically.
int32 default_data_refresh_window_days = 16
Default data refresh window on days. Only meaningful when `data_refresh_type` = `SLIDING_WINDOW`.
bool manual_runs_disabled = 17
Disables backfilling and manual run scheduling for the data source.
optional minimum_schedule_interval = 18
The minimum interval for scheduler to schedule runs.
The type of authorization needed for this data source.
Used in:
AUTHORIZATION_TYPE_UNSPECIFIED = 0
Type unspecified.
AUTHORIZATION_CODE = 1
Use OAuth 2 authorization codes that can be exchanged for a refresh token on the backend.
GOOGLE_PLUS_AUTHORIZATION_CODE = 2
Return an authorization code for a given Google+ page that can then be exchanged for a refresh token on the backend.
Represents how the data source supports data auto refresh.
Used in:
DATA_REFRESH_TYPE_UNSPECIFIED = 0
The data source won't support data auto refresh, which is default value.
SLIDING_WINDOW = 1
The data source supports data auto refresh, and runs will be scheduled for the past few days. Does not allow custom values to be set for each transfer config.
CUSTOM_SLIDING_WINDOW = 2
The data source supports data auto refresh, and runs will be scheduled for the past few days. Allows custom values to be set for each transfer config.
message
datatransfer.proto:197Represents a data source parameter with validation rules, so that parameters can be rendered in the UI. These parameters are given to us by supported data sources, and include all needed information for rendering and validation. Thus, whoever uses this api can decide to generate either generic ui, or custom data source specific forms.
Used in:
string param_id = 1
Parameter identifier.
string display_name = 2
Parameter display name in the user interface.
string description = 3
Parameter description.
DataSourceParameter.Type type = 4
Parameter type.
bool required = 5
Is parameter required.
bool repeated = 6
Can parameter have multiple values.
string validation_regex = 7
Regular expression which can be used for parameter validation.
repeated string allowed_values = 8
All possible values for the parameter.
optional min_value = 9
For integer and double values specifies minimum allowed value.
optional max_value = 10
For integer and double values specifies maxminum allowed value.
repeated fields = 11
When parameter is a record, describes child fields.
string validation_description = 12
Description of the requirements for this field, in case the user input does not fulfill the regex pattern or min/max values.
string validation_help_url = 13
URL to a help document to further explain the naming requirements.
bool immutable = 14
Cannot be changed after initial creation.
bool recurse = 15
If set to true, schema should be taken from the parent with the same parameter_id. Only applicable when parameter type is RECORD.
Parameter type.
Used in:
TYPE_UNSPECIFIED = 0
Type unspecified.
STRING = 1
String parameter.
INTEGER = 2
Integer parameter (64-bits). Will be serialized to json as string.
DOUBLE = 3
Double precision floating point parameter.
BOOLEAN = 4
Boolean parameter.
RECORD = 5
Record parameter.
PLUS_PAGE = 6
Page ID for a Google+ Page.
Represents which runs should be pulled.
Used in:
RUN_ATTEMPT_UNSPECIFIED = 0
All runs should be returned.
LATEST = 1
Only latest run per day should be returned.
message
transfer.proto:39Represents a data transfer configuration. A transfer configuration contains all metadata needed to perform a data transfer. For example, `destination_dataset_id` specifies where data should be stored. When a new transfer configuration is created, the specified `destination_dataset_id` is created when needed and shared with the appropriate data source service account.
Used as response type in: DataTransferService.CreateTransferConfig, DataTransferService.GetTransferConfig, DataTransferService.UpdateTransferConfig
Used as field type in: , ,
string name = 1
The resource name of the transfer config. Transfer config names have the form `projects/{project_id}/transferConfigs/{config_id}`. Where `config_id` is usually a uuid, even though it is not guaranteed or required. The name is ignored when creating a transfer config.
string destination_dataset_id = 2
The BigQuery target dataset id.
string display_name = 3
User specified display name for the data transfer.
string data_source_id = 5
Data source id. Cannot be changed once data transfer is created.
optional params = 9
Data transfer specific parameters.
string schedule = 7
Data transfer schedule. If the data source does not support a custom schedule, this should be empty. If it is empty, the default value for the data source will be used. The specified times are in UTC. Examples of valid format: `1st,3rd monday of month 15:30`, `every wed,fri of jan,jun 13:15`, and `first sunday of quarter 00:00`. See more explanation about the format here: https://cloud.google.com/appengine/docs/flexible/python/scheduling-jobs-with-cron-yaml#the_schedule_format NOTE: the granularity should be at least 8 hours, or less frequent.
int32 data_refresh_window_days = 12
The number of days to look back to automatically refresh the data. For example, if `data_refresh_window_days = 10`, then every day BigQuery reingests data for [today-10, today-1], rather than ingesting data for just [today-1]. Only valid if the data source supports the feature. Set the value to 0 to use the default value.
bool disabled = 13
Is this config disabled. When set to true, no runs are scheduled for a given transfer.
optional update_time = 4
Output only. Data transfer modification time. Ignored by server on input.
optional next_run_time = 8
Output only. Next time when data transfer will run.
TransferState state = 10
Output only. State of the most recently updated transfer run.
int64 user_id = 11
Output only. Unique ID of the user on whose behalf transfer is done. Applicable only to data sources that do not support service accounts. When set to 0, the data source service account credentials are used. May be negative. Note, that this identifier is not stable. It may change over time even for the same user.
string dataset_region = 14
Output only. Region in which BigQuery dataset is located.
message
transfer.proto:163Represents a user facing message for a particular data transfer run.
Used in:
optional message_time = 1
Time when message was logged.
TransferMessage.MessageSeverity severity = 2
Message severity.
string message_text = 3
Message text.
Represents data transfer user facing message severity.
Used in: ,
MESSAGE_SEVERITY_UNSPECIFIED = 0
No severity specified.
INFO = 1
Informational message.
WARNING = 2
Warning message.
ERROR = 3
Error message.
message
transfer.proto:107Represents a data transfer run.
Used as response type in: DataTransferService.GetTransferRun
Used as field type in: ,
string name = 1
The resource name of the transfer run. Transfer run names have the form `projects/{project_id}/locations/{location}/transferConfigs/{config_id}/runs/{run_id}`. The name is ignored when creating a transfer run.
optional schedule_time = 3
Minimum time after which a transfer run can be started.
optional run_time = 10
For batch transfer runs, specifies the date and time that data should be ingested.
optional error_status = 21
Status of the transfer run.
optional start_time = 4
Output only. Time when transfer run was started. Parameter ignored by server for input requests.
optional end_time = 5
Output only. Time when transfer run ended. Parameter ignored by server for input requests.
optional update_time = 6
Output only. Last time the data transfer run state was updated.
optional params = 9
Output only. Data transfer specific parameters.
string destination_dataset_id = 2
Output only. The BigQuery target dataset id.
string data_source_id = 7
Output only. Data source id.
TransferState state = 8
Data transfer run state. Ignored for input requests.
int64 user_id = 11
Output only. Unique ID of the user on whose behalf transfer is done. Applicable only to data sources that do not support service accounts. When set to 0, the data source service account credentials are used. May be negative. Note, that this identifier is not stable. It may change over time even for the same user.
string schedule = 12
Output only. Describes the schedule of this transfer run if it was created as part of a regular schedule. For batch transfer runs that are scheduled manually, this is empty. NOTE: the system might choose to delay the schedule depending on the current load, so `schedule_time` doesn't always matches this.
Represents data transfer run state.
Used in: , ,
TRANSFER_STATE_UNSPECIFIED = 0
State placeholder.
PENDING = 2
Data transfer is scheduled and is waiting to be picked up by data transfer backend.
RUNNING = 3
Data transfer is in progress.
SUCCEEDED = 4
Data transfer completed successsfully.
FAILED = 5
Data transfer failed.
CANCELLED = 6
Data transfer is cancelled.
DEPRECATED. Represents data transfer type.
Used in:
TRANSFER_TYPE_UNSPECIFIED = 0
Invalid or Unknown transfer type placeholder.
BATCH = 1
Batch data transfer.
STREAMING = 2
Streaming data transfer. Streaming data source currently doesn't support multiple transfer configs per project.