package google.cloud.bigquery.storage.v1
Get desktop application:
View/edit binary Protocol Buffers messages
service BigQueryRead
storage.proto:46BigQuery Read API. The Read API can be used to read data from BigQuery.
rpc CreateReadSession (, )
storage.proto:71Creates a new read session. A read session divides the contents of a BigQuery table into one or more streams, which can then be used to read data from the table. The read session also specifies properties of the data to be read, such as a list of columns or a push-down filter describing the rows to be returned. A particular row can be read by at most one stream. When the caller has reached the end of each stream in the session, then all the data in the table has been read. Data is assigned to each stream such that roughly the same number of rows can be read from each stream. Because the server-side unit for assigning data is collections of rows, the API does not guarantee that each stream will return the same number or rows. Additionally, the limits are enforced based on the number of pre-filtered rows, so some filters can lead to lopsided assignments. Read sessions automatically expire 6 hours after they are created and do not require manual clean-up by the caller.
message
storage.proto:232Request message for `CreateReadSession`.
string parent = 1
Required. The request project that owns the session, in the form of `projects/{project_id}`.
optional read_session = 2
Required. Session to be created.
int32 max_stream_count = 3
Max initial number of streams. If unset or zero, the server will provide a value of streams so as to produce reasonable throughput. Must be non-negative. The number of streams may be lower than the requested number, depending on the amount parallelism that is reasonable for the table. There is a default system max limit of 1,000. This must be greater than or equal to preferred_min_stream_count. Typically, clients should either leave this unset to let the system to determine an upper bound OR set this a size for the maximum "units of work" it can gracefully handle.
int32 preferred_min_stream_count = 4
The minimum preferred stream count. This parameter can be used to inform the service that there is a desired lower bound on the number of streams. This is typically a target parallelism of the client (e.g. a Spark cluster with N-workers would set this to a low multiple of N to ensure good cluster utilization). The system will make a best effort to provide at least this number of streams, but in some cases might provide less.
rpc ReadRows (, stream )
storage.proto:87Reads rows from the stream in the format prescribed by the ReadSession. Each response contains one or more table rows, up to a maximum of 100 MiB per response; read requests which attempt to read individual rows larger than 100 MiB will fail. Each request also returns a set of stream statistics reflecting the current state of the stream.
message
storage.proto:269Request message for `ReadRows`.
string read_stream = 1
Required. Stream to read rows from.
int64 offset = 2
The offset requested must be less than the last row read from Read. Requesting a larger offset is undefined. If not specified, start reading from offset zero.
message
storage.proto:319Response from calling `ReadRows` may include row data, progress and throttling information.
oneof rows
Row data is returned in format specified during session creation.
avro_rows = 3
Serialized row data in AVRO format.
arrow_record_batch = 4
Serialized row data in Arrow RecordBatch format.
int64 row_count = 6
Number of serialized rows in the rows block.
optional stats = 2
Statistics for the stream.
optional throttle_state = 5
Throttling state. If unset, the latest response still describes the current throttling status.
oneof schema
The schema for the read. If read_options.selected_fields is set, the schema may be different from the table schema as it will only contain the selected fields. This schema is equivalent to the one returned by CreateSession. This field is only populated in the first ReadRowsResponse RPC.
avro_schema = 7
Output only. Avro schema.
arrow_schema = 8
Output only. Arrow schema.
optional int64 uncompressed_byte_size = 9
Optional. If the row data in this ReadRowsResponse is compressed, then uncompressed byte size is the original size of the uncompressed row data. If it is set to a value greater than 0, then decompress into a buffer of size uncompressed_byte_size using the compression codec that was requested during session creation time and which is specified in TableReadOptions.response_compression_codec in ReadSession. This value is not set if no response_compression_codec was not requested and it is -1 if the requested compression would not have reduced the size of this ReadRowsResponse's row data. This attempts to match Apache Arrow's behavior described here https://github.com/apache/arrow/issues/15102 where the uncompressed length may be set to -1 to indicate that the data that follows is not compressed, which can be useful for cases where compression does not yield appreciable savings. When uncompressed_byte_size is not greater than 0, the client should skip decompression.
rpc SplitReadStream (, )
storage.proto:106Splits a given `ReadStream` into two `ReadStream` objects. These `ReadStream` objects are referred to as the primary and the residual streams of the split. The original `ReadStream` can still be read from in the same manner as before. Both of the returned `ReadStream` objects can also be read from, and the rows returned by both child streams will be the same as the rows read from the original stream. Moreover, the two child streams will be allocated back-to-back in the original `ReadStream`. Concretely, it is guaranteed that for streams original, primary, and residual, that original[0-j] = primary[0-j] and original[j-n] = residual[0-m] once the streams have been read to completion.
message
storage.proto:371Request message for `SplitReadStream`.
string name = 1
Required. Name of the stream to split.
double fraction = 2
A value in the range (0.0, 1.0) that specifies the fractional point at which the original stream should be split. The actual split point is evaluated on pre-filtered rows, so if a filter is provided, then there is no guarantee that the division of the rows between the new child streams will be proportional to this fractional value. Additionally, because the server-side unit for assigning data is collections of rows, this fraction will always map to a data storage boundary on the server side.
message
storage.proto:391Response message for `SplitReadStream`.
optional primary_stream = 1
Primary stream, which contains the beginning portion of |original_stream|. An empty value indicates that the original stream can no longer be split.
optional remainder_stream = 2
Remainder stream, which contains the tail of |original_stream|. An empty value indicates that the original stream can no longer be split.
service BigQueryWrite
storage.proto:120BigQuery Write API. The Write API can be used to write data to BigQuery. For supplementary information about the Write API, see: https://cloud.google.com/bigquery/docs/write-api
rpc CreateWriteStream (, )
storage.proto:133Creates a write stream to the given table. Additionally, every table has a special stream named '_default' to which data can be written. This stream doesn't need to be created using CreateWriteStream. It is a stream that can be used simultaneously by any number of clients. Data written to this stream is considered committed as soon as an acknowledgement is received.
message
storage.proto:403Request message for `CreateWriteStream`.
string parent = 1
Required. Reference to the table to which the stream belongs, in the format of `projects/{project}/datasets/{dataset}/tables/{table}`.
optional write_stream = 2
Required. Stream to be created.
rpc AppendRows (stream , stream )
storage.proto:172Appends data to the given stream. If `offset` is specified, the `offset` is checked against the end of stream. The server returns `OUT_OF_RANGE` in `AppendRowsResponse` if an attempt is made to append to an offset beyond the current end of the stream or `ALREADY_EXISTS` if user provides an `offset` that has already been written to. User can retry with adjusted offset within the same RPC connection. If `offset` is not specified, append happens at the end of the stream. The response contains an optional offset at which the append happened. No offset information will be returned for appends to a default stream. Responses are received in the same order in which requests are sent. There will be one response for each successful inserted request. Responses may optionally embed error information if the originating AppendRequest was not successfully processed. The specifics of when successfully appended data is made visible to the table are governed by the type of stream: * For COMMITTED streams (which includes the default stream), data is visible immediately upon successful append. * For BUFFERED streams, data is made visible via a subsequent `FlushRows` rpc which advances a cursor to a newer offset in the stream. * For PENDING streams, data is not made visible until the stream itself is finalized (via the `FinalizeWriteStream` rpc), and the stream is explicitly committed via the `BatchCommitWriteStreams` rpc.
message
storage.proto:424Request message for `AppendRows`. Because AppendRows is a bidirectional streaming RPC, certain parts of the AppendRowsRequest need only be specified for the first request before switching table destinations. You can also switch table destinations within the same connection for the default stream. The size of a single AppendRowsRequest must be less than 10 MB in size. Requests larger than this return an error, typically `INVALID_ARGUMENT`.
string write_stream = 1
Required. The write_stream identifies the append operation. It must be provided in the following scenarios: * In the first request to an AppendRows connection. * In all subsequent requests to an AppendRows connection, if you use the same connection to write to multiple tables or change the input schema for default streams. For explicitly created write streams, the format is: * `projects/{project}/datasets/{dataset}/tables/{table}/streams/{id}` For the special default stream, the format is: * `projects/{project}/datasets/{dataset}/tables/{table}/streams/_default`. An example of a possible sequence of requests with write_stream fields within a single connection: * r1: {write_stream: stream_name_1} * r2: {write_stream: /*omit*/} * r3: {write_stream: /*omit*/} * r4: {write_stream: stream_name_2} * r5: {write_stream: stream_name_2} The destination changed in request_4, so the write_stream field must be populated in all subsequent requests in this stream.
optional offset = 2
If present, the write is only performed if the next append offset is same as the provided value. If not present, the write is performed at the current end of stream. Specifying a value for this field is not allowed when calling AppendRows for the '_default' stream.
oneof rows
Input rows. The `writer_schema` field must be specified at the initial request and currently, it will be ignored if specified in following requests. Following requests must have data in the same format as the initial request.
proto_rows = 4
Rows in proto format.
string trace_id = 6
Id set by client to annotate its identity. Only initial request setting is respected.
map<string, AppendRowsRequest.MissingValueInterpretation> missing_value_interpretations = 7
A map to indicate how to interpret missing value for some fields. Missing values are fields present in user schema but missing in rows. The key is the field name. The value is the interpretation of missing values for the field. For example, a map {'foo': NULL_VALUE, 'bar': DEFAULT_VALUE} means all missing values in field foo are interpreted as NULL, all missing values in field bar are interpreted as the default value of field bar in table schema. If a field is not in this map and has missing values, the missing values in this field are interpreted as NULL. This field only applies to the current request, it won't affect other requests on the connection. Currently, field name can only be top-level column name, can't be a struct field path like 'foo.bar'.
AppendRowsRequest.MissingValueInterpretation default_missing_value_interpretation = 8
Optional. Default missing value interpretation for all columns in the table. When a value is specified on an `AppendRowsRequest`, it is applied to all requests on the connection from that point forward, until a subsequent `AppendRowsRequest` sets it to a different value. `missing_value_interpretation` can override `default_missing_value_interpretation`. For example, if you want to write `NULL` instead of using default values for some columns, you can set `default_missing_value_interpretation` to `DEFAULT_VALUE` and at the same time, set `missing_value_interpretations` to `NULL_VALUE` on those columns.
message
storage.proto:553Response message for `AppendRows`.
oneof response
append_result = 1
Result if the append is successful.
error = 2
Error returned when problems were encountered. If present, it indicates rows were not accepted into the system. Users can retry or continue with other append requests within the same connection. Additional information about error signalling: ALREADY_EXISTS: Happens when an append specified an offset, and the backend already has received data at this offset. Typically encountered in retry scenarios, and can be ignored. OUT_OF_RANGE: Returned when the specified offset in the stream is beyond the current end of the stream. INVALID_ARGUMENT: Indicates a malformed request or data. ABORTED: Request processing is aborted because of prior failures. The request can be retried if previous failure is addressed. INTERNAL: Indicates server side error(s) that can be retried.
optional updated_schema = 3
If backend detects a schema update, pass it to user so that user can use it to input new type of message. It will be empty when no schema updates have occurred.
repeated row_errors = 4
If a request failed due to corrupted rows, no rows in the batch will be appended. The API will return row level error info, so that the caller can remove the bad rows and retry the request.
string write_stream = 5
The target of the append operation. Matches the write_stream in the corresponding request.
rpc GetWriteStream (, )
storage.proto:181Gets information about a write stream.
message
storage.proto:604Request message for `GetWriteStreamRequest`.
string name = 1
Required. Name of the stream to get, in the form of `projects/{project}/datasets/{dataset}/tables/{table}/streams/{stream}`.
WriteStreamView view = 3
Indicates whether to get full or partial view of the WriteStream. If not set, view returned will be basic.
rpc FinalizeWriteStream (, )
storage.proto:191Finalize a write stream so that no new data can be appended to the stream. Finalize is not supported on the '_default' stream.
message
storage.proto:648Request message for invoking `FinalizeWriteStream`.
string name = 1
Required. Name of the stream to finalize, in the form of `projects/{project}/datasets/{dataset}/tables/{table}/streams/{stream}`.
message
storage.proto:660Response message for `FinalizeWriteStream`.
int64 row_count = 1
Number of rows in the finalized stream.
Atomically commits a group of `PENDING` streams that belong to the same `parent` table. Streams must be finalized before commit and cannot be committed multiple times. Once a stream is committed, data in the stream becomes available for read operations.
message
storage.proto:620Request message for `BatchCommitWriteStreams`.
string parent = 1
Required. Parent table that all the streams should belong to, in the form of `projects/{project}/datasets/{dataset}/tables/{table}`.
repeated string write_streams = 2
Required. The group of streams that will be committed atomically.
message
storage.proto:633Response message for `BatchCommitWriteStreams`.
optional commit_time = 1
The time at which streams were committed in microseconds granularity. This field will only exist when there are no stream errors. **Note** if this field is not set, it means the commit was not successful.
repeated stream_errors = 2
Stream level error if commit failed. Only streams with error will be in the list. If empty, there is no error and all streams are committed successfully. If non empty, certain streams have errors and ZERO stream is committed due to atomicity guarantee.
rpc FlushRows (, )
storage.proto:222Flushes rows to a BUFFERED stream. If users are appending rows to BUFFERED stream, flush operation is required in order for the rows to become available for reading. A Flush operation flushes up to any previously flushed offset in a BUFFERED stream, to the offset specified in the request. Flush is not supported on the _default stream, since it is not BUFFERED.
message
storage.proto:666Request message for `FlushRows`.
string write_stream = 1
Required. The stream that is the target of the flush operation.
optional offset = 2
Ending offset of the flush operation. Rows before this offset(including this offset) will be flushed.
message
storage.proto:681Respond message for `FlushRows`.
int64 offset = 1
The rows before this offset (including this offset) are flushed.
An enum to indicate how to interpret missing values of fields that are present in user schema but missing in rows. A missing value can represent a NULL or a column default value defined in BigQuery table schema.
Used in:
MISSING_VALUE_INTERPRETATION_UNSPECIFIED = 0
Invalid missing value interpretation. Requests with this value will be rejected.
NULL_VALUE = 1
Missing value is interpreted as NULL.
DEFAULT_VALUE = 2
Missing value is interpreted as column default value if declared in the table schema, NULL otherwise.
message
storage.proto:427ProtoData contains the data rows and schema when constructing append requests.
Used in:
optional writer_schema = 1
The protocol buffer schema used to serialize the data. Provide this value whenever: * You send the first request of an RPC connection. * You change the input schema. * You specify a new destination table.
optional rows = 2
Serialized row data in protobuf message format. Currently, the backend expects the serialized rows to adhere to proto2 semantics when appending rows, particularly with respect to how default values are encoded.
message
storage.proto:555AppendResult is returned for successful append requests.
Used in:
optional offset = 1
The row offset at which the last append occurred. The offset will not be set if appending using default streams.
message
arrow.proto:38Arrow RecordBatch.
Used in:
bytes serialized_record_batch = 1
IPC-serialized Arrow RecordBatch.
int64 row_count = 2
[Deprecated] The count of rows in `serialized_record_batch`. Please use the format-independent ReadRowsResponse.row_count instead.
message
arrow.proto:32Arrow schema as specified in https://arrow.apache.org/docs/python/api/datatypes.html and serialized to bytes using IPC: https://arrow.apache.org/docs/format/Columnar.html#serialization-and-interprocess-communication-ipc See code samples on how this message can be deserialized.
Used in: ,
bytes serialized_schema = 1
IPC serialized Arrow schema.
message
arrow.proto:48Contains options specific to Arrow Serialization.
Used in:
ArrowSerializationOptions.CompressionCodec buffer_compression = 2
The compression codec to use for Arrow buffers in serialized record batches.
Compression codec's supported by Arrow.
Used in:
COMPRESSION_UNSPECIFIED = 0
If unspecified no compression will be used.
LZ4_FRAME = 1
LZ4 Frame (https://github.com/lz4/lz4/blob/dev/doc/lz4_Frame_format.md)
ZSTD = 2
Zstandard compression.
message
avro.proto:34Avro rows.
Used in:
bytes serialized_binary_rows = 1
Binary serialized rows in a block.
int64 row_count = 2
[Deprecated] The count of rows in the returning block. Please use the format-independent ReadRowsResponse.row_count instead.
message
avro.proto:27Avro schema.
Used in: ,
string schema = 1
Json serialized schema, as described at https://avro.apache.org/docs/1.8.1/spec.html.
message
avro.proto:44Contains options specific to Avro Serialization.
Used in:
bool enable_display_name_attribute = 1
Enable displayName attribute in Avro schema. The Avro specification requires field names to be alphanumeric. By default, in cases when column names do not conform to these requirements (e.g. non-ascii unicode codepoints) and Avro is requested as an output format, the CreateReadSession call will fail. Setting this field to true, populates avro field names with a placeholder value and populates a "displayName" attribute for every avro field with the original column name.
Data format for input or output data.
Used in:
DATA_FORMAT_UNSPECIFIED = 0
Data format is unspecified.
AVRO = 1
Avro is a standard open source row based file format. See https://avro.apache.org/ for more details.
ARROW = 2
Arrow is a standard open source column-based message format. See https://arrow.apache.org/ for more details.
message
protobuf.proto:42Used in:
repeated bytes serialized_rows = 1
A sequence of rows serialized as a Protocol Buffer. See https://developers.google.com/protocol-buffers/docs/overview for more information on deserializing this field.
message
protobuf.proto:29ProtoSchema describes the schema of the serialized protocol buffer data rows.
Used in:
optional proto_descriptor = 1
Descriptor for input message. The provided descriptor must be self contained, such that data rows sent can be fully decoded using only the single descriptor. For data rows that are compositions of multiple independent messages, this means the descriptor may need to be transformed to only use nested types: https://developers.google.com/protocol-buffers/docs/proto#nested For additional information for how proto types and values map onto BigQuery see: https://cloud.google.com/bigquery/docs/write-api#data_type_conversions
message
stream.proto:48Information about the ReadSession.
Used as response type in: BigQueryRead.CreateReadSession
Used as field type in:
string name = 1
Output only. Unique identifier for the session, in the form `projects/{project_id}/locations/{location}/sessions/{session_id}`.
optional expire_time = 2
Output only. Time at which the session becomes invalid. After this time, subsequent requests to read this Session will return errors. The expire_time is automatically assigned and currently cannot be specified or updated.
DataFormat data_format = 3
Immutable. Data format of the output data. DATA_FORMAT_UNSPECIFIED not supported.
oneof schema
The schema for the read. If read_options.selected_fields is set, the schema may be different from the table schema as it will only contain the selected fields.
avro_schema = 4
Output only. Avro schema.
arrow_schema = 5
Output only. Arrow schema.
string table = 6
Immutable. Table that this ReadSession is reading from, in the form `projects/{project_id}/datasets/{dataset_id}/tables/{table_id}`
optional table_modifiers = 7
Optional. Any modifiers which are applied when reading from the specified table.
optional read_options = 8
Optional. Read options for this session (e.g. column selection, filters).
repeated streams = 10
Output only. A list of streams created with the session. At least one stream is created with the session. In the future, larger request_stream_count values *may* result in this list being unpopulated, in that case, the user will need to use a List method to get the streams instead, which is not yet available.
int64 estimated_total_bytes_scanned = 12
Output only. An estimate on the number of bytes this session will scan when all streams are completely consumed. This estimate is based on metadata from the table which might be incomplete or stale.
int64 estimated_total_physical_file_size = 15
Output only. A pre-projected estimate of the total physical size of files (in bytes) that this session will scan when all streams are consumed. This estimate is independent of the selected columns and can be based on incomplete or stale metadata from the table. This field is only set for BigLake tables.
int64 estimated_row_count = 14
Output only. An estimate on the number of rows present in this session's streams. This estimate is based on metadata from the table which might be incomplete or stale.
string trace_id = 13
Optional. ID set by client to annotate a session identity. This does not need to be strictly unique, but instead the same ID should be used to group logically connected sessions (e.g. All using the same ID for all sessions needed to complete a Spark SQL query is reasonable). Maximum length is 256 bytes.
message
stream.proto:55Additional attributes when reading a table.
Used in:
optional snapshot_time = 1
The snapshot time of the table. If not set, interpreted as now.
message
stream.proto:61Options dictating how we read a table.
Used in:
repeated string selected_fields = 1
Optional. The names of the fields in the table to be returned. If no field names are specified, then all fields in the table are returned. Nested fields -- the child elements of a STRUCT field -- can be selected individually using their fully-qualified names, and will be returned as record fields containing only the selected nested fields. If a STRUCT field is specified in the selected fields list, all of the child elements will be returned. As an example, consider a table with the following schema: { "name": "struct_field", "type": "RECORD", "mode": "NULLABLE", "fields": [ { "name": "string_field1", "type": "STRING", . "mode": "NULLABLE" }, { "name": "string_field2", "type": "STRING", "mode": "NULLABLE" } ] } Specifying "struct_field" in the selected fields list will result in a read session schema with the following logical structure: struct_field { string_field1 string_field2 } Specifying "struct_field.string_field1" in the selected fields list will result in a read session schema with the following logical structure: struct_field { string_field1 } The order of the fields in the read session schema is derived from the table schema and does not correspond to the order in which the fields are specified in this list.
string row_restriction = 2
SQL text filtering statement, similar to a WHERE clause in a query. Aggregates are not supported. Examples: "int_field > 5" "date_field = CAST('2014-9-27' as DATE)" "nullable_field is not NULL" "st_equals(geo_field, st_geofromtext("POINT(2, 2)"))" "numeric_field BETWEEN 1.0 AND 5.0" Restricted to a maximum length for 1 MB.
oneof output_format_serialization_options
arrow_serialization_options = 3
Optional. Options specific to the Apache Arrow output format.
avro_serialization_options = 4
Optional. Options specific to the Apache Avro output format
optional double sample_percentage = 5
Optional. Specifies a table sampling percentage. Specifically, the query planner will use TABLESAMPLE SYSTEM (sample_percentage PERCENT). The sampling percentage is applied at the data block granularity. It will randomly choose for each data block whether to read the rows in that data block. For more details, see https://cloud.google.com/bigquery/docs/table-sampling)
optional TableReadOptions.ResponseCompressionCodec response_compression_codec = 6
Optional. Set response_compression_codec when creating a read session to enable application-level compression of ReadRows responses.
Specifies which compression codec to attempt on the entire serialized response payload (either Arrow record batch or Avro rows). This is not to be confused with the Apache Arrow native compression codecs specified in ArrowSerializationOptions. For performance reasons, when creating a read session requesting Arrow responses, setting both native Arrow compression and application-level response compression will not be allowed - choose, at most, one kind of compression.
Used in:
RESPONSE_COMPRESSION_CODEC_UNSPECIFIED = 0
Default is no compression.
RESPONSE_COMPRESSION_CODEC_LZ4 = 2
Use raw LZ4 compression.
message
stream.proto:242Information about a single stream that gets data out of the storage system. Most of the information about `ReadStream` instances is aggregated, making `ReadStream` lightweight.
Used in: ,
string name = 1
Output only. Name of the stream, in the form `projects/{project_id}/locations/{location}/sessions/{session_id}/streams/{stream_id}`.
message
storage.proto:756The message that presents row level error info in a request.
Used in:
int64 index = 1
Index of the malformed row in the request.
RowError.RowErrorCode code = 2
Structured error reason for a row error.
string message = 3
Description of the issue encountered when processing the row.
Error code for `RowError`.
Used in:
ROW_ERROR_CODE_UNSPECIFIED = 0
Default error.
FIELDS_ERROR = 1
One or more fields in the row has errors.
message
storage.proto:690Structured custom BigQuery Storage error message. The error can be attached as error details in the returned rpc Status. In particular, the use of error codes allows more structured error handling, and reduces the need to evaluate unstructured error text strings.
Used in:
StorageError.StorageErrorCode code = 1
BigQuery Storage specific error code.
string entity = 2
Name of the failed entity.
string error_message = 3
Message that describes the error.
Error code for `StorageError`.
Used in:
STORAGE_ERROR_CODE_UNSPECIFIED = 0
Default error.
TABLE_NOT_FOUND = 1
Table is not found in the system.
STREAM_ALREADY_COMMITTED = 2
Stream is already committed.
STREAM_NOT_FOUND = 3
Stream is not found.
INVALID_STREAM_TYPE = 4
Invalid Stream type. For example, you try to commit a stream that is not pending.
INVALID_STREAM_STATE = 5
Invalid Stream state. For example, you try to commit a stream that is not finalized or is garbaged.
STREAM_FINALIZED = 6
Stream is finalized.
SCHEMA_MISMATCH_EXTRA_FIELDS = 7
There is a schema mismatch and it is caused by user schema has extra field than bigquery schema.
OFFSET_ALREADY_EXISTS = 8
Offset already exists.
OFFSET_OUT_OF_RANGE = 9
Offset out of range.
CMEK_NOT_PROVIDED = 10
Customer-managed encryption key (CMEK) not provided for CMEK-enabled data.
INVALID_CMEK_PROVIDED = 11
Customer-managed encryption key (CMEK) was incorrectly provided.
CMEK_ENCRYPTION_ERROR = 12
There is an encryption error while using customer-managed encryption key.
KMS_SERVICE_ERROR = 13
Key Management Service (KMS) service returned an error, which can be retried.
KMS_PERMISSION_DENIED = 14
Permission denied while using customer-managed encryption key.
message
storage.proto:292Estimated stream statistics for a given read Stream.
Used in:
optional progress = 2
Represents the progress of the current stream.
message
storage.proto:293Used in:
double at_response_start = 1
The fraction of rows assigned to the stream that have been processed by the server so far, not including the rows in the current response message. This value, along with `at_response_end`, can be used to interpolate the progress made as the rows in the message are being processed using the following formula: `at_response_start + (at_response_end - at_response_start) * rows_processed_from_response / rows_in_response`. Note that if a filter is provided, the `at_response_end` value of the previous response may not necessarily be equal to the `at_response_start` value of the current response.
double at_response_end = 2
Similar to `at_response_start`, except that this value includes the rows in the current response.
message
table.proto:37TableFieldSchema defines a single field/column within a table schema.
Used in:
string name = 1
Required. The field name. The name must contain only letters (a-z, A-Z), numbers (0-9), or underscores (_), and must start with a letter or underscore. The maximum length is 128 characters.
TableFieldSchema.Type type = 2
Required. The field data type.
TableFieldSchema.Mode mode = 3
Optional. The field mode. The default value is NULLABLE.
repeated fields = 4
Optional. Describes the nested schema fields if the type property is set to STRUCT.
string description = 6
Optional. The field description. The maximum length is 1,024 characters.
int64 max_length = 7
Optional. Maximum length of values of this field for STRINGS or BYTES. If max_length is not specified, no maximum length constraint is imposed on this field. If type = "STRING", then max_length represents the maximum UTF-8 length of strings in this field. If type = "BYTES", then max_length represents the maximum number of bytes in this field. It is invalid to set this field if type is not "STRING" or "BYTES".
int64 precision = 8
Optional. Precision (maximum number of total digits in base 10) and scale (maximum number of digits in the fractional part in base 10) constraints for values of this field for NUMERIC or BIGNUMERIC. It is invalid to set precision or scale if type is not "NUMERIC" or "BIGNUMERIC". If precision and scale are not specified, no value range constraint is imposed on this field insofar as values are permitted by the type. Values of this NUMERIC or BIGNUMERIC field must be in this range when: * Precision (P) and scale (S) are specified: [-10^(P-S) + 10^(-S), 10^(P-S) - 10^(-S)] * Precision (P) is specified but not scale (and thus scale is interpreted to be equal to zero): [-10^P + 1, 10^P - 1]. Acceptable values for precision and scale if both are specified: * If type = "NUMERIC": 1 <= precision - scale <= 29 and 0 <= scale <= 9. * If type = "BIGNUMERIC": 1 <= precision - scale <= 38 and 0 <= scale <= 38. Acceptable values for precision if only precision is specified but not scale (and thus scale is interpreted to be equal to zero): * If type = "NUMERIC": 1 <= precision <= 29. * If type = "BIGNUMERIC": 1 <= precision <= 38. If scale is specified but not precision, then it is invalid.
int64 scale = 9
Optional. See documentation for precision.
string default_value_expression = 10
Optional. A SQL expression to specify the [default value] (https://cloud.google.com/bigquery/docs/default-values) for this field.
optional range_element_type = 11
Optional. The subtype of the RANGE, if the type of this field is RANGE. If the type is RANGE, this field is required. Possible values for the field element type of a RANGE include: * DATE * DATETIME * TIMESTAMP
message
table.proto:103Represents the type of a field element.
Used in:
Type type = 1
Required. The type of a field element.
Used in:
MODE_UNSPECIFIED = 0
Illegal value
NULLABLE = 1
REQUIRED = 2
REPEATED = 3
Used in: ,
TYPE_UNSPECIFIED = 0
Illegal value
STRING = 1
64K, UTF8
INT64 = 2
64-bit signed
DOUBLE = 3
64-bit IEEE floating point
STRUCT = 4
Aggregate type
BYTES = 5
64K, Binary
BOOL = 6
2-valued
TIMESTAMP = 7
64-bit signed usec since UTC epoch
DATE = 8
Civil date - Year, Month, Day
TIME = 9
Civil time - Hour, Minute, Second, Microseconds
DATETIME = 10
Combination of civil date and civil time
GEOGRAPHY = 11
Geography object
NUMERIC = 12
Numeric value
BIGNUMERIC = 13
BigNumeric value
INTERVAL = 14
Interval
JSON = 15
JSON, String
RANGE = 16
RANGE
message
table.proto:31Schema of a table. This schema is a subset of google.cloud.bigquery.v2.TableSchema containing information necessary to generate valid message to write to BigQuery.
Used in: ,
repeated fields = 1
Describes the fields in a table.
message
storage.proto:285Information on if the current connection is being throttled.
Used in:
int32 throttle_percent = 1
How much this connection is being throttled. Zero means no throttling, 100 means fully throttled.
message
stream.proto:271Information about a single stream that gets data inside the storage system.
Used as response type in: BigQueryWrite.CreateWriteStream, BigQueryWrite.GetWriteStream
Used as field type in:
string name = 1
Output only. Name of the stream, in the form `projects/{project}/datasets/{dataset}/tables/{table}/streams/{stream}`.
WriteStream.Type type = 2
Immutable. Type of the stream.
optional create_time = 3
Output only. Create time of the stream. For the _default stream, this is the creation_time of the table.
optional commit_time = 4
Output only. Commit time of the stream. If a stream is of `COMMITTED` type, then it will have a commit_time same as `create_time`. If the stream is of `PENDING` type, empty commit_time means it is not committed.
optional table_schema = 5
Output only. The schema of the destination table. It is only returned in `CreateWriteStream` response. Caller should generate data that's compatible with this schema to send in initial `AppendRowsRequest`. The table schema could go out of date during the life time of the stream.
WriteStream.WriteMode write_mode = 7
Immutable. Mode of the stream.
string location = 8
Immutable. The geographic location where the stream's dataset resides. See https://cloud.google.com/bigquery/docs/locations for supported locations.
Type enum of the stream.
Used in:
TYPE_UNSPECIFIED = 0
Unknown type.
COMMITTED = 1
Data will commit automatically and appear as soon as the write is acknowledged.
PENDING = 2
Data is invisible until the stream is committed.
BUFFERED = 3
Data is only visible up to the offset to which it was flushed.
Mode enum of the stream.
Used in:
WRITE_MODE_UNSPECIFIED = 0
Unknown type.
INSERT = 1
Insert new records into the table. It is the default value if customers do not specify it.
WriteStreamView is a view enum that controls what details about a write stream should be returned.
Used in:
WRITE_STREAM_VIEW_UNSPECIFIED = 0
The default / unset value.
BASIC = 1
The BASIC projection returns basic metadata about a write stream. The basic view does not include schema information. This is the default view returned by GetWriteStream.
FULL = 2
The FULL projection returns all available write stream metadata, including the schema. CreateWriteStream returns the full projection of write stream metadata.