Get desktop application:
View/edit binary Protocol Buffers messages
To better permit DRY, instance.executeQuery() and scope.executeQuery() are here rather than in their respective services.
AsyncCancelHandleRequest ================== Calls the cancel function on the QueryHandle returned from startQuery().
AsyncDiscardResultsRequest ================== Calls the discard function on the QueryResultHandle.
Some modes of some SDKs will not have a QueryResult, in which case this should be surfaced in the performer caps, and the driver should not be sending it.
RPCs pertaining to server async query execution via StartQuery()
AsyncFetchStatusRequest ================== The driver is requesting the SDK call the FetchStatus request on QueryHandle. The performer should block until this (or failure) has been returned from that call. The performer must hold onto the QueryStatus object keyed by the query_handle, regardless of ResultsReady().
AsyncQueryStatusResultHandleRequest ================== Calls ResultHandle() on the QueryStatus returned from the latest AsyncFetchStatus call. The driver may call this when ResultsReady() was false, to test error handling. If the driver calls this before an AsyncFetchStatus call, this can be treated as a driver bug. On success, the performer stores the SDK's QueryResultHandle internally, keyed by query_handle.
Same as CloseQueryResultRequest except for all query_handles.
(message has no fields)
CloseQueryResultRequest ======================= Executes nothing on the SDK, and is purely for cleanup purposes. Tells the performer a query_handle is no longer needed. The performer should not cancel anything as a result of this call, or block until row iteration or metadata is complete, or similar. It may only do memory cleanup, remove the query_handle from internal collections, and do similar "tidyup". This should clean up ALL state associated with the query_handle, regardless of whether it originated from executeQuery() or startQuery(). For startQuery() handles, this includes any stored QueryHandle, QueryStatus, QueryResultHandle, and QueryResult.
ExecuteQueryRequest ================== Tells the performer it should **asynchronously** call [instance|scope].executeQuery(), immediately after it returns the ExecuteQueryResponse. To stress: the performer returns an ExecuteQueryResponse immediately, and *then* starts executing executeQuery(), in the background. So no failures from executeQuery() can be reflected in the ExecuteQueryResponse. The driver may use QueryResultRequest to check the executeQuery() result later. Asynchronicity and query_handle ------------------------------- Though not every platform has equally strong support for it, we need queries to be executed asynchronously. This is partly to allow cancelling queries at any time, given that cancellation is a key part of the Columnar PRD. So the performer will execute the query asynchronously. An SDK that is natively SDK-async will have no issue with this. An SDK-sync SDK on the other hand, will need to execute the blocking executeQuery() call in a background thread/coroutine/fibre etc. Either way, the performer holds on to any failures and results until they are requested by the driver. It associates that call and all such resources tied to it with a "query_handle" that it creates and returns in the ExecuteQueryResponse. The driver will subsequently use this handle to ask the performer to iterate, cancel, etc. The ONLY time this query_handle and its resources get destroyed, is on a CloseQueryResultRequest. The performer should not e.g. automatically discard cancellation tokens or other resources just because iteration is complete, as we want to be able to test user errors. Note the performer should never buffer more than 1 row, as we need to test slow consumers and backpressure. See QueryRowRequest for more details.
This will be used anytime the performer processes a row. It should be used for queryResult.rowsAs<T>(), or row.contentAs<T>(), or whatever is most idiomatic for the SDK. It will be sent following these rules if performer declares support for: ROW_DESERIALIZATION_STATIC_ROW_TYPING - always sent (as row streaming is only possible if a <T> is set). ROW_DESERIALIZATION_DYNAMIC_ROW_TYPING - not sent (as the Deserializer controls how the content is output). ROW_DESERIALIZATION_STATIC_ROW_TYPING_INDIVIDUAL - not sent (the equivalent field on the QueryRowRequest will be used instead).
The driver may cancel this operation, and it could be at any time before or during both the executeQuery call and the row streaming. The performer should prepare for this as needed for the SDK. For instance, if cancellation is done with a cancellation token, the performer should create it here.
QueryCancelRequest ================== The driver is emulating the user cancelling the operation. The performer should make no assumptions on if, when or how often the driver will call this. The intent is that the performer cancels whatever is happening, in the SDK-idiomatic way. That could be during the initial executeQuery(), or the subsequent row iteration, or potentially while fetching the metadata - or at any point before, between or after those operations. So the performer may end up using a range of cancellation methods depending on what phase in the query this request arrives. It is intentionally left up to the performer to decide how to interpret cancellation in each phase, as it is idiomatic for each SDK. But per the RFC - it is mandatory that cancellation be possible, in each phase. Edge cases ---------- To clarify some edge cases - if the cancellation arrives: 1. Before the performer has asynchronously started executeQuery(): the performer blocks until it has, and then cancels. 2. After the operation is completed and metadata has been fetched: if there is something to cancel (such as a cancellation token) then the SDK cancels it. Otherwise it is a no-op. (Remember that all resources associated with a given query_handle should be cleaned up ONLY in a CloseQueryResult). The performer should not discard the query_handle or associated resources after cancellation, to allow testing user error such as double-cancellation. Error handling -------------- There is no QueryCancelResponse, as per the RFC, cancellation is a fire-and-forget operation that should not fail. If it does fail (perhaps for some platform-specific reason or when testing an edge case like a user cancellation an operation twice), the error should be reported in the EmptyResultOrFailureResponse. Note this error is that reported by the cancellation itself, e.g. the `cancellationToken.cancel()` or the `queryResult.cancel()` or the `queryPromise.cancel()`, as idiomatic. It is NOT the result of the `executeQuery` call or any ongoing row iteration - those are fetched by the driver separately.
QueryMetadataRequest ==================== Equivalent to `queryResult.metadata()`. The performer should make no assumptions on if, when or how often the driver will call this, and it should not discard the query_handle and any associated resources after it is called (instead wait for CloseQueryResult). As always, the performer should model the returned QueryResultMetadataResponse based purely on the SDK's behaviour, and provide no interpretation of its own. If called by the driver before row iteration is complete then, per RFC, the SDK is expected to fast-fail. If for some reason it does not, and it blocks until metadata is available - the performer should follow suit, and not try to provide the correct behaviour to satisfy the tests.
Get the QueryResult from the executeQuery()
QueryResultRequest ================== The driver is requesting the `QueryResult` returned from the asynchronously-executing executeQuery(). The performer should block until this (or failure) has been returned from that call. Some modes of some SDKs will not have a QueryResult, in which case this should be surfaced in the performer caps, and the driver should not be sending it.
QueryRowRequest =============== The driver is emulating the user iterating through one row. It is essential that we have the ability to test slow consumers and their effect on backpressure, so the performer must not introduce any buffering of its own, and must ask for or wait for exactly one row from the SDK on each QueryRowRequest. Specifically: In a pull-based model --------------------- The performer simulates the user calling `rowIterator.next()`, or similar. If the only way to iterate rows is `for async (row in rows)` or similar, then the performer will need to block inside the for-loop repeatedly, only allowing the next row to be fetched as a QueryRowRequest comes in from the driver. In a push-based model --------------------- The performer should wait for the SDK to send the next row (and only the next row), if one hasn't been sent already. In practice this is likely to be done with a concurrent queue that can contain a maximum of exactly 1 element, and having the row callback (or equivalent) block if this queue is full. Remember - though push-based, we need to simulate the user not keeping up with the rows, e.g. taking too long inside the callback. The performer should make no assumptions on if, when or how often the driver will call this, and it should not discard the query_handle or associated resources after all rows are completed to allow testing user errors.
This can be sent iff performer declares support for ROW_DESERIALIZATION_STATIC_ROW_TYPING_INDIVIDUAL. If sent then the performer needs to deserialize the next row accordingly.
If row iteration fails, raise it here.
The executeQuery() call failed while we were waiting for a row. Only used by push-based models that don't separate QueryResult from row iteration. Most performers should not set this.
StartQueryRequest ================== Tells the performer it should **synchronously** call [instance|scope].startQuery(). Unlike ExecuteQueryRequest, this is a synchronous call that should return only once startQuery() has returned, and it should return a UUID representing the query_handle from that call. All subsequent calls from the driver that reference that query_handle.
All RPCs related to the Columnar SDK.
The performer should close all Columnar Cluster objects it has open. In the RFC this is `cluster.close()`.
The performer should close a previously-opened Columnar Cluster instance. In the RFC this is `cluster.close()`. It can safely assume that if that instance does not exist, then this is a driver bug - it does not have to handle this.
Columnar Cluster resource management.
Requests the performer use the SDK to create a connection to a Columnar cluster.
The id to use for this connection.
Details of the cluster connection. As with all fields (see rule 1 in README.md), the performer should pass all fields directly to the SDK without manipulation. E.g. it should not prepend couchbase:// to cluster_connection_string or similar.
Can apply SDK tunables that can't be represented generically in GRPC. One example is for performance testing, where we may want to experiment with various approaches in say the Java SDK. So one run may send "com.couchbase.executorMaxThreadCount"="10", and the next set it to "20". Interpretation of the tunables, and how to apply them inside the SDK, is completely SDK-dependent. The intent is that the tunables be associated with the cluster connection. Generally performers can ignore this field unless they are explicitly planning on using it.
Performer management.
(message has no fields)
Identifies the version of the Columnar SDK. For example, "1.1.2".
ColumnarService.ClusterNewInstance and everything that exists at time of commit under those messages.
ColumnarService.ClusterClose and ColumnarService.CloseAllClusters, and everything that exists at time of commit under those messages.
ColumnarCrossService.ExecuteQuery and everything that exists at time of commit under those messages, for instance.executeQuery().
ColumnarCrossService.ExecuteQuery and everything that exists at time of commit under those messages, for scope.executeQuery().
WWhether the SDK supports async queries, that is startQuery.
Updates the credential on a previously-created cluster connection. Maps to the EA RFC's `cluster.credential(newCredential)`.
Requests the performer to update the credential on an existing cluster connection.
Used in:
The SDKs for Capella Columnar
The SDKs for Enterprise Analytics and (in future) Capella Analytics
Used in:
Used in:
The result of calling ResultsReady() on the QueryStatus.
The string representation of the QueryStatus (e.g. via toString()).
Used in: ,
Authenticates using a client certificate (mTLS). The SDK authenticates the client during the TLS handshake.
Used in:
PEM-encoded X509 client certificate
PEM-encoded private key
Authenticates using a JSON Web Token (JWT). The SDK sets an "Authorization: Bearer <jwt>" header on every HTTP request.
Used in:
Used in:
Used in:
Used in:
trustOnlyPemFile is not exposed, as it will not work well with Docker-based FIT
Used in:
An exception derived from ColumnarError
Used in:
If the ColumnarError is a specific subclass exception, include the specific error. If not provided then this is to be a generic ColumnarError
Broadly this is the result of outputting the exception - what the user would see if they wrote it into their logs. E.g. for Java `ex.toString()`, for C++ `std::cout << err`. It is expected to contain the error context, per the RFC.
ContentAs ========= Where the SDK supports a rowsAs<T> or row.contentAs<T> or similar - this controls the user setting the `T`. It's only sent to performers that declare support for ROW_DESERIALIZATION_STATIC_ROW_TYPING or ROW_DESERIALIZATION_STATIC_ROW_TYPING_INDIVIDUAL. For ROW_DESERIALIZATION_DYNAMIC_ROW_TYPING performers/SDKs, T doesn't exist, and the output type is decided purely by the Deserializer.
Used in: ,
T = byte[] Intended to get the result as a byte array, and return that directly. The result is returned in ContentTypes.content_was_bytes. Pseudocode for performer: byte[] returnOverGRPC = someResult.contentAs[Array[Byte]]()
T = Map This will usually be used for JSON objects, e.g. the list items can be heterogeneous and can be any of the JSON types, including further lists or maps. The result is returned in ContentTypes.content_was_map. Pseudocode for performer: Map result = row.contentAs<Map>() or queryResult.rowsAs<Map>() byte[] returnOverGRPC = platformDependentMethodToSerializeToJSON(result)
T = List This will usually be used for JSON lists, e.g. the list items can be heterogeneous and can be any of the JSON types, including further lists or maps. The result is returned in ContentTypes.content_was_list. Pseudocode for performer: List result = row.contentAs<List>() or queryResult.rowsAs<List>() byte[] returnOverGRPC = platformDependentMethodToSerializeToJSON(result)
T = String Intended to get the result as string, and return that directly. The result is returned in ContentTypes.content_was_string. Pseudocode for performer: String result = row.contentAs<String>() or queryResult.rowsAs<String>()
"ContentWas" because this was the content the performer got from the SDK, before it serialized it up to send back to the driver.
Used in:
Used in:
Used in: , ,
Will only be sent if performer declares support for supports_passthrough_deserializer.
Will only be sent if performer declares support for supports_custom_deserializer.
* **Custom Deserializer** Uses a custom `Deserializer` implementation with specific serialization and deserialization logic. **Deserialization Logic:** ```java public <T> T deserialize(Class<T> target, byte[] input) { // Parse the byte array into a JSON object JsonObject jsonObject = parseByteArrayToJsonObject(input); // Upsert the "Serialized" field to false, "Serialized" key may or may not present while deserializing. jsonObject.put("Serialized", false); // Convert the JSON object back to the target type and return return convertJsonObjectToTargetType(jsonObject, target); } ``` **Implementation Details:** - **`parseByteArrayToJsonObject(byte[] input)`**: - Deserializes the byte array into a `JsonObject`. - **`convertJsonObjectToTargetType(JsonObject jsonObject, Class<T> target)`**: - Converts the `JsonObject` to the specified target type `T`. **Notes for Performers:** - Implementations in other languages should follow the same logical steps using equivalent constructs. - Proper error handling should be included to manage deserialization exceptions. - This deserializer assumes that the row is a JSON object. If the byte array cannot be parsed into a JSON object, then the deserializer should return/raise an error.
Used in:
(message has no fields)
Used in:
(message has no fields)
Used in:
(message has no fields)
Used as response type in: ColumnarCrossService.AsyncCancelHandle, ColumnarCrossService.AsyncDiscardResults, ColumnarCrossService.AsyncFetchResults, ColumnarCrossService.AsyncQueryStatusResultHandle, ColumnarCrossService.CloseAllQueryResults, ColumnarCrossService.CloseQueryResult, ColumnarCrossService.QueryCancel, ColumnarCrossService.QueryResult, ColumnarService.CloseAllClusters, ColumnarService.ClusterClose, ColumnarService.ClusterNewInstance, ColumnarService.SetCredential
Used in: , , , , ,
Used in:
The priority option is NOT in the Analytics SDKs, its only in the Columnar SDKs
A google.protobuf.ListValue is essentially a JSON array
A google.protobuf.Struct is essentially a JSON object
Used in:
Some metadata and config on how the SDK should execute the request and return responses.
Used in: , , ,
Which mode the SDK should run in. See `FetchPerformerCapsResponse` for an explanation.
For operations that exist at a Cluster object level.
Used in: , , ,
A previously-created Cluster. The performer can assume that if this Cluster does not exist then that is a FIT driver bug, and does not need to handle this case.
For operations that exist at a Scope object level.
Used in: ,
A previously-created Cluster. The performer can assume that if this Cluster does not exist then that is a FIT driver bug, and does not need to handle this case.
Used to represent the detailed InvalidCredentialException sub-ColumnarError type
No additional info here yet
Used in:
(message has no fields)
Currently empty.
Used in:
(message has no fields)
Used in:
The SDK implements a dispatch timeout, which is optional. The SDK may or may not implement this feature.
How the SDK exposes executeQuery().
Used in:
If the SDK includes the PassthroughDeserializer. This is likely to go hand-in-hand with ROW_DESERIALIZATION_DYNAMIC_ROW_TYPING.
Covers whether the SDK follows the "standard" model of an initial `executeQuery()` call returning a `QueryResult` that can then be iterated through.
Used in:
Should never be sent. Included as a standard protobuf best practice.
An SDK that has a "standard" model something like the following: `QueryResult result = executeQuery(...)` `PromiseLike<QueryResult> result = executeQuery(...)`
An SDK that looks something like the following (likely just used by push-based SDKs): `QueryMetadata metadata = executeQuery(...)`
Covers how/if SDKs allow users to convert rows directly into specific types.
Used in:
Should never be sent. Included as a standard protobuf best practice.
If the user can specify that rows be converted into a particular type, but only for all rows. Something like: `Iterator<UserDAO> users = queryResult.rowsAs<UserDAO>()` or `executeQuery<UserDAO>("SELECT", (row: UserDAO) => {})`
If the user can specify that rows be converted into a particular type, and can specify a different type for each row. Something like: ``` const rows = queryResult.rows() for (const row in rows) { row.contentAs<UserDAO>() } ``` or ``` executeQuery("SELECT", row => row.contentAs<UserDAO>())` ```
If rows are generally exposed however the Deserializer exposes them, with the type being decided dynamically by the Deserializer. Something like: ``` const queryResult = await executeQuery("SELECT") const rows = queryResult.rows() for await (const row of rows) { // with default Deserializer row is probably a platform-native object, like a JS object here in Node } ``` or with a custom Deserializer ``` const queryResult = await executeQuery("SELECT", queryOptions().deserializer(UserDAODeserializer)) const rows = queryResult.rows() for (const row in rows) { // row is of type UserDAO } ```
Covers how a user allows users to stream and iterate rows.
Used in:
Should never be sent. Included as a standard protobuf best practice.
If row iteration in this SDK mode looks something like (it might be a Stream or similar rather than an Iterator): ``` var iterator = queryResult.rowsAs<SomeContent>() while (iterator.hasNext()) { SomeContent row = iterator.next() } ``` or ``` var iterator = queryResult.rowsAs() while (iterator.hasNext()) { var row = iterator.next() } ```
If row iteration in this SDK mode looks something like: ``` executeQuery("SELECT...", (row) => { }) ```
All Columnar SDKs should be streaming-centric, but some may have optional modes that are fully buffered. If row iteration in this SDK mode looks something like: `List<SomeContent> rows = queryResult.rowsAs<SomeContent>()` or `List rows = queryResult.rows()` The important distinction is that the rows are fully buffered by the SDK already, by the point `executeQuery` returns.
Currently empty.
(message has no fields)
Used to represent any error that does not derive from ColumnarException/ColumnarError
Used in:
Broadly this is the result of outputting the exception - what the user would see if they wrote it into their logs. E.g. for Java `ex.toString()`.
Used in:
Should never be sent and performers do not need to handle it. Included as a standard protobuf best practice.
If it does not fall into one of the other, better categories below.
Used to represent the detailed QueryException sub-ColumnarError type
Used in:
Used to represent the detailed QueryNotFoundException sub-ColumnarError type
No additional info here yet
Used in:
(message has no fields)
Used in:
Used in:
Used in:
Used in:
A single row. It will usually be present, but is allowed to be optional to handle some niche cases, such as the end_of_stream going-beyond-the-last-row case, below.
`end_of_stream` is intended to emulate the user idiomatically discovering that there are no more rows. The "for loop" ends, the it.hasNext() returns false, the push-based ExecuteQuery returns - this sort of concept. It is allowed to return true either with the final row, or on the next row (in which csae the `row` field will be left empty). Use whichever is most idiomatic for the SDK.
Used in:
Present iff the driver requested deserialization via sending content_as.
Metadata that all responses will need.
Allows the performer to report how long an operation took.
Used in: , , ,
Clocks are hard. Many OS/platform combinations cannot guarantee nanosecond level precision of a wallclock time, but can provide such precision for elapsed time. `elapsedNanos` is intended to be, as precisely as the platform can measure it, the exact time taken by the operation in nanoseconds. There is rarely JSON processing in Columnar, and performance requirements in general are looser, so unlike Classic performers do not need to start this time just before the operation is sent into the SDK (e.g. after any JSON processing and/or options handling). They are free to start it at any reasonable point between the RPC beginning and the SDK call being made, and end is soon after the SDk call ends. This should ease performer implementation. `initiated` is a wallclock time, used to place this operation into a one second bucket. This should be as accurate as the platform can provide (often realistically this is only accurate to 10 millis or so). For failed operations: `initiated` needs to be the wallclock time the operation was initiated, not when it failed which can be potentially X seconds later. `elapsed_nanos` should be the time elapsed between `initiated` and the exception being raised.
Should never be sent. Included as a standard protobuf best practice.
Used in:
Used in:
The type of error sdk returns if cluster connection fails when user passes wrong username or password.
Enum to specify the type of bootstrap error.
Used in:
Should never be sent. Included as a standard protobuf best practice.
The SDK returns error type for COLUMNAR_TIMEOUT_EXCEPTION, when a connection attempt fails after the minimum of the three timeout expires ie connect, dispatch and query
The SDK returns an error of type COLUMNAR_ERROR when a connection attempt fast fails without any sub exception like timeout.
The SDK returns a error of some other type of topology tracking error when a connection attempt fails.
The SDK returns a error which driver can't understand
Enum to specify the type of invalid credential error.
Used in:
Should never be sent. Included as a standard protobuf best practice.
The SDK returns an error of type INVALID_CREDENTIAL_EXCEPTION when a connection attempt fails due to incorrect username or password.
The SDK returns error type for COLUMNAR_TIMEOUT_EXCEPTION, when a connection attempt fails after the minimum of the three timeout expires ie connect, dispatch and query
The SDK returns a error which driver can't understand
Used in:
Used in:
Represents the ColumnarError's sub types
Used in:
Used to represent the detailed TimeoutException sub-ColumnarError type
No additional info here yet
Used in:
(message has no fields)