package arrow.flight.protocol

Get desktop application:
View/edit binary Protocol Buffers messages

A flight service is an endpoint for retrieving or storing Arrow data. A flight service can expose one or more predefined endpoints that can be accessed using the Arrow Flight Protocol. Additionally, a flight service can expose a set of actions that are available.

• rpc DoAction (Action, stream Result)

  Flight.proto:137

  Flight services can support an arbitrary number of simple actions in addition to the possible ListFlights, GetFlightInfo, DoGet, DoPut operations that are potentially available. DoAction allows a flight client to do a specific action against a flight service. An action includes opaque request and response objects that are specific to the type action being undertaken.

  message Action

  Flight.proto:206

  An opaque action specific for the service.

  • string type = 1

  • bytes body = 2

  message Result

  Flight.proto:214

  An opaque result returned after executing an action.

  • bytes body = 1

• rpc DoExchange (stream FlightData, stream FlightData)

  Flight.proto:127

  Open a bidirectional data channel for a given descriptor. This allows clients to send and receive arbitrary Arrow data and application-specific metadata in a single logical stream. In contrast to DoGet/DoPut, this is more suited for clients offloading computation (rather than storage) to a Flight service.

• rpc DoGet (Ticket, stream FlightData)

  Flight.proto:108

  Retrieve a single stream associated with a particular descriptor associated with the referenced ticket. A Flight can be composed of one or more streams where each stream can be retrieved using a separate opaque ticket that the flight service uses for managing a collection of streams.

• rpc DoPut (stream FlightData, stream PutResult)

  Flight.proto:118

  Push a stream to the flight service associated with a particular flight stream. This allows a client of a flight service to upload a stream of data. Depending on the particular flight service, a client consumer could be allowed to upload a single stream per descriptor or an unlimited number. In the latter, the service might implement a 'seal' action that can be applied to a descriptor once all streams are uploaded.

• rpc GetFlightInfo (FlightDescriptor, FlightInfo)

  Flight.proto:66

  For a given FlightDescriptor, get information about how the flight can be consumed. This is a useful interface if the consumer of the interface already can identify the specific flight to consume. This interface can also allow a consumer to generate a flight stream through a specified descriptor. For example, a flight descriptor might be something that includes a SQL statement or a Pickled Python operation that will be executed. In those cases, the descriptor will not be previously available within the list of available streams provided by ListFlights but will be available for consumption for the duration defined by the specific flight service.

• rpc GetSchema (FlightDescriptor, SchemaResult)

  Flight.proto:100

  For a given FlightDescriptor, get the Schema as described in Schema.fbs::Schema This is used when a consumer needs the Schema of flight stream. Similar to GetFlightInfo this interface may generate a new flight that was not previously available in ListFlights.

  message SchemaResult

  Flight.proto:221

  Wrap the result of a getSchema call

  • bytes schema = 1

    The schema of the dataset in its IPC form: 4 bytes - an optional IPC_CONTINUATION_TOKEN prefix 4 bytes - the byte length of the payload a flatbuffer Message whose header is the Schema

• rpc Handshake (stream HandshakeRequest, stream HandshakeResponse)

  Flight.proto:42

  Handshake between client and server. Depending on the server, the handshake may be required to determine the token that should be used for future operations. Both request and response are streams to allow multiple round-trips depending on auth mechanism.

• rpc ListActions (Empty, stream ActionType)

  Flight.proto:144

  A flight service exposes all of the available action types that it has along with descriptions. This allows different flight consumers to understand the capabilities of the flight service.

  message Empty

  Flight.proto:184

  (message has no fields)

  message ActionType

  Flight.proto:190

  Describes an available action, including both the name used for execution along with a short description of the purpose of the action.

  • string type = 1

  • string description = 2

• rpc ListFlights (Criteria, stream FlightInfo)

  Flight.proto:52

  Get a list of available streams given a particular criteria. Most flight services will expose one or more streams that are readily available for retrieval. This api allows listing the streams available for consumption. A user can also provide a criteria. The criteria can limit the subset of streams that can be listed via this interface. Each flight service allows its own definition of how to consume criteria.

  message Criteria

  Flight.proto:199

  A service specific expression that can be used to return a limited set of available Arrow Flight streams.

  • bytes expression = 1

• rpc PollFlightInfo (FlightDescriptor, PollInfo)

  Flight.proto:92

  For a given FlightDescriptor, start a query and get information to poll its execution status. This is a useful interface if the query may be a long-running query. The first PollFlightInfo call should return as quickly as possible. (GetFlightInfo doesn't return until the query is complete.) A client can consume any available results before the query is completed. See PollInfo.info for details. A client can poll the updated query status by calling PollFlightInfo() with PollInfo.flight_descriptor. A server should not respond until the result would be different from last time. That way, the client can "long poll" for updates without constantly making requests. Clients can set a short timeout to avoid blocking calls if desired. A client can't use PollInfo.flight_descriptor after PollInfo.expiration_time passes. A server might not accept the retry descriptor anymore and the query may be cancelled. A client may use the CancelFlightInfo action with PollInfo.info to cancel the running query.

  message PollInfo

  Flight.proto:337

  The information to process a long-running query.

  • optional FlightInfo info = 1

    The currently available results. If "flight_descriptor" is not specified, the query is complete and "info" specifies all results. Otherwise, "info" contains partial query results. Note that each PollInfo response contains a complete FlightInfo (not just the delta between the previous and current FlightInfo). Subsequent PollInfo responses may only append new endpoints to info. Clients can begin fetching results via DoGet(Ticket) with the ticket in the info before the query is completed. FlightInfo.ordered is also valid.

  • optional FlightDescriptor flight_descriptor = 2

    The descriptor the client should use on the next try. If unset, the query is complete.

  • optional double progress = 3

    Query progress. If known, must be in [0.0, 1.0] but need not be monotonic or nondecreasing. If unknown, do not set.

  • optional google.protobuf.Timestamp expiration_time = 4

    Expiration time for this request. After this passes, the server might not accept the retry descriptor anymore (and the query may be cancelled). This may be updated on a call to PollFlightInfo.

A message for doing simple auth.

• string username = 2

• string password = 3

The request of the CancelFlightInfo action. The request should be stored in Action.body.

• optional FlightInfo info = 1

The result of the CancelFlightInfo action. The result should be stored in Result.body.

• CancelStatus status = 1

The result of a cancel operation. This is used by CancelFlightInfoResult.status.

Used in: CancelFlightInfoResult

• CANCEL_STATUS_UNSPECIFIED = 0

  The cancellation status is unknown. Servers should avoid using this value (send a NOT_FOUND error if the requested query is not known). Clients can retry the request.

• CANCEL_STATUS_CANCELLED = 1

  The cancellation request is complete. Subsequent requests with the same payload may return CANCELLED or a NOT_FOUND error.

• CANCEL_STATUS_CANCELLING = 2

  The cancellation request is in progress. The client may retry the cancellation request.

• CANCEL_STATUS_NOT_CANCELLABLE = 3

  The query is not cancellable. The client should not retry the cancellation request.

Request message for the "Close Session" action. The exiting session is referenced via a cookie header.

(message has no fields)

The result of closing a session.

• CloseSessionResult.Status status = 1

Used in: CloseSessionResult

• UNSPECIFIED = 0

  Protobuf deserialization fallback value: The session close status is unknown or not recognized. Servers should avoid using this value (send a NOT_FOUND error if the requested session is not known or expired). Clients can retry the request.

• CLOSED = 1

  The session close request is complete. Subsequent requests with the same session produce a NOT_FOUND error.

• CLOSING = 2

  The session close request is in progress. The client may retry the close request.

• NOT_CLOSEABLE = 3

  The session is not closeable. The client should not retry the close request.

A batch of Arrow data as part of a stream of batches.

Used as request type in: FlightService.DoExchange, FlightService.DoPut, io.deephaven.flightjs.protocol.BrowserFlightService.NextDoExchange, io.deephaven.flightjs.protocol.BrowserFlightService.NextDoPut, io.deephaven.flightjs.protocol.BrowserFlightService.OpenDoExchange, io.deephaven.flightjs.protocol.BrowserFlightService.OpenDoPut

Used as response type in: FlightService.DoExchange, FlightService.DoGet, io.deephaven.flightjs.protocol.BrowserFlightService.OpenDoExchange

• optional FlightDescriptor flight_descriptor = 1

  The descriptor of the data. This is only relevant when a client is starting a new DoPut stream.

• bytes data_header = 2

  Header for message data as described in Message.fbs::Message.

• bytes app_metadata = 3

  Application-defined metadata.

• bytes data_body = 1000

  The actual batch of Arrow data. Preferably handled with minimal-copies coming last in the definition to help with sidecar patterns (it is expected that some implementations will fetch this field off the wire with specialized code to avoid extra memory copies).

The name or tag for a Flight. May be used as a way to retrieve or generate a flight or be used to expose a set of previously defined flights.

Used as request type in: FlightService.GetFlightInfo, FlightService.GetSchema, FlightService.PollFlightInfo

Used as field type in: FlightData, FlightInfo, PollInfo

• FlightDescriptor.DescriptorType type = 1

• bytes cmd = 2

  Opaque value used to express a command. Should only be defined when type = CMD.

• repeated string path = 3

  List of strings identifying a particular dataset. Should only be defined when type = PATH.

Describes what type of descriptor is defined.

Used in: FlightDescriptor

• UNKNOWN = 0

  Protobuf pattern, not used.

• PATH = 1

  A named path that identifies a dataset. A path is composed of a string or list of strings describing a particular dataset. This is conceptually similar to a path inside a filesystem.

• CMD = 2

  An opaque command to generate a dataset.

A particular stream or split associated with a flight.

Used in: FlightInfo, RenewFlightEndpointRequest

• optional Ticket ticket = 1

  Token used to retrieve this stream.

• repeated Location location = 2

  A list of URIs where this ticket can be redeemed via DoGet(). If the list is empty, the expectation is that the ticket can only be redeemed on the current service where the ticket was generated. If the list is not empty, the expectation is that the ticket can be redeemed at any of the locations, and that the data returned will be equivalent. In this case, the ticket may only be redeemed at one of the given locations, and not (necessarily) on the current service. If one of the given locations is "arrow-flight-reuse-connection://?", the client may redeem the ticket on the service where the ticket was generated (i.e., the same as above), in addition to the other locations. (This URI was chosen to maximize compatibility, as 'scheme:' or 'scheme://' are not accepted by Java's java.net.URI.) In other words, an application can use multiple locations to represent redundant and/or load balanced services.

• optional google.protobuf.Timestamp expiration_time = 3

  Expiration time of this stream. If present, clients may assume they can retry DoGet requests. Otherwise, it is application-defined whether DoGet requests may be retried.

• bytes app_metadata = 4

  Application-defined metadata. There is no inherent or required relationship between this and the app_metadata fields in the FlightInfo or resulting FlightData messages. Since this metadata is application-defined, a given application could define there to be a relationship, but there is none required by the spec.

The access coordinates for retrieval of a dataset. With a FlightInfo, a consumer is able to determine how to retrieve a dataset.

Used as response type in: FlightService.GetFlightInfo, FlightService.ListFlights

Used as field type in: CancelFlightInfoRequest, PollInfo

• bytes schema = 1

  The schema of the dataset in its IPC form: 4 bytes - an optional IPC_CONTINUATION_TOKEN prefix 4 bytes - the byte length of the payload a flatbuffer Message whose header is the Schema

• optional FlightDescriptor flight_descriptor = 2

  The descriptor associated with this info.

• repeated FlightEndpoint endpoint = 3

  A list of endpoints associated with the flight. To consume the whole flight, all endpoints (and hence all Tickets) must be consumed. Endpoints can be consumed in any order. In other words, an application can use multiple endpoints to represent partitioned data. If the returned data has an ordering, an application can use "FlightInfo.ordered = true" or should return the all data in a single endpoint. Otherwise, there is no ordering defined on endpoints or the data within. A client can read ordered data by reading data from returned endpoints, in order, from front to back. Note that a client may ignore "FlightInfo.ordered = true". If an ordering is important for an application, an application must choose one of them: * An application requires that all clients must read data in returned endpoints order. * An application must return the all data in a single endpoint.

• int64 total_records = 4

  Set these to -1 if unknown.

• int64 total_bytes = 5

• bool ordered = 6

  FlightEndpoints are in the same order as the data.

• bytes app_metadata = 7

  Application-defined metadata. There is no inherent or required relationship between this and the app_metadata fields in the FlightEndpoints or resulting FlightData messages. Since this metadata is application-defined, a given application could define there to be a relationship, but there is none required by the spec.

EXPERIMENTAL: A request to access the session options for the current server session. The existing session is referenced via a cookie header or similar (see SetSessionOptionsRequest above); it is an error to make this request with a missing, invalid, or expired session cookie header or other implementation-defined session reference token.

(message has no fields)

EXPERIMENTAL: The result containing the current server session options.

• map<string, SessionOptionValue> session_options = 1

The request that a client provides to a server on handshake.

Used as request type in: FlightService.Handshake, io.deephaven.flightjs.protocol.BrowserFlightService.NextHandshake, io.deephaven.flightjs.protocol.BrowserFlightService.OpenHandshake

• uint64 protocol_version = 1

  A defined protocol version

• bytes payload = 2

  Arbitrary auth/handshake info.

Used as response type in: FlightService.Handshake, io.deephaven.flightjs.protocol.BrowserFlightService.OpenHandshake

• uint64 protocol_version = 1

  A defined protocol version

• bytes payload = 2

  Arbitrary auth/handshake info.

A location where a Flight service will accept retrieval of a particular stream given a ticket.

Used in: FlightEndpoint

• string uri = 1

* The response message associated with the submission of a DoPut.

Used as response type in: FlightService.DoPut, io.deephaven.flightjs.protocol.BrowserFlightService.OpenDoPut

• bytes app_metadata = 1

The request of the RenewFlightEndpoint action. The request should be stored in Action.body.

• optional FlightEndpoint endpoint = 1

EXPERIMENTAL: Union of possible value types for a Session Option to be set to. By convention, an attempt to set a valueless SessionOptionValue should attempt to unset or clear the named option value on the server.

Used in: GetSessionOptionsResult, SetSessionOptionsRequest

• oneof option_value

  • string string_value = 1

  • bool bool_value = 2

  • sfixed64 int64_value = 3

  • double double_value = 4

  • SessionOptionValue.StringListValue string_list_value = 5

Used in: SessionOptionValue

• repeated string values = 1

EXPERIMENTAL: A request to set session options for an existing or new (implicit) server session. Sessions are persisted and referenced via a transport-level state management, typically RFC 6265 HTTP cookies when using an HTTP transport. The suggested cookie name or state context key is 'arrow_flight_session_id', although implementations may freely choose their own name. Session creation (if one does not already exist) is implied by this RPC request, however server implementations may choose to initiate a session that also contains client-provided session options at any other time, e.g. on authentication, or when any other call is made and the server wishes to use a session to persist any state (or lack thereof).

• map<string, SessionOptionValue> session_options = 1

EXPERIMENTAL: The results (individually) of setting a set of session options. Option names should only be present in the response if they were not successfully set on the server; that is, a response without an Error for a name provided in the SetSessionOptionsRequest implies that the named option value was set successfully.

• map<string, SetSessionOptionsResult.Error> errors = 1

Used in: SetSessionOptionsResult

• ErrorValue value = 1

Used in: Error

• UNSPECIFIED = 0

  Protobuf deserialization fallback value: The status is unknown or unrecognized. Servers should avoid using this value. The request may be retried by the client.

• INVALID_NAME = 1

  The given session option name is invalid.

• INVALID_VALUE = 2

  The session option value or type is invalid.

• ERROR = 3

  The session option cannot be set.

An opaque identifier that the service can use to retrieve a particular portion of a stream. Tickets are meant to be single use. It is an error/application-defined behavior to reuse a ticket.

Used as request type in: FlightService.DoGet

Used as field type in: FlightEndpoint

• bytes ticket = 1