package arrow.flight.protocol

Get desktop application:
View/edit binary Protocol Buffers messages

A flight service is an endpoint for retrieving or storing Arrow data. A flight service can expose one or more predefined endpoints that can be accessed using the Arrow Flight Protocol. Additionally, a flight service can expose a set of actions that are available.

• rpc Handshake (stream HandshakeRequest, stream HandshakeResponse)

  Flight.proto:42

  Handshake between client and server. Depending on the server, the handshake may be required to determine the token that should be used for future operations. Both request and response are streams to allow multiple round-trips depending on auth mechanism.

  message HandshakeRequest

  Flight.proto:151

  The request that a client provides to a server on handshake.

  • uint64 protocol_version = 1

    A defined protocol version

  • bytes payload = 2

    Arbitrary auth/handshake info.

  message HandshakeResponse

  Flight.proto:164

  • uint64 protocol_version = 1

    A defined protocol version

  • bytes payload = 2

    Arbitrary auth/handshake info.

• rpc ListFlights (Criteria, stream FlightInfo)

  Flight.proto:52

  Get a list of available streams given a particular criteria. Most flight services will expose one or more streams that are readily available for retrieval. This api allows listing the streams available for consumption. A user can also provide a criteria. The criteria can limit the subset of streams that can be listed via this interface. Each flight service allows its own definition of how to consume criteria.

  message Criteria

  Flight.proto:200

  A service specific expression that can be used to return a limited set of available Arrow Flight streams.

  • bytes expression = 1

• rpc GetFlightInfo (FlightDescriptor, FlightInfo)

  Flight.proto:66

  For a given FlightDescriptor, get information about how the flight can be consumed. This is a useful interface if the consumer of the interface already can identify the specific flight to consume. This interface can also allow a consumer to generate a flight stream through a specified descriptor. For example, a flight descriptor might be something that includes a SQL statement or a Pickled Python operation that will be executed. In those cases, the descriptor will not be previously available within the list of available streams provided by ListFlights but will be available for consumption for the duration defined by the specific flight service.

• rpc PollFlightInfo (FlightDescriptor, PollInfo)

  Flight.proto:92

  For a given FlightDescriptor, start a query and get information to poll its execution status. This is a useful interface if the query may be a long-running query. The first PollFlightInfo call should return as quickly as possible. (GetFlightInfo doesn't return until the query is complete.) A client can consume any available results before the query is completed. See PollInfo.info for details. A client can poll the updated query status by calling PollFlightInfo() with PollInfo.flight_descriptor. A server should not respond until the result would be different from last time. That way, the client can "long poll" for updates without constantly making requests. Clients can set a short timeout to avoid blocking calls if desired. A client can't use PollInfo.flight_descriptor after PollInfo.expiration_time passes. A server might not accept the retry descriptor anymore and the query may be cancelled. A client may use the CancelFlightInfo action with PollInfo.info to cancel the running query.

  message PollInfo

  Flight.proto:386

  The information to process a long-running query.

  • optional FlightInfo info = 1

    The currently available results. If "flight_descriptor" is not specified, the query is complete and "info" specifies all results. Otherwise, "info" contains partial query results. Note that each PollInfo response contains a complete FlightInfo (not just the delta between the previous and current FlightInfo). Subsequent PollInfo responses may only append new endpoints to info. Clients can begin fetching results via DoGet(Ticket) with the ticket in the info before the query is completed. FlightInfo.ordered is also valid.

  • optional FlightDescriptor flight_descriptor = 2

    The descriptor the client should use on the next try. If unset, the query is complete.

  • optional double progress = 3

    Query progress. If known, must be in [0.0, 1.0] but need not be monotonic or nondecreasing. If unknown, do not set.

  • optional google.protobuf.Timestamp expiration_time = 4

    Expiration time for this request. After this passes, the server might not accept the retry descriptor anymore (and the query may be cancelled). This may be updated on a call to PollFlightInfo.

• rpc GetSchema (FlightDescriptor, SchemaResult)

  Flight.proto:100

  For a given FlightDescriptor, get the Schema as described in Schema.fbs::Schema This is used when a consumer needs the Schema of flight stream. Similar to GetFlightInfo this interface may generate a new flight that was not previously available in ListFlights.

  message SchemaResult

  Flight.proto:270

  Wrap the result of a getSchema call

  • bytes schema = 1

    The schema of the dataset in its IPC form: 4 bytes - an optional IPC_CONTINUATION_TOKEN prefix 4 bytes - the byte length of the payload a flatbuffer Message whose header is the Schema

• rpc DoGet (Ticket, stream FlightData)

  Flight.proto:108

  Retrieve a single stream associated with a particular descriptor associated with the referenced ticket. A Flight can be composed of one or more streams where each stream can be retrieved using a separate opaque ticket that the flight service uses for managing a collection of streams.

• rpc DoPut (stream FlightData, stream PutResult)

  Flight.proto:118

  Push a stream to the flight service associated with a particular flight stream. This allows a client of a flight service to upload a stream of data. Depending on the particular flight service, a client consumer could be allowed to upload a single stream per descriptor or an unlimited number. In the latter, the service might implement a 'seal' action that can be applied to a descriptor once all streams are uploaded.

  message PutResult

  Flight.proto:526

  * The response message associated with the submission of a DoPut.

  • bytes app_metadata = 1

• rpc DoExchange (stream FlightData, stream FlightData)

  Flight.proto:127

  Open a bidirectional data channel for a given descriptor. This allows clients to send and receive arbitrary Arrow data and application-specific metadata in a single logical stream. In contrast to DoGet/DoPut, this is more suited for clients offloading computation (rather than storage) to a Flight service.

• rpc DoAction (Action, stream Result)

  Flight.proto:137

  Flight services can support an arbitrary number of simple actions in addition to the possible ListFlights, GetFlightInfo, DoGet, DoPut operations that are potentially available. DoAction allows a flight client to do a specific action against a flight service. An action includes opaque request and response objects that are specific to the type action being undertaken.

  message Action

  Flight.proto:207

  An opaque action specific for the service.

  • string type = 1

  • bytes body = 2

  message Result

  Flight.proto:233

  An opaque result returned after executing an action.

  • bytes body = 1

• rpc ListActions (Empty, stream ActionType)

  Flight.proto:144

  A flight service exposes all of the available action types that it has along with descriptions. This allows different flight consumers to understand the capabilities of the flight service.

  message Empty

  Flight.proto:185

  (message has no fields)

  message ActionType

  Flight.proto:191

  Describes an available action, including both the name used for execution along with a short description of the purpose of the action.

  • string type = 1

  • string description = 2

A message for doing simple auth.

• string username = 2

• string password = 3

The request of the CancelFlightInfo action. The request should be stored in Action.body.

• optional FlightInfo info = 1

The result of the CancelFlightInfo action. The result should be stored in Result.body.

• CancelStatus status = 1

The result of a cancel operation. This is used by CancelFlightInfoResult.status.

Used in: CancelFlightInfoResult

• CANCEL_STATUS_UNSPECIFIED = 0

  The cancellation status is unknown. Servers should avoid using this value (send a NOT_FOUND error if the requested query is not known). Clients can retry the request.

• CANCEL_STATUS_CANCELLED = 1

  The cancellation request is complete. Subsequent requests with the same payload may return CANCELLED or a NOT_FOUND error.

• CANCEL_STATUS_CANCELLING = 2

  The cancellation request is in progress. The client may retry the cancellation request.

• CANCEL_STATUS_NOT_CANCELLABLE = 3

  The query is not cancellable. The client should not retry the cancellation request.

A batch of Arrow data as part of a stream of batches.

Used as request type in: FlightService.DoExchange, FlightService.DoPut

Used as response type in: FlightService.DoExchange, FlightService.DoGet

• optional FlightDescriptor flight_descriptor = 1

  The descriptor of the data. This is only relevant when a client is starting a new DoPut stream.

• bytes data_header = 2

  Header for message data as described in Message.fbs::Message.

• bytes app_metadata = 3

  Application-defined metadata.

• bytes data_body = 1000

  The actual batch of Arrow data. Preferably handled with minimal-copies coming last in the definition to help with sidecar patterns (it is expected that some implementations will fetch this field off the wire with specialized code to avoid extra memory copies).

The name or tag for a Flight. May be used as a way to retrieve or generate a flight or be used to expose a set of previously defined flights.

Used as request type in: FlightService.GetFlightInfo, FlightService.GetSchema, FlightService.PollFlightInfo

Used as field type in: FlightData, FlightInfo, PollInfo

• FlightDescriptor.DescriptorType type = 1

• bytes cmd = 2

  Opaque value used to express a command. Should only be defined when type = CMD.

• repeated string path = 3

  List of strings identifying a particular dataset. Should only be defined when type = PATH.

Describes what type of descriptor is defined.

Used in: FlightDescriptor

• UNKNOWN = 0

  Protobuf pattern, not used.

• PATH = 1

  A named path that identifies a dataset. A path is composed of a string or list of strings describing a particular dataset. This is conceptually similar to a path inside a filesystem.

• CMD = 2

  An opaque command to generate a dataset.

A particular stream or split associated with a flight.

Used in: FlightInfo, RenewFlightEndpointRequest

• optional Ticket ticket = 1

  Token used to retrieve this stream.

• repeated Location location = 2

  A list of URIs where this ticket can be redeemed via DoGet(). If the list is empty, the expectation is that the ticket can only be redeemed on the current service where the ticket was generated. If the list is not empty, the expectation is that the ticket can be redeemed at any of the locations, and that the data returned will be equivalent. In this case, the ticket may only be redeemed at one of the given locations, and not (necessarily) on the current service. In other words, an application can use multiple locations to represent redundant and/or load balanced services.

• optional google.protobuf.Timestamp expiration_time = 3

  Expiration time of this stream. If present, clients may assume they can retry DoGet requests. Otherwise, it is application-defined whether DoGet requests may be retried.

• bytes app_metadata = 4

  Application-defined metadata. There is no inherent or required relationship between this and the app_metadata fields in the FlightInfo or resulting FlightData messages. Since this metadata is application-defined, a given application could define there to be a relationship, but there is none required by the spec.

The access coordinates for retrieval of a dataset. With a FlightInfo, a consumer is able to determine how to retrieve a dataset.

Used as response type in: FlightService.GetFlightInfo, FlightService.ListFlights

Used as field type in: CancelFlightInfoRequest, PollInfo

• bytes schema = 1

  The schema of the dataset in its IPC form: 4 bytes - an optional IPC_CONTINUATION_TOKEN prefix 4 bytes - the byte length of the payload a flatbuffer Message whose header is the Schema

• optional FlightDescriptor flight_descriptor = 2

  The descriptor associated with this info.

• repeated FlightEndpoint endpoint = 3

  A list of endpoints associated with the flight. To consume the whole flight, all endpoints (and hence all Tickets) must be consumed. Endpoints can be consumed in any order. In other words, an application can use multiple endpoints to represent partitioned data. If the returned data has an ordering, an application can use "FlightInfo.ordered = true" or should return the all data in a single endpoint. Otherwise, there is no ordering defined on endpoints or the data within. A client can read ordered data by reading data from returned endpoints, in order, from front to back. Note that a client may ignore "FlightInfo.ordered = true". If an ordering is important for an application, an application must choose one of them: * An application requires that all clients must read data in returned endpoints order. * An application must return the all data in a single endpoint.

• int64 total_records = 4

  Set these to -1 if unknown.

• int64 total_bytes = 5

• bool ordered = 6

  FlightEndpoints are in the same order as the data.

• bytes app_metadata = 7

  Application-defined metadata. There is no inherent or required relationship between this and the app_metadata fields in the FlightEndpoints or resulting FlightData messages. Since this metadata is application-defined, a given application could define there to be a relationship, but there is none required by the spec.

A location where a Flight service will accept retrieval of a particular stream given a ticket.

Used in: FlightEndpoint

• string uri = 1

The request of the RenewFlightEndpoint action. The request should be stored in Action.body.

• optional FlightEndpoint endpoint = 1

An opaque identifier that the service can use to retrieve a particular portion of a stream. Tickets are meant to be single use. It is an error/application-defined behavior to reuse a ticket.

Used as request type in: FlightService.DoGet

Used as field type in: FlightEndpoint

• bytes ticket = 1