package tensorflow.serving

Get desktop application:
View/edit binary Protocol Buffers messages

ModelService provides methods to query and update the state of the server, e.g. which models/versions are being served.

• rpc GetModelStatus (GetModelStatusRequest, GetModelStatusResponse)

  model_service.proto:17

  Gets status of model. If the ModelSpec in the request does not specify version, information about all versions of the model will be returned. If the ModelSpec in the request does specify a version, the status of only that version will be returned.

  message GetModelStatusRequest

  get_model_status.proto:12

  GetModelStatusRequest contains a ModelSpec indicating the model for which to get status.

  • optional ModelSpec model_spec = 1

    Model Specification. If version is not specified, information about all versions of the model will be returned. If a version is specified, the status of only that version will be returned.

  message GetModelStatusResponse

  get_model_status.proto:64

  Response for ModelStatusRequest on successful run.

  • repeated ModelVersionStatus model_version_status = 1

    Version number and status information for applicable model version(s).

• rpc HandleReloadConfigRequest (ReloadConfigRequest, ReloadConfigResponse)

  model_service.proto:22

  Reloads the set of served models. The new config supersedes the old one, so if a model is omitted from the new config it will be unloaded and no longer served.

  message ReloadConfigRequest

  model_management.proto:9

  • optional ModelServerConfig config = 1

  message ReloadConfigResponse

  model_management.proto:13

  • optional StatusProto status = 1

open source marker; do not remove PredictionService provides access to machine-learned models loaded by model_servers.

• rpc Classify (ClassificationRequest, ClassificationResponse)

  prediction_service.proto:17

  Classify.

• rpc GetModelMetadata (GetModelMetadataRequest, GetModelMetadataResponse)

  prediction_service.proto:29

  GetModelMetadata - provides access to metadata for loaded models.

  message GetModelMetadataRequest

  get_model_metadata.proto:15

  • optional ModelSpec model_spec = 1

    Model Specification indicating which model we are querying for metadata. If version is not specified, will use the latest (numerical) version.

  • repeated string metadata_field = 2

    Metadata fields to get. Currently supported: "signature_def".

  message GetModelMetadataResponse

  get_model_metadata.proto:23

  • optional ModelSpec model_spec = 1

    Model Specification indicating which model this metadata belongs to.

  • map<string, google.protobuf.Any> metadata = 2

    Map of metadata field name to metadata field. The options for metadata field name are listed in GetModelMetadataRequest. Currently supported: "signature_def".

• rpc MultiInference (MultiInferenceRequest, MultiInferenceResponse)

  prediction_service.proto:26

  MultiInference API for multi-headed models.

• rpc Predict (PredictRequest, PredictResponse)

  prediction_service.proto:23

  Predict -- provides access to loaded TensorFlow model.

• rpc Regress (RegressionRequest, RegressionResponse)

  prediction_service.proto:20

  Regress.

SessionService defines a service with which a client can interact to execute Tensorflow model inference. The SessionService::SessionRun method is similar to MasterService::RunStep of Tensorflow, except that all sessions are ready to run, and you request a specific model/session with ModelSpec.

• rpc SessionRun (SessionRunRequest, SessionRunResponse)

  session_service.proto:50

  Runs inference of a given model.

Batching parameters. Each individual parameter is optional. If omitted, the default value from the relevant batching config struct (SharedBatchScheduler ::Options or BatchSchedulerRetrier::Options) is used.

SharedBatchScheduler options (see shared_batch_scheduler.h):

Used in: SessionBundleConfig

• optional google.protobuf.Int64Value max_batch_size = 1

  The maximum size of each batch. IMPORTANT: As discussed above, use 'max_batch_size * 2' client threads to achieve high throughput with batching.

• optional google.protobuf.Int64Value batch_timeout_micros = 2

  If a task has been enqueued for this amount of time (in microseconds), and a thread is available, the scheduler will immediately form a batch from enqueued tasks and assign the batch to the thread for processing, even if the batch's size is below 'max_batch_size'.

• optional google.protobuf.Int64Value max_enqueued_batches = 3

  The maximum length of the queue, in terms of the number of batches. (A batch that has been scheduled on a thread is considered to have been removed from the queue.)

• optional google.protobuf.Int64Value num_batch_threads = 4

  The number of threads to use to process batches. Must be >= 1, and should be tuned carefully.

• optional google.protobuf.StringValue thread_pool_name = 5

  The name to use for the pool of batch threads.

• repeated int64 allowed_batch_sizes = 6

  The allowed batch sizes. (Ignored if left empty.) Requirements: - The entries must be in increasing order. - The final entry must equal 'max_batch_size'.

• bool pad_variable_length_inputs = 7

  Whether to pad variable-length inputs when a batch is formed.

A single class.

Used in: Classifications

• string label = 1

  Label or name of the class.

• float score = 2

  Score for this class (e.g., the probability the item belongs to this class). As per the proto3 default-value semantics, if the score is missing, it should be treated as 0.

Used as request type in: PredictionService.Classify

Used as field type in: ClassifyLog

• optional ModelSpec model_spec = 1

  Model Specification. If version is not specified, will use the latest (numerical) version.

• optional Input input = 2

  Input data.

Used as response type in: PredictionService.Classify

Used as field type in: ClassifyLog

• optional ModelSpec model_spec = 2

  Effective Model Specification used for classification.

• optional ClassificationResult result = 1

  Result of the classification.

Contains one result per input example, in the same order as the input in ClassificationRequest.

Used in: ClassificationResponse, InferenceResult

• repeated Classifications classifications = 1

List of classes for a single item (tensorflow.Example).

Used in: ClassificationResult

• repeated Class classes = 1

Used in: PredictionLog

• optional ClassificationRequest request = 1

• optional ClassificationResponse response = 2

• string string_field = 1

• string string_field = 1

Specifies one or more fully independent input Examples. See examples at: https://github.com/tensorflow/tensorflow/blob/master/tensorflow/core/example/example.proto

Used in: Input

• repeated Example examples = 1

Specifies one or more independent input Examples, with a common context Example. The common use case for context is to cleanly and optimally specify some features that are common across multiple examples. See example below with a search query as the context and multiple restaurants to perform some inference on. context: { feature: { key : "query" value: { bytes_list: { value: [ "pizza" ] } } } } examples: { feature: { key : "cuisine" value: { bytes_list: { value: [ "Pizzeria" ] } } } } examples: { feature: { key : "cuisine" value: { bytes_list: { value: [ "Taqueria" ] } } } } Implementations of ExampleListWithContext merge the context Example into each of the Examples. Note that feature keys must not be duplicated between the Examples and context Example, or the behavior is undefined. See also: tensorflow/core/example/example.proto https://developers.google.com/protocol-buffers/docs/proto3#maps

Used in: Input

• repeated Example examples = 1

• optional Example context = 2

Config proto for FileSystemStoragePathSource.

• repeated FileSystemStoragePathSourceConfig.ServableToMonitor servables = 5

  The servables to monitor for new versions, and aspire.

• string servable_name = 1

  A single servable name/base_path pair to monitor. DEPRECATED: Use 'servables' instead. TODO(b/30898016): Stop using these fields, and ultimately remove them here.

• string base_path = 2

• int64 file_system_poll_wait_seconds = 3

  How long to wait between file-system polling to look for children of 'base_path', in seconds. For testing use only: a negative value disables the polling thread.

• bool fail_if_zero_versions_at_startup = 4

  If true, then FileSystemStoragePathSource::Create() and ::UpdateConfig() fail if, for any configured servables, the file system doesn't currently contain at least one version under the base path. (Otherwise, it will emit a warning and keep pinging the file system to check for a version to appear later.)

A servable name and base path to look for versions of the servable.

Used in: FileSystemStoragePathSourceConfig

• string servable_name = 1

  The servable name to supply in aspired-versions callback calls. Child paths of 'base_path' are considered to be versions of this servable.

• string base_path = 2

  The path to monitor, i.e. look for child paths of the form base_path/123.

• optional ServableVersionPolicy servable_version_policy = 4

  The policy to determines the number of versions of the servable to be served at the same time.

A policy that dictates which version(s) of a servable should be served.

Used in: ServableToMonitor, ModelConfig

• oneof policy_choice

  • ServableVersionPolicy.Latest latest = 100

  • ServableVersionPolicy.All all = 101

  • ServableVersionPolicy.Specific specific = 102

Serve all versions found on disk.

Used in: ServableVersionPolicy

(message has no fields)

Serve the latest versions (i.e. the ones with the highest version numbers), among those found on disk. This is the default policy, with the default number of versions as 1.

Used in: ServableVersionPolicy

• uint32 num_versions = 1

  Number of latest versions to serve. (The default is 1.)

Serve a specific version (or set of versions). This policy is useful for rolling back to a specific version, or for canarying a specific version while still serving a separate stable version.

Used in: ServableVersionPolicy

• repeated int64 versions = 1

  The version numbers to serve.

Config proto for HashmapSourceAdapter.

• HashmapSourceAdapterConfig.Format format = 1

The format used by the file containing a serialized hashmap.

Used in: HashmapSourceAdapterConfig

• SIMPLE_CSV = 0

  A simple kind of CSV text file of the form: key0,value0\n key1,value1\n ...

Inference result, matches the type of request or is an error.

Used in: MultiInferenceResponse

• optional ModelSpec model_spec = 1

• oneof result

  • ClassificationResult classification_result = 2

  • RegressionResult regression_result = 3

Inference request such as classification, regression, etc...

Used in: MultiInferenceRequest

• optional ModelSpec model_spec = 1

  Model Specification. If version is not specified, will use the latest (numerical) version. All ModelSpecs in a MultiInferenceRequest must access the same model name.

• string method_name = 2

  Signature's method_name. Should be one of the method names defined in third_party/tensorflow/python/saved_model/signature_constants.py. e.g. "tensorflow/serving/classify".

Used in: ClassificationRequest, MultiInferenceRequest, RegressionRequest

• oneof kind

  • ExampleList example_list = 1

  • ExampleListWithContext example_list_with_context = 2

Used in: LoggingConfig

• string type = 1

  Identifies the type of the LogCollector we will use to collect these logs.

• string filename_prefix = 2

  The prefix to use for the filenames of the logs.

Metadata logged along with the request logs.

Used in: PredictionLog

• optional ModelSpec model_spec = 1

• optional SamplingConfig sampling_config = 2

  TODO(b/33279154): Add more metadata as mentioned in the bug.

Configuration for logging query/responses.

Used in: ModelConfig

• optional LogCollectorConfig log_collector_config = 1

• optional SamplingConfig sampling_config = 2

• optional google.protobuf.Any any_field = 1

Common configuration for loading a model being served.

Used in: ModelConfigList

• string name = 1

  Name of the model.

• string base_path = 2

  Base path to the model, excluding the version directory. E.g> for a model at /foo/bar/my_model/123, where 123 is the version, the base path is /foo/bar/my_model. (This can be changed once a model is in serving, *if* the underlying data remains the same. Otherwise there are no guarantees about whether the old or new data will be used for model versions currently loaded.)

• ModelType model_type = 3

  Type of model. TODO(b/31336131): DEPRECATED. Please use 'model_platform' instead.

• string model_platform = 4

  Type of model (e.g. "tensorflow"). (This cannot be changed once a model is in serving.)

• optional FileSystemStoragePathSourceConfig.ServableVersionPolicy model_version_policy = 7

  Version policy for the model indicating which version(s) of the model to load and make available for serving simultaneously. The default option is to serve only the latest version of the model. (This can be changed once a model is in serving.)

• map<string, int64> version_labels = 8

  String labels to associate with versions of the model, allowing inference queries to refer to versions by label instead of number. Multiple labels can map to the same version, but not vice-versa. An envisioned use-case for these labels is canarying tentative versions. For example, one can assign labels "stable" and "canary" to two specific versions. Perhaps initially "stable" is assigned to version 0 and "canary" to version 1. Once version 1 passes canary, one can shift the "stable" label to refer to version 1 (at that point both labels map to the same version -- version 1 -- which is fine). Later once version 2 is ready to canary one can move the "canary" label to version 2. And so on.

• optional LoggingConfig logging_config = 6

  Configures logging requests and responses, to the model. (This can be changed once a model is in serving.)

Static list of models to be loaded for serving.

Used in: ModelServerConfig

• repeated ModelConfig config = 1

ModelServer config.

Used in: ReloadConfigRequest

• oneof config

  ModelServer takes either a static file-based model config list or an Any proto representing custom model config that is fetched dynamically at runtime (through network RPC, custom service, etc.).

  • ModelConfigList model_config_list = 1

  • google.protobuf.Any custom_model_config = 2

Metadata for an inference request such as the model name and version.

Used in: ClassificationRequest, ClassificationResponse, GetModelMetadataRequest, GetModelMetadataResponse, GetModelStatusRequest, InferenceResult, InferenceTask, LogMetadata, PredictRequest, PredictResponse, RegressionRequest, RegressionResponse, SessionRunRequest, SessionRunResponse

• string name = 1

  Required servable name.

• oneof version_choice

  Optional choice of which version of the model to use. Recommended to be left unset in the common case. Should be specified only when there is a strong version consistency requirement. When left unspecified, the system will serve the best available version. This is typically the latest version, though during version transitions, notably when serving on a fleet of instances, may be either the previous or new version.

  • google.protobuf.Int64Value version = 2

    Use this specific version number.

  • string version_label = 4

    Use the version associated with the given label.

• string signature_name = 3

  A named signature to evaluate. If unspecified, the default signature will be used.

The type of model. TODO(b/31336131): DEPRECATED.

Used in: ModelConfig

• MODEL_TYPE_UNSPECIFIED = 0

• TENSORFLOW = 1

• OTHER = 2

Version number, state, and status for a single version of a model.

Used in: GetModelStatusResponse

• int64 version = 1

  Model version.

• ModelVersionStatus.State state = 2

  Model state.

• optional StatusProto status = 3

  Model status.

States that map to ManagerState enum in tensorflow_serving/core/servable_state.h

Used in: ModelVersionStatus

• UNKNOWN = 0

  Default value.

• START = 10

  The manager is tracking this servable, but has not initiated any action pertaining to it.

• LOADING = 20

  The manager has decided to load this servable. In particular, checks around resource availability and other aspects have passed, and the manager is about to invoke the loader's Load() method.

• AVAILABLE = 30

  The manager has successfully loaded this servable and made it available for serving (i.e. GetServableHandle(id) will succeed). To avoid races, this state is not reported until *after* the servable is made available.

• UNLOADING = 40

  The manager has decided to make this servable unavailable, and unload it. To avoid races, this state is reported *before* the servable is made unavailable.

• END = 50

  This servable has reached the end of its journey in the manager. Either it loaded and ultimately unloaded successfully, or it hit an error at some point in its lifecycle.

Configuration for monitoring.

• optional PrometheusConfig prometheus_config = 1

Used in: PredictionLog

• optional MultiInferenceRequest request = 1

• optional MultiInferenceResponse response = 2

Inference request containing one or more requests.

Used as request type in: PredictionService.MultiInference

Used as field type in: MultiInferenceLog

• repeated InferenceTask tasks = 1

  Inference tasks.

• optional Input input = 2

  Input data.

Inference request containing one or more responses.

Used as response type in: PredictionService.MultiInference

Used as field type in: MultiInferenceLog

• repeated InferenceResult results = 1

  List of results; one for each InferenceTask in the request, returned in the same order as the request.

Configuration for a servable platform e.g. tensorflow or other ML systems.

Used in: PlatformConfigMap

• optional google.protobuf.Any source_adapter_config = 1

  The config proto for a SourceAdapter in the StoragePathSourceAdapter registry.

• map<string, PlatformConfig> platform_configs = 1

  A map from a platform name to a platform config. The platform name is used in ModelConfig.model_platform.

Used in: PredictionLog

• optional PredictRequest request = 1

• optional PredictResponse response = 2

PredictRequest specifies which TensorFlow model to run, as well as how inputs are mapped to tensors and how outputs are filtered before returning to user.

Used as request type in: PredictionService.Predict

Used as field type in: PredictLog

• optional ModelSpec model_spec = 1

  Model Specification. If version is not specified, will use the latest (numerical) version.

• map<string, TensorProto> inputs = 2

  Input tensors. Names of input tensor are alias names. The mapping from aliases to real input tensor names is stored in the SavedModel export as a prediction SignatureDef under the 'inputs' field.

• repeated string output_filter = 3

  Output filter. Names specified are alias names. The mapping from aliases to real output tensor names is stored in the SavedModel export as a prediction SignatureDef under the 'outputs' field. Only tensors specified here will be run/fetched and returned, with the exception that when none is specified, all tensors specified in the named signature will be run/fetched and returned.

Response for PredictRequest on successful run.

Used as response type in: PredictionService.Predict

Used as field type in: PredictLog

• optional ModelSpec model_spec = 2

  Effective Model Specification used to process PredictRequest.

• map<string, TensorProto> outputs = 1

  Output tensors.

Logged model inference request.

• optional LogMetadata log_metadata = 1

• oneof log_type

  • ClassifyLog classify_log = 2

  • RegressLog regress_log = 3

  • PredictLog predict_log = 6

  • MultiInferenceLog multi_inference_log = 4

  • SessionRunLog session_run_log = 5

Configuration for Prometheus monitoring.

Used in: MonitoringConfig

• bool enable = 1

  Whether to expose Prometheus metrics.

• string path = 2

  The endpoint to expose Prometheus metrics. If not specified, PrometheusExporter::kPrometheusPath value is used.

Used in: PredictionLog

• optional RegressionRequest request = 1

• optional RegressionResponse response = 2

Regression result for a single item (tensorflow.Example).

Used in: RegressionResult

• float value = 1

Used as request type in: PredictionService.Regress

Used as field type in: RegressLog

• optional ModelSpec model_spec = 1

  Model Specification. If version is not specified, will use the latest (numerical) version.

• optional Input input = 2

  Input data.

Used as response type in: PredictionService.Regress

Used as field type in: RegressLog

• optional ModelSpec model_spec = 2

  Effective Model Specification used for regression.

• optional RegressionResult result = 1

Contains one result per input example, in the same order as the input in RegressionRequest.

Used in: InferenceResult, RegressionResponse

• repeated Regression regressions = 1

One kind of resource on one device (or type of device).

Used in: ResourceAllocation.Entry

• string device = 1

  The type of device on which the resource resides, e.g. CPU or GPU.

• optional google.protobuf.UInt32Value device_instance = 2

  A specific instance of the device of type 'device' to which the resources are bound (instances are assumed to be numbered 0, 1, ...). When representing the resources required by a servable that has yet to be loaded, this field is optional. If not set, it denotes that the servable's resources are not (yet) bound to a specific instance.

• string kind = 3

  The kind of resource on the device (instance), e.g. RAM or compute share. A given type of resource should have a standard unit that represents the smallest useful quantization. We strongly recommend including the unit (e.g. bytes or millicores) in this string, as in "ram_bytes".

An allocation of one or more kinds of resources, along with the quantity of each. Used to denote the resources that a servable (or collection of servables) will use or is currently using. Also used to denote resources available to the serving system for loading more servables.

• repeated ResourceAllocation.Entry resource_quantities = 1

A collection of resources, each with a quantity. Treated as a resource-> quantity map, i.e. no resource can repeat and the order is immaterial.

Used in: ResourceAllocation

• optional Resource resource = 1

• uint64 quantity = 2

Configuration for a secure gRPC channel

• string server_key = 1

  private server key for SSL

• string server_cert = 2

  public server certificate

• string custom_ca = 3

  custom certificate authority

• bool client_verify = 4

  valid client certificate required ?

Used in: LogMetadata, LoggingConfig

• double sampling_rate = 1

  Requests will be logged uniformly at random with this probability. Valid range: [0, 1.0].

Config proto for SavedModelBundleSourceAdapter.

• optional SessionBundleConfig legacy_config = 1000

  A SessionBundleConfig. FOR INTERNAL USE ONLY DURING TRANSITION TO SAVED_MODEL. WILL BE DEPRECATED. TODO(b/32248363): Replace this field with the "real" field(s).

Configuration parameters for a SessionBundle, with optional batching.

Used in: SavedModelBundleSourceAdapterConfig, SessionBundleSourceAdapterConfig

• string session_target = 1

  The TensorFlow runtime to connect to. See full documentation in tensorflow/core/public/session_options.h. For single machine serving, we recommend using the empty string "", which will configure the local TensorFlow runtime implementation. This provides the best isolation currently available across multiple Session servables.

• optional ConfigProto session_config = 2

  TensorFlow Session configuration options. See details at tensorflow/core/protobuf/config.proto.

• optional BatchingParameters batching_parameters = 3

  If set, each emitted session is wrapped with a layer that schedules Run() calls in batches. The batching layer is transparent to the client (implements the tensorflow::Session API). IMPORTANT: With batching enabled, client threads will spend most of their time blocked on Session::Run() calls, waiting for enough peer threads to also call Session::Run() such that a large batch can be formed. For good throughput, we recommend setting the number of client threads equal to roughly twice the maximum batch size ('max_batch_size' below). The batching layer uses a SharedBatchScheduler to coordinate batching across multiple session servables emitted by this source adapter. A BatchSchedulerRetrier is added on top of each batching session.

• optional google.protobuf.Int32Value session_run_load_threadpool_index = 4

  If set, session run calls use a separate threadpool for restore and init ops as part of loading the session-bundle. The value of this field should correspond to the index of the tensorflow::ThreadPoolOptionProto defined as part of `session_config.session_inter_op_thread_pool`.

• uint64 experimental_transient_ram_bytes_during_load = 5

  EXPERIMENTAL. THIS FIELD MAY CHANGE OR GO AWAY. USE WITH CAUTION. Transient memory used while loading a model, which is released once the loading phase has completed. (This is on top of the memory used in steady- state while the model is in memory after it has finished loading.) TODO(b/38376838): This is a temporary hack, and it applies to all models. Remove it once resource estimates are moved inside SavedModel.

• repeated string saved_model_tags = 6

  Set of SavedModel tags identifying the specific meta graph def to be loaded.

• repeated NamedTensorProto experimental_fixed_input_tensors = 778

  EXPERIMENTAL. THIS FIELD MAY CHANGE OR GO AWAY. USE WITH CAUTION. Input tensors to append to every Session::Run() call.

• bool enable_model_warmup = 779

  Enables model warmup.

Config proto for SessionBundleSourceAdapter.

• optional SessionBundleConfig config = 1

Used in: PredictionLog

• optional SessionRunRequest request = 1

• optional SessionRunResponse response = 2

Used as request type in: SessionService.SessionRun

Used as field type in: SessionRunLog

• optional ModelSpec model_spec = 1

  Model Specification. If version is not specified, will use the latest (numerical) version.

• repeated NamedTensorProto feed = 2

  Tensors to be fed in the step. Each feed is a named tensor.

• repeated string fetch = 3

  Fetches. A list of tensor names. The caller expects a tensor to be returned for each fetch[i] (see RunResponse.tensor). The order of specified fetches does not change the execution order.

• repeated string target = 4

  Target Nodes. A list of node names. The named nodes will be run to but their outputs will not be fetched.

• optional RunOptions options = 5

  Options for the run call. **Currently ignored.**

Used as response type in: SessionService.SessionRun

Used as field type in: SessionRunLog

• optional ModelSpec model_spec = 3

  Effective Model Specification used for session run.

• repeated NamedTensorProto tensor = 1

  NOTE: The order of the returned tensors may or may not match the fetch order specified in RunRequest.

• optional RunMetadata metadata = 2

  Returned metadata if requested in the options.

Message returned for "signature_def" field.

• map<string, SignatureDef> signature_def = 1

Config proto for StaticStoragePathSource.

• string servable_name = 1

  The single servable name, version number and path to supply statically.

• int64 version_num = 2

• string version_path = 3

Status that corresponds to Status in third_party/tensorflow/core/lib/core/status.h.

Used in: ModelVersionStatus, ReloadConfigResponse

• error.Code error_code = 1

  Error code.

• string error_message = 2

  Error message. Will only be set if an error was encountered.