package tensorflow.serving

Get desktop application:
View/edit binary Protocol Buffers messages

ModelService provides methods to query and update the state of the server, e.g. which models/versions are being served.

• rpc GetModelStatus (GetModelStatusRequest, GetModelStatusResponse)

  model_service.proto:17

  Gets status of model. If the ModelSpec in the request does not specify version, information about all versions of the model will be returned. If the ModelSpec in the request does specify a version, the status of only that version will be returned.

  message GetModelStatusRequest

  get_model_status.proto:12

  GetModelStatusRequest contains a ModelSpec indicating the model for which to get status.

  • optional ModelSpec model_spec = 1

    Model Specification. If version is not specified, information about all versions of the model will be returned. If a version is specified, the status of only that version will be returned.

  message GetModelStatusResponse

  get_model_status.proto:64

  Response for ModelStatusRequest on successful run.

  • repeated ModelVersionStatus model_version_status = 1

    Version number and status information for applicable model version(s).

• rpc HandleReloadConfigRequest (ReloadConfigRequest, ReloadConfigResponse)

  model_service.proto:22

  Reloads the set of served models. The new config supersedes the old one, so if a model is omitted from the new config it will be unloaded and no longer served.

  message ReloadConfigRequest

  model_management.proto:10

  • optional ModelServerConfig config = 1

  message ReloadConfigResponse

  model_management.proto:14

  • optional StatusProto status = 1

open source marker; do not remove PredictionService provides access to machine-learned models loaded by model_servers.

• rpc Classify (ClassificationRequest, ClassificationResponse)

  prediction_service.proto:17

  Classify.

• rpc GetModelMetadata (GetModelMetadataRequest, GetModelMetadataResponse)

  prediction_service.proto:29

  GetModelMetadata - provides access to metadata for loaded models.

  message GetModelMetadataRequest

  get_model_metadata.proto:15

  • optional ModelSpec model_spec = 1

    Model Specification indicating which model we are querying for metadata. If version is not specified, will use the latest (numerical) version.

  • repeated string metadata_field = 2

    Metadata fields to get. Currently supported: "signature_def".

  message GetModelMetadataResponse

  get_model_metadata.proto:23

  • optional ModelSpec model_spec = 1

    Model Specification indicating which model this metadata belongs to.

  • map<string, google.protobuf.Any> metadata = 2

    Map of metadata field name to metadata field. The options for metadata field name are listed in GetModelMetadataRequest. Currently supported: "signature_def".

• rpc MultiInference (MultiInferenceRequest, MultiInferenceResponse)

  prediction_service.proto:26

  MultiInference API for multi-headed models.

• rpc Predict (PredictRequest, PredictResponse)

  prediction_service.proto:23

  Predict -- provides access to loaded TensorFlow model.

• rpc Regress (RegressionRequest, RegressionResponse)

  prediction_service.proto:20

  Regress.

SessionService defines a service with which a client can interact to execute Tensorflow model inference. The SessionService::SessionRun method is similar to MasterService::RunStep of Tensorflow, except that all sessions are ready to run, and you request a specific model/session with ModelSpec.

• rpc SessionRun (SessionRunRequest, SessionRunResponse)

  session_service.proto:55

  Runs inference of a given model.

A single class.

Used in: Classifications

• string label = 1

  Label or name of the class.

• float score = 2

  Score for this class (e.g., the probability the item belongs to this class). As per the proto3 default-value semantics, if the score is missing, it should be treated as 0.

Used as request type in: PredictionService.Classify

Used as field type in: ClassifyLog

• optional ModelSpec model_spec = 1

  Model Specification. If version is not specified, will use the latest (numerical) version.

• optional Input input = 2

  Input data.

Used as response type in: PredictionService.Classify

Used as field type in: ClassifyLog

• optional ModelSpec model_spec = 2

  Effective Model Specification used for classification.

• optional ClassificationResult result = 1

  Result of the classification.

Contains one result per input example, in the same order as the input in ClassificationRequest.

Used in: ClassificationResponse, InferenceResult

• repeated Classifications classifications = 1

List of classes for a single item (tensorflow.Example).

Used in: ClassificationResult

• repeated Class classes = 1

Used in: PredictionLog

• optional ClassificationRequest request = 1

• optional ClassificationResponse response = 2

Specifies one or more fully independent input Examples. See examples at: https://github.com/tensorflow/tensorflow/blob/master/tensorflow/core/example/example.proto

Used in: Input

• repeated Example examples = 1

Specifies one or more independent input Examples, with a common context Example. The common use case for context is to cleanly and optimally specify some features that are common across multiple examples. See example below with a search query as the context and multiple restaurants to perform some inference on. context: { features: { feature: { key : "query" value: { bytes_list: { value: [ "pizza" ] } } } } } examples: { features: { feature: { key : "cuisine" value: { bytes_list: { value: [ "Pizzeria" ] } } } } } examples: { features: { feature: { key : "cuisine" value: { bytes_list: { value: [ "Taqueria" ] } } } } } Implementations of ExampleListWithContext merge the context Example into each of the Examples. Note that feature keys must not be duplicated between the Examples and context Example, or the behavior is undefined. See also: tensorflow/core/example/example.proto https://developers.google.com/protocol-buffers/docs/proto3#maps

Used in: Input

• repeated Example examples = 1

• optional Example context = 2

Config proto for FileSystemStoragePathSource.

• repeated FileSystemStoragePathSourceConfig.ServableToMonitor servables = 5

  The servables to monitor for new versions, and aspire.

• string servable_name = 1

  A single servable name/base_path pair to monitor. DEPRECATED: Use 'servables' instead. TODO(b/30898016): Stop using these fields, and ultimately remove them here.

• string base_path = 2

• int64 file_system_poll_wait_seconds = 3

  How long to wait between file-system polling to look for children of 'base_path', in seconds. If set to zero, filesystem will be polled exactly once. If set to a negative value (for testing use only), polling will be entirely disabled.

• bool fail_if_zero_versions_at_startup = 4

  If true, then FileSystemStoragePathSource::Create() and ::UpdateConfig() fail if, for any configured servables, the file system doesn't currently contain at least one version under the base path. (Otherwise, it will emit a warning and keep pinging the file system to check for a version to appear later.) DEPRECATED: Use 'servable_versions_always_present' instead, which includes this behavior. TODO(b/30898016): Remove 2019-10-31 or later.

• bool servable_versions_always_present = 6

  If true, the servable is always expected to exist on the underlying filesystem. FileSystemStoragePathSource::Create() and ::UpdateConfig() will fail if, for any configured servables, the file system doesn't currently contain at least one version under the base path. In addition, if a polling loop find the base path empty, it will not unload existing servables.

A servable name and base path to look for versions of the servable.

Used in: FileSystemStoragePathSourceConfig

• string servable_name = 1

  The servable name to supply in aspired-versions callback calls. Child paths of 'base_path' are considered to be versions of this servable.

• string base_path = 2

  The path to monitor, i.e. look for child paths of the form base_path/123.

• optional ServableVersionPolicy servable_version_policy = 4

  The policy to determines the number of versions of the servable to be served at the same time.

A policy that dictates which version(s) of a servable should be served.

Used in: ServableToMonitor, ModelConfig

• oneof policy_choice

  • ServableVersionPolicy.Latest latest = 100

  • ServableVersionPolicy.All all = 101

  • ServableVersionPolicy.Specific specific = 102

Serve all versions found on disk.

Used in: ServableVersionPolicy

(message has no fields)

Serve the latest versions (i.e. the ones with the highest version numbers), among those found on disk. This is the default policy, with the default number of versions as 1.

Used in: ServableVersionPolicy

• uint32 num_versions = 1

  Number of latest versions to serve. (The default is 1.)

Serve a specific version (or set of versions). This policy is useful for rolling back to a specific version, or for canarying a specific version while still serving a separate stable version.

Used in: ServableVersionPolicy

• repeated int64 versions = 1

  The version numbers to serve.

Inference result, matches the type of request or is an error.

Used in: MultiInferenceResponse

• optional ModelSpec model_spec = 1

• oneof result

  • ClassificationResult classification_result = 2

  • RegressionResult regression_result = 3

Inference request such as classification, regression, etc...

Used in: MultiInferenceRequest

• optional ModelSpec model_spec = 1

  Model Specification. If version is not specified, will use the latest (numerical) version. All ModelSpecs in a MultiInferenceRequest must access the same model name.

• string method_name = 2

  Signature's method_name. Should be one of the method names defined in third_party/tensorflow/python/saved_model/signature_constants.py. e.g. "tensorflow/serving/classify".

Used in: ClassificationRequest, MultiInferenceRequest, RegressionRequest

• oneof kind

  • ExampleList example_list = 1

  • ExampleListWithContext example_list_with_context = 2

Used in: LoggingConfig

• string type = 1

  Identifies the type of the LogCollector we will use to collect these logs.

• string filename_prefix = 2

  The prefix to use for the filenames of the logs.

Metadata logged along with the request logs.

Used in: PredictionLog

• optional ModelSpec model_spec = 1

• optional SamplingConfig sampling_config = 2

• repeated string saved_model_tags = 3

  List of tags used to load the relevant MetaGraphDef from SavedModel.

  TODO(b/33279154): Add more metadata as mentioned in the bug.

Configuration for logging query/responses.

Used in: ModelConfig

• optional LogCollectorConfig log_collector_config = 1

• optional SamplingConfig sampling_config = 2

Common configuration for loading a model being served.

Used in: ModelConfigList

• string name = 1

  Name of the model.

• string base_path = 2

  Base path to the model, excluding the version directory. E.g> for a model at /foo/bar/my_model/123, where 123 is the version, the base path is /foo/bar/my_model. (This can be changed once a model is in serving, *if* the underlying data remains the same. Otherwise there are no guarantees about whether the old or new data will be used for model versions currently loaded.)

• ModelType model_type = 3

  Type of model. TODO(b/31336131): DEPRECATED. Please use 'model_platform' instead.

• string model_platform = 4

  Type of model (e.g. "tensorflow"). (This cannot be changed once a model is in serving.)

• optional FileSystemStoragePathSourceConfig.ServableVersionPolicy model_version_policy = 7

  Version policy for the model indicating which version(s) of the model to load and make available for serving simultaneously. The default option is to serve only the latest version of the model. (This can be changed once a model is in serving.)

• map<string, int64> version_labels = 8

  String labels to associate with versions of the model, allowing inference queries to refer to versions by label instead of number. Multiple labels can map to the same version, but not vice-versa. An envisioned use-case for these labels is canarying tentative versions. For example, one can assign labels "stable" and "canary" to two specific versions. Perhaps initially "stable" is assigned to version 0 and "canary" to version 1. Once version 1 passes canary, one can shift the "stable" label to refer to version 1 (at that point both labels map to the same version -- version 1 -- which is fine). Later once version 2 is ready to canary one can move the "canary" label to version 2. And so on.

• optional LoggingConfig logging_config = 6

  Configures logging requests and responses, to the model. (This can be changed once a model is in serving.)

Static list of models to be loaded for serving.

Used in: ModelServerConfig

• repeated ModelConfig config = 1

ModelServer config.

Used in: ReloadConfigRequest

• oneof config

  ModelServer takes either a static file-based model config list or an Any proto representing custom model config that is fetched dynamically at runtime (through network RPC, custom service, etc.).

  • ModelConfigList model_config_list = 1

  • google.protobuf.Any custom_model_config = 2

Metadata for an inference request such as the model name and version.

Used in: ClassificationRequest, ClassificationResponse, GetModelMetadataRequest, GetModelMetadataResponse, GetModelStatusRequest, InferenceResult, InferenceTask, LogMetadata, PredictRequest, PredictResponse, RegressionRequest, RegressionResponse, SessionRunRequest, SessionRunResponse

• string name = 1

  Required servable name.

• oneof version_choice

  Optional choice of which version of the model to use. Recommended to be left unset in the common case. Should be specified only when there is a strong version consistency requirement. When left unspecified, the system will serve the best available version. This is typically the latest version, though during version transitions, notably when serving on a fleet of instances, may be either the previous or new version.

  • google.protobuf.Int64Value version = 2

    Use this specific version number.

  • string version_label = 4

    Use the version associated with the given label.

• string signature_name = 3

  A named signature to evaluate. If unspecified, the default signature will be used.

The type of model. TODO(b/31336131): DEPRECATED.

Used in: ModelConfig

• MODEL_TYPE_UNSPECIFIED = 0

• TENSORFLOW = 1

• OTHER = 2

Version number, state, and status for a single version of a model.

Used in: GetModelStatusResponse

• int64 version = 1

  Model version.

• ModelVersionStatus.State state = 2

  Model state.

• optional StatusProto status = 3

  Model status.

States that map to ManagerState enum in tensorflow_serving/core/servable_state.h

Used in: ModelVersionStatus

• UNKNOWN = 0

  Default value.

• START = 10

  The manager is tracking this servable, but has not initiated any action pertaining to it.

• LOADING = 20

  The manager has decided to load this servable. In particular, checks around resource availability and other aspects have passed, and the manager is about to invoke the loader's Load() method.

• AVAILABLE = 30

  The manager has successfully loaded this servable and made it available for serving (i.e. GetServableHandle(id) will succeed). To avoid races, this state is not reported until *after* the servable is made available.

• UNLOADING = 40

  The manager has decided to make this servable unavailable, and unload it. To avoid races, this state is reported *before* the servable is made unavailable.

• END = 50

  This servable has reached the end of its journey in the manager. Either it loaded and ultimately unloaded successfully, or it hit an error at some point in its lifecycle.

Configuration for monitoring.

• optional PrometheusConfig prometheus_config = 1

Used in: PredictionLog

• optional MultiInferenceRequest request = 1

• optional MultiInferenceResponse response = 2

Inference request containing one or more requests.

Used as request type in: PredictionService.MultiInference

Used as field type in: MultiInferenceLog

• repeated InferenceTask tasks = 1

  Inference tasks.

• optional Input input = 2

  Input data.

Inference request containing one or more responses.

Used as response type in: PredictionService.MultiInference

Used as field type in: MultiInferenceLog

• repeated InferenceResult results = 1

  List of results; one for each InferenceTask in the request, returned in the same order as the request.

Configuration for a servable platform e.g. tensorflow or other ML systems.

Used in: PlatformConfigMap

• optional google.protobuf.Any source_adapter_config = 1

  The config proto for a SourceAdapter in the StoragePathSourceAdapter registry.

• map<string, PlatformConfig> platform_configs = 1

  A map from a platform name to a platform config. The platform name is used in ModelConfig.model_platform.

Used in: PredictionLog

• optional PredictRequest request = 1

• optional PredictResponse response = 2

PredictRequest specifies which TensorFlow model to run, as well as how inputs are mapped to tensors and how outputs are filtered before returning to user.

Used as request type in: PredictionService.Predict

Used as field type in: PredictLog

• optional ModelSpec model_spec = 1

  Model Specification. If version is not specified, will use the latest (numerical) version.

• map<string, TensorProto> inputs = 2

  Input tensors. Names of input tensor are alias names. The mapping from aliases to real input tensor names is stored in the SavedModel export as a prediction SignatureDef under the 'inputs' field.

• repeated string output_filter = 3

  Output filter. Names specified are alias names. The mapping from aliases to real output tensor names is stored in the SavedModel export as a prediction SignatureDef under the 'outputs' field. Only tensors specified here will be run/fetched and returned, with the exception that when none is specified, all tensors specified in the named signature will be run/fetched and returned.

Response for PredictRequest on successful run.

Used as response type in: PredictionService.Predict

Used as field type in: PredictLog

• optional ModelSpec model_spec = 2

  Effective Model Specification used to process PredictRequest.

• map<string, TensorProto> outputs = 1

  Output tensors.

Logged model inference request.

• optional LogMetadata log_metadata = 1

• oneof log_type

  • ClassifyLog classify_log = 2

  • RegressLog regress_log = 3

  • PredictLog predict_log = 6

  • MultiInferenceLog multi_inference_log = 4

  • SessionRunLog session_run_log = 5

Configuration for Prometheus monitoring.

Used in: MonitoringConfig

• bool enable = 1

  Whether to expose Prometheus metrics.

• string path = 2

  The endpoint to expose Prometheus metrics. If not specified, PrometheusExporter::kPrometheusPath value is used.

Used in: PredictionLog

• optional RegressionRequest request = 1

• optional RegressionResponse response = 2

Regression result for a single item (tensorflow.Example).

Used in: RegressionResult

• float value = 1

Used as request type in: PredictionService.Regress

Used as field type in: RegressLog

• optional ModelSpec model_spec = 1

  Model Specification. If version is not specified, will use the latest (numerical) version.

• optional Input input = 2

  Input data.

Used as response type in: PredictionService.Regress

Used as field type in: RegressLog

• optional ModelSpec model_spec = 2

  Effective Model Specification used for regression.

• optional RegressionResult result = 1

Contains one result per input example, in the same order as the input in RegressionRequest.

Used in: InferenceResult, RegressionResponse

• repeated Regression regressions = 1

Configuration for a secure gRPC channel

• string server_key = 1

  private server key for SSL

• string server_cert = 2

  public server certificate

• string custom_ca = 3

  custom certificate authority

• bool client_verify = 4

  valid client certificate required ?

Used in: LogMetadata, LoggingConfig

• double sampling_rate = 1

  Requests will be logged uniformly at random with this probability. Valid range: [0, 1.0].

Used in: PredictionLog

• optional SessionRunRequest request = 1

• optional SessionRunResponse response = 2

Used as request type in: SessionService.SessionRun

Used as field type in: SessionRunLog

• optional ModelSpec model_spec = 1

  Model Specification. If version is not specified, will use the latest (numerical) version.

• repeated NamedTensorProto feed = 2

  Tensors to be fed in the step. Each feed is a named tensor.

• repeated string fetch = 3

  Fetches. A list of tensor names. The caller expects a tensor to be returned for each fetch[i] (see RunResponse.tensor). The order of specified fetches does not change the execution order.

• repeated string target = 4

  Target Nodes. A list of node names. The named nodes will be run to but their outputs will not be fetched.

• bool tensor_name_is_alias = 6

  If true, treat names in feed/fetch/target as alias names than actual tensor names (that appear in the TF graph). Alias names are resolved to actual names using `SignatureDef` in SavedModel associated with the model.

• optional RunOptions options = 5

  Options for the run call. **Currently ignored.**

Used as response type in: SessionService.SessionRun

Used as field type in: SessionRunLog

• optional ModelSpec model_spec = 3

  Effective Model Specification used for session run.

• repeated NamedTensorProto tensor = 1

  NOTE: The order of the returned tensors may or may not match the fetch order specified in RunRequest.

• optional RunMetadata metadata = 2

  Returned metadata if requested in the options.

Message returned for "signature_def" field.

• map<string, SignatureDef> signature_def = 1

Status that corresponds to Status in third_party/tensorflow/core/lib/core/status.h.

Used in: ModelVersionStatus, ReloadConfigResponse

• error.Code error_code = 1

  Error code.

• string error_message = 2

  Error message. Will only be set if an error was encountered.