package tensorflow.serving
Get desktop application:
View/edit binary Protocol Buffers messages
ModelService provides methods to query and update the state of the server, e.g. which models/versions are being served.
rpc GetModelStatus (, )
model_service.proto:17Gets status of model. If the ModelSpec in the request does not specify version, information about all versions of the model will be returned. If the ModelSpec in the request does specify a version, the status of only that version will be returned.
message
get_model_status.proto:12GetModelStatusRequest contains a ModelSpec indicating the model for which to get status.
optional model_spec = 1
Model Specification. If version is not specified, information about all versions of the model will be returned. If a version is specified, the status of only that version will be returned.
message
get_model_status.proto:64Response for ModelStatusRequest on successful run.
repeated model_version_status = 1
Version number and status information for applicable model version(s).
Reloads the set of served models. The new config supersedes the old one, so if a model is omitted from the new config it will be unloaded and no longer served.
message
model_management.proto:9optional config = 1
message
model_management.proto:13optional status = 1
open source marker; do not remove PredictionService provides access to machine-learned models loaded by model_servers.
rpc Classify (, )
prediction_service.proto:17Classify.
GetModelMetadata - provides access to metadata for loaded models.
message
get_model_metadata.proto:15optional model_spec = 1
Model Specification indicating which model we are querying for metadata. If version is not specified, will use the latest (numerical) version.
repeated string metadata_field = 2
Metadata fields to get. Currently supported: "signature_def".
message
get_model_metadata.proto:23optional model_spec = 1
Model Specification indicating which model this metadata belongs to.
map<string, > metadata = 2
Map of metadata field name to metadata field. The options for metadata field name are listed in GetModelMetadataRequest. Currently supported: "signature_def".
MultiInference API for multi-headed models.
rpc Predict (, )
prediction_service.proto:23Predict -- provides access to loaded TensorFlow model.
rpc Regress (, )
prediction_service.proto:20Regress.
SessionService defines a service with which a client can interact to execute Tensorflow model inference. The SessionService::SessionRun method is similar to MasterService::RunStep of Tensorflow, except that all sessions are ready to run, and you request a specific model/session with ModelSpec.
rpc SessionRun (, )
session_service.proto:50Runs inference of a given model.
message
session_bundle_config.proto:70Batching parameters. Each individual parameter is optional. If omitted, the default value from the relevant batching config struct (SharedBatchScheduler ::Options or BatchSchedulerRetrier::Options) is used.
SharedBatchScheduler options (see shared_batch_scheduler.h):
Used in:
optional max_batch_size = 1
The maximum size of each batch. IMPORTANT: As discussed above, use 'max_batch_size * 2' client threads to achieve high throughput with batching.
optional batch_timeout_micros = 2
If a task has been enqueued for this amount of time (in microseconds), and a thread is available, the scheduler will immediately form a batch from enqueued tasks and assign the batch to the thread for processing, even if the batch's size is below 'max_batch_size'.
optional max_enqueued_batches = 3
The maximum length of the queue, in terms of the number of batches. (A batch that has been scheduled on a thread is considered to have been removed from the queue.)
optional num_batch_threads = 4
The number of threads to use to process batches. Must be >= 1, and should be tuned carefully.
optional thread_pool_name = 5
The name to use for the pool of batch threads.
repeated int64 allowed_batch_sizes = 6
The allowed batch sizes. (Ignored if left empty.) Requirements: - The entries must be in increasing order. - The final entry must equal 'max_batch_size'.
bool pad_variable_length_inputs = 7
Whether to pad variable-length inputs when a batch is formed.
message
classification.proto:11A single class.
Used in:
string label = 1
Label or name of the class.
float score = 2
Score for this class (e.g., the probability the item belongs to this class). As per the proto3 default-value semantics, if the score is missing, it should be treated as 0.
message
classification.proto:33Used as request type in: PredictionService.Classify
Used as field type in:
optional model_spec = 1
Model Specification. If version is not specified, will use the latest (numerical) version.
optional input = 2
Input data.
message
classification.proto:42Used as response type in: PredictionService.Classify
Used as field type in:
optional model_spec = 2
Effective Model Specification used for classification.
optional result = 1
Result of the classification.
message
classification.proto:27Contains one result per input example, in the same order as the input in ClassificationRequest.
Used in: ,
repeated classifications = 1
message
classification.proto:21List of classes for a single item (tensorflow.Example).
Used in:
repeated classes = 1
message
prediction_log.proto:14Used in:
optional request = 1
optional response = 2
message
input.proto:15Specifies one or more fully independent input Examples. See examples at: https://github.com/tensorflow/tensorflow/blob/master/tensorflow/core/example/example.proto
Used in:
repeated examples = 1
message
input.proto:66Specifies one or more independent input Examples, with a common context Example. The common use case for context is to cleanly and optimally specify some features that are common across multiple examples. See example below with a search query as the context and multiple restaurants to perform some inference on. context: { feature: { key : "query" value: { bytes_list: { value: [ "pizza" ] } } } } examples: { feature: { key : "cuisine" value: { bytes_list: { value: [ "Pizzeria" ] } } } } examples: { feature: { key : "cuisine" value: { bytes_list: { value: [ "Taqueria" ] } } } } Implementations of ExampleListWithContext merge the context Example into each of the Examples. Note that feature keys must not be duplicated between the Examples and context Example, or the behavior is undefined. See also: tensorflow/core/example/example.proto https://developers.google.com/protocol-buffers/docs/proto3#maps
Used in:
repeated examples = 1
optional context = 2
Config proto for FileSystemStoragePathSource.
repeated servables = 5
The servables to monitor for new versions, and aspire.
string servable_name = 1
A single servable name/base_path pair to monitor. DEPRECATED: Use 'servables' instead. TODO(b/30898016): Stop using these fields, and ultimately remove them here.
string base_path = 2
int64 file_system_poll_wait_seconds = 3
How long to wait between file-system polling to look for children of 'base_path', in seconds. For testing use only: a negative value disables the polling thread.
bool fail_if_zero_versions_at_startup = 4
If true, then FileSystemStoragePathSource::Create() and ::UpdateConfig() fail if, for any configured servables, the file system doesn't currently contain at least one version under the base path. (Otherwise, it will emit a warning and keep pinging the file system to check for a version to appear later.)
A servable name and base path to look for versions of the servable.
Used in:
string servable_name = 1
The servable name to supply in aspired-versions callback calls. Child paths of 'base_path' are considered to be versions of this servable.
string base_path = 2
The path to monitor, i.e. look for child paths of the form base_path/123.
optional servable_version_policy = 4
The policy to determines the number of versions of the servable to be served at the same time.
A policy that dictates which version(s) of a servable should be served.
Used in: ,
oneof policy_choice
latest = 100
all = 101
specific = 102
Serve all versions found on disk.
Used in:
(message has no fields)
Serve the latest versions (i.e. the ones with the highest version numbers), among those found on disk. This is the default policy, with the default number of versions as 1.
Used in:
uint32 num_versions = 1
Number of latest versions to serve. (The default is 1.)
Serve a specific version (or set of versions). This policy is useful for rolling back to a specific version, or for canarying a specific version while still serving a separate stable version.
Used in:
repeated int64 versions = 1
The version numbers to serve.
message
hashmap_source_adapter.proto:6Config proto for HashmapSourceAdapter.
HashmapSourceAdapterConfig.Format format = 1
The format used by the file containing a serialized hashmap.
Used in:
SIMPLE_CSV = 0
A simple kind of CSV text file of the form: key0,value0\n key1,value1\n ...
message
inference.proto:36Inference result, matches the type of request or is an error.
Used in:
optional model_spec = 1
oneof result
classification_result = 2
regression_result = 3
message
inference.proto:23Inference request such as classification, regression, etc...
Used in:
optional model_spec = 1
Model Specification. If version is not specified, will use the latest (numerical) version. All ModelSpecs in a MultiInferenceRequest must access the same model name.
string method_name = 2
Signature's method_name. Should be one of the method names defined in third_party/tensorflow/python/saved_model/signature_constants.py. e.g. "tensorflow/serving/classify".
message
input.proto:71Used in: , ,
oneof kind
example_list = 1
example_list_with_context = 2
message
log_collector_config.proto:6Used in:
string type = 1
Identifies the type of the LogCollector we will use to collect these logs.
string filename_prefix = 2
The prefix to use for the filenames of the logs.
message
logging.proto:10Metadata logged along with the request logs.
Used in:
optional model_spec = 1
optional sampling_config = 2
TODO(b/33279154): Add more metadata as mentioned in the bug.
message
logging_config.proto:15Configuration for logging query/responses.
Used in:
optional log_collector_config = 1
optional sampling_config = 2
message
model_server_config.proto:19Common configuration for loading a model being served.
Used in:
string name = 1
Name of the model.
string base_path = 2
Base path to the model, excluding the version directory. E.g> for a model at /foo/bar/my_model/123, where 123 is the version, the base path is /foo/bar/my_model. (This can be changed once a model is in serving, *if* the underlying data remains the same. Otherwise there are no guarantees about whether the old or new data will be used for model versions currently loaded.)
ModelType model_type = 3
Type of model. TODO(b/31336131): DEPRECATED. Please use 'model_platform' instead.
string model_platform = 4
Type of model (e.g. "tensorflow"). (This cannot be changed once a model is in serving.)
optional model_version_policy = 7
Version policy for the model indicating which version(s) of the model to load and make available for serving simultaneously. The default option is to serve only the latest version of the model. (This can be changed once a model is in serving.)
map<string, int64> version_labels = 8
String labels to associate with versions of the model, allowing inference queries to refer to versions by label instead of number. Multiple labels can map to the same version, but not vice-versa. An envisioned use-case for these labels is canarying tentative versions. For example, one can assign labels "stable" and "canary" to two specific versions. Perhaps initially "stable" is assigned to version 0 and "canary" to version 1. Once version 1 passes canary, one can shift the "stable" label to refer to version 1 (at that point both labels map to the same version -- version 1 -- which is fine). Later once version 2 is ready to canary one can move the "canary" label to version 2. And so on.
optional logging_config = 6
Configures logging requests and responses, to the model. (This can be changed once a model is in serving.)
message
model_server_config.proto:71Static list of models to be loaded for serving.
Used in:
repeated config = 1
message
model_server_config.proto:76ModelServer config.
Used in:
oneof config
ModelServer takes either a static file-based model config list or an Any proto representing custom model config that is fetched dynamically at runtime (through network RPC, custom service, etc.).
model_config_list = 1
custom_model_config = 2
message
model.proto:9Metadata for an inference request such as the model name and version.
Used in: , , , , , , , , , , , , ,
string name = 1
Required servable name.
oneof version_choice
Optional choice of which version of the model to use. Recommended to be left unset in the common case. Should be specified only when there is a strong version consistency requirement. When left unspecified, the system will serve the best available version. This is typically the latest version, though during version transitions, notably when serving on a fleet of instances, may be either the previous or new version.
version = 2
Use this specific version number.
string version_label = 4
Use the version associated with the given label.
string signature_name = 3
A named signature to evaluate. If unspecified, the default signature will be used.
The type of model. TODO(b/31336131): DEPRECATED.
Used in:
MODEL_TYPE_UNSPECIFIED = 0
TENSORFLOW = 1
OTHER = 2
message
get_model_status.proto:20Version number, state, and status for a single version of a model.
Used in:
int64 version = 1
Model version.
ModelVersionStatus.State state = 2
Model state.
optional status = 3
Model status.
States that map to ManagerState enum in tensorflow_serving/core/servable_state.h
Used in:
UNKNOWN = 0
Default value.
START = 10
The manager is tracking this servable, but has not initiated any action pertaining to it.
LOADING = 20
The manager has decided to load this servable. In particular, checks around resource availability and other aspects have passed, and the manager is about to invoke the loader's Load() method.
AVAILABLE = 30
The manager has successfully loaded this servable and made it available for serving (i.e. GetServableHandle(id) will succeed). To avoid races, this state is not reported until *after* the servable is made available.
UNLOADING = 40
The manager has decided to make this servable unavailable, and unload it. To avoid races, this state is reported *before* the servable is made unavailable.
END = 50
This servable has reached the end of its journey in the manager. Either it loaded and ultimately unloaded successfully, or it hit an error at some point in its lifecycle.
message
monitoring_config.proto:17Configuration for monitoring.
optional prometheus_config = 1
message
prediction_log.proto:29Used in:
optional request = 1
optional response = 2
message
inference.proto:46Inference request containing one or more requests.
Used as request type in: PredictionService.MultiInference
Used as field type in:
repeated tasks = 1
Inference tasks.
optional input = 2
Input data.
message
inference.proto:55Inference request containing one or more responses.
Used as response type in: PredictionService.MultiInference
Used as field type in:
repeated results = 1
List of results; one for each InferenceTask in the request, returned in the same order as the request.
message
platform_config.proto:9Configuration for a servable platform e.g. tensorflow or other ML systems.
Used in:
optional source_adapter_config = 1
The config proto for a SourceAdapter in the StoragePathSourceAdapter registry.
message
platform_config.proto:15map<string, > platform_configs = 1
A map from a platform name to a platform config. The platform name is used in ModelConfig.model_platform.
message
prediction_log.proto:24Used in:
optional request = 1
optional response = 2
message
predict.proto:12PredictRequest specifies which TensorFlow model to run, as well as how inputs are mapped to tensors and how outputs are filtered before returning to user.
Used as request type in: PredictionService.Predict
Used as field type in:
optional model_spec = 1
Model Specification. If version is not specified, will use the latest (numerical) version.
map<string, > inputs = 2
Input tensors. Names of input tensor are alias names. The mapping from aliases to real input tensor names is stored in the SavedModel export as a prediction SignatureDef under the 'inputs' field.
repeated string output_filter = 3
Output filter. Names specified are alias names. The mapping from aliases to real output tensor names is stored in the SavedModel export as a prediction SignatureDef under the 'outputs' field. Only tensors specified here will be run/fetched and returned, with the exception that when none is specified, all tensors specified in the named signature will be run/fetched and returned.
message
predict.proto:34Response for PredictRequest on successful run.
Used as response type in: PredictionService.Predict
Used as field type in:
optional model_spec = 2
Effective Model Specification used to process PredictRequest.
map<string, > outputs = 1
Output tensors.
message
prediction_log.proto:40Logged model inference request.
optional log_metadata = 1
oneof log_type
classify_log = 2
regress_log = 3
predict_log = 6
multi_inference_log = 4
session_run_log = 5
message
monitoring_config.proto:7Configuration for Prometheus monitoring.
Used in:
bool enable = 1
Whether to expose Prometheus metrics.
string path = 2
The endpoint to expose Prometheus metrics. If not specified, PrometheusExporter::kPrometheusPath value is used.
message
prediction_log.proto:19Used in:
optional request = 1
optional response = 2
message
regression.proto:11Regression result for a single item (tensorflow.Example).
Used in:
float value = 1
message
regression.proto:23Used as request type in: PredictionService.Regress
Used as field type in:
optional model_spec = 1
Model Specification. If version is not specified, will use the latest (numerical) version.
optional input = 2
Input data.
message
regression.proto:32Used as response type in: PredictionService.Regress
Used as field type in:
optional model_spec = 2
Effective Model Specification used for regression.
optional result = 1
message
regression.proto:17Contains one result per input example, in the same order as the input in RegressionRequest.
Used in: ,
repeated regressions = 1
message
resources.proto:14One kind of resource on one device (or type of device).
Used in:
string device = 1
The type of device on which the resource resides, e.g. CPU or GPU.
optional device_instance = 2
A specific instance of the device of type 'device' to which the resources are bound (instances are assumed to be numbered 0, 1, ...). When representing the resources required by a servable that has yet to be loaded, this field is optional. If not set, it denotes that the servable's resources are not (yet) bound to a specific instance.
string kind = 3
The kind of resource on the device (instance), e.g. RAM or compute share. A given type of resource should have a standard unit that represents the smallest useful quantization. We strongly recommend including the unit (e.g. bytes or millicores) in this string, as in "ram_bytes".
message
resources.proto:38An allocation of one or more kinds of resources, along with the quantity of each. Used to denote the resources that a servable (or collection of servables) will use or is currently using. Also used to denote resources available to the serving system for loading more servables.
repeated resource_quantities = 1
message
resources.proto:41A collection of resources, each with a quantity. Treated as a resource-> quantity map, i.e. no resource can repeat and the order is immaterial.
Used in:
optional resource = 1
uint64 quantity = 2
message
ssl_config.proto:7Configuration for a secure gRPC channel
string server_key = 1
private server key for SSL
string server_cert = 2
public server certificate
string custom_ca = 3
custom certificate authority
bool client_verify = 4
valid client certificate required ?
message
logging_config.proto:8Used in: ,
double sampling_rate = 1
Requests will be logged uniformly at random with this probability. Valid range: [0, 1.0].
Config proto for SavedModelBundleSourceAdapter.
optional legacy_config = 1000
A SessionBundleConfig. FOR INTERNAL USE ONLY DURING TRANSITION TO SAVED_MODEL. WILL BE DEPRECATED. TODO(b/32248363): Replace this field with the "real" field(s).
message
session_bundle_config.proto:10Configuration parameters for a SessionBundle, with optional batching.
Used in: ,
string session_target = 1
The TensorFlow runtime to connect to. See full documentation in tensorflow/core/public/session_options.h. For single machine serving, we recommend using the empty string "", which will configure the local TensorFlow runtime implementation. This provides the best isolation currently available across multiple Session servables.
optional session_config = 2
TensorFlow Session configuration options. See details at tensorflow/core/protobuf/config.proto.
optional batching_parameters = 3
If set, each emitted session is wrapped with a layer that schedules Run() calls in batches. The batching layer is transparent to the client (implements the tensorflow::Session API). IMPORTANT: With batching enabled, client threads will spend most of their time blocked on Session::Run() calls, waiting for enough peer threads to also call Session::Run() such that a large batch can be formed. For good throughput, we recommend setting the number of client threads equal to roughly twice the maximum batch size ('max_batch_size' below). The batching layer uses a SharedBatchScheduler to coordinate batching across multiple session servables emitted by this source adapter. A BatchSchedulerRetrier is added on top of each batching session.
optional session_run_load_threadpool_index = 4
If set, session run calls use a separate threadpool for restore and init ops as part of loading the session-bundle. The value of this field should correspond to the index of the tensorflow::ThreadPoolOptionProto defined as part of `session_config.session_inter_op_thread_pool`.
uint64 experimental_transient_ram_bytes_during_load = 5
EXPERIMENTAL. THIS FIELD MAY CHANGE OR GO AWAY. USE WITH CAUTION. Transient memory used while loading a model, which is released once the loading phase has completed. (This is on top of the memory used in steady- state while the model is in memory after it has finished loading.) TODO(b/38376838): This is a temporary hack, and it applies to all models. Remove it once resource estimates are moved inside SavedModel.
repeated string saved_model_tags = 6
Set of SavedModel tags identifying the specific meta graph def to be loaded.
repeated experimental_fixed_input_tensors = 778
EXPERIMENTAL. THIS FIELD MAY CHANGE OR GO AWAY. USE WITH CAUTION. Input tensors to append to every Session::Run() call.
bool enable_model_warmup = 779
Enables model warmup.
Config proto for SessionBundleSourceAdapter.
optional config = 1
message
prediction_log.proto:34Used in:
optional request = 1
optional response = 2
message
session_service.proto:11Used as request type in: SessionService.SessionRun
Used as field type in:
optional model_spec = 1
Model Specification. If version is not specified, will use the latest (numerical) version.
repeated feed = 2
Tensors to be fed in the step. Each feed is a named tensor.
repeated string fetch = 3
Fetches. A list of tensor names. The caller expects a tensor to be returned for each fetch[i] (see RunResponse.tensor). The order of specified fetches does not change the execution order.
repeated string target = 4
Target Nodes. A list of node names. The named nodes will be run to but their outputs will not be fetched.
optional options = 5
Options for the run call. **Currently ignored.**
message
session_service.proto:32Used as response type in: SessionService.SessionRun
Used as field type in:
optional model_spec = 3
Effective Model Specification used for session run.
repeated tensor = 1
NOTE: The order of the returned tensors may or may not match the fetch order specified in RunRequest.
optional metadata = 2
Returned metadata if requested in the options.
message
get_model_metadata.proto:11Message returned for "signature_def" field.
map<string, > signature_def = 1
Config proto for StaticStoragePathSource.
string servable_name = 1
The single servable name, version number and path to supply statically.
int64 version_num = 2
string version_path = 3
message
status.proto:11Status that corresponds to Status in third_party/tensorflow/core/lib/core/status.h.
Used in: ,
diplomacy.tensorflow.error.Code error_code = 1
Error code.
string error_message = 2
Error message. Will only be set if an error was encountered.