package tensorflow.serving

Get desktop application:
View/edit binary Protocol Buffers messages

Config for the batch op rewriter. This should be serialized and set a param in RewriterConfig with key kBatchOpRewriteParamKey.

• bool enable_adaptive_shared_batching_thread_pool = 4

• map<string, BatchOpRewriteConfig.AdaptiveBatchSchedulerOption> model_scheduler_options = 1

  Keyed by model name, meaning all batch-ops in one saved model would use the same adaptive-batch-scheduler option.

The options for tensorflow::serving::AdaptiveSharedBatchScheduler. See AdaptiveSharedBatchScheduler::Options for meaning of each field. NOTE: Leave this unset to pick up default settings which should work for most scenarios. Example scenarios when tuning helps: * Latency sensitive

Used in: BatchOpRewriteConfig

• optional google.protobuf.UInt32Value min_inflight_batches_limit = 1

• optional google.protobuf.UInt32Value initial_inflight_batches_limit = 2

• optional google.protobuf.UInt32Value max_inflight_batches_limit = 3

• optional google.protobuf.UInt32Value batches_to_average_over = 4

  You can use QPS as a reference to decide how quickly to react to workload changes.