package tensorflow.data

Get desktop application:
View/edit binary Protocol Buffers messages

• rpc ClientHeartbeat (ClientHeartbeatRequest, ClientHeartbeatResponse)

  dispatcher.proto:265

  Heartbeats from the client. This lets the dispatcher know that the client is still active, and gives the dispatcher a chance to notify the client of new tasks.

  message ClientHeartbeatRequest

  dispatcher.proto:176

  Next tag: 5

  • int64 iteration_client_id = 1

    The iteration client id to heartbeat for.

  • oneof optional_current_round

    Reports which round the client is currently reading from when doing round-robin reads.

    • int64 current_round = 2

  • oneof optional_blocked_round

    Reports whether the client has successfully blocked the indicated round from starting. This enables the dispatcher to add a new task in the blocked round or later.

    • int64 blocked_round = 4

  message ClientHeartbeatResponse

  dispatcher.proto:194

  Next tag: 5

  • repeated TaskInfo task_info = 1

    A list of all tasks that the client should read from.

  • oneof optional_block_round

    Tells the client not to start the given round if possible.

    • int64 block_round = 3

  • bool iteration_finished = 2

    Whether the iteration has finished.

  • DeploymentMode deployment_mode = 4

    tf.data service deployment mode. Supported values are "REMOTE", "COLOCATED", and "HYBRID". If unspecified, it is assumed to be "REMOTE".

• rpc GetDataServiceConfig (GetDataServiceConfigRequest, GetDataServiceConfigResponse)

  dispatcher.proto:275

  Returns the config of a data service cluster.

  message GetDataServiceConfigRequest

  dispatcher.proto:105

  Next tag: 1

  (message has no fields)

  message GetDataServiceConfigResponse

  dispatcher.proto:108

  Next tag: 2

  • optional DataServiceConfig config = 1

• rpc GetDataServiceMetadata (GetDataServiceMetadataRequest, GetDataServiceMetadataResponse)

  dispatcher.proto:271

  Returns the data service metadata for the registered dataset.

  message GetDataServiceMetadataRequest

  dispatcher.proto:94

  Next tag: 2

  • string dataset_id = 1

    The dataset id to get the data service dataset metadata.

  message GetDataServiceMetadataResponse

  dispatcher.proto:100

  Next tag: 2

  • optional DataServiceMetadata metadata = 1

    The retrieved data service dataset metadata.

• rpc GetDatasetDef (GetDatasetDefRequest, GetDatasetDefResponse)

  dispatcher.proto:231

  Gets a dataset definition.

  message GetDatasetDefRequest

  dispatcher.proto:43

  Next tag: 2

  • string dataset_id = 1

  message GetDatasetDefResponse

  dispatcher.proto:48

  Next tag: 2

  • optional DatasetDef dataset_def = 1

• rpc GetOrCreateIteration (GetOrCreateIterationRequest, GetOrCreateIterationResponse)

  dispatcher.proto:251

  Gets an iteration if it already exists, otherwise creates it.

  message GetOrCreateIterationRequest

  dispatcher.proto:140

  Next tag: 3

  • int64 job_id = 1

    The job to create an iteration for.

  • int64 repetition = 2

    Which repetition of the job to read from.

  message GetOrCreateIterationResponse

  dispatcher.proto:148

  Next tag: 2

  • int64 iteration_client_id = 1

    An id for the client that will read from the iteration. When the client is done with the iteration, they should call ReleaseIterationClient with this id.

• rpc GetOrCreateJob (GetOrCreateJobRequest, GetOrCreateJobResponse)

  dispatcher.proto:248

  Gets a job if it already exists, otherwise creates it.

  message GetOrCreateJobRequest

  dispatcher.proto:113

  Next tag: 7

  • string dataset_id = 1

    The id of the dataset to create a job for.

  • optional ProcessingModeDef processing_mode_def = 2

    A mode controlling how the tf.data service produces data for the job.

  • oneof optional_job_name

    Optional job name identifying a shared job. If not set, the RPC will always create a new job.

    • string job_name = 3

  • oneof optional_num_consumers

    Optional number of consumers. If set, the job's tasks will provide their elements to consumers round-robin.

    • int64 num_consumers = 4

  • bool use_cross_trainer_cache = 5

    True if cross-trainer cache is enabled.

  • TargetWorkers target_workers = 6

    Specifies which workers the client of this job reads from.

  message GetOrCreateJobResponse

  dispatcher.proto:135

  Next tag: 2

  • int64 job_id = 1

• rpc GetOrRegisterDataset (GetOrRegisterDatasetRequest, GetOrRegisterDatasetResponse)

  dispatcher.proto:244

  Registers a dataset with the server, or returns its id if it is already registered. The dataset is constructed in a new graph, so it must not refer to external resources or variables.

  message GetOrRegisterDatasetRequest

  dispatcher.proto:74

  Next tag: 5

  • optional DatasetDef dataset = 1

    The dataset to register.

  • optional DataServiceMetadata metadata = 3

    Metadata related to tf.data service.

  • oneof optional_dataset_id

    • string dataset_id = 4

      If provided, tf.data service will register the dataset with the specified ID. Otherwise, it will generate a unique dataset ID.

  message GetOrRegisterDatasetResponse

  dispatcher.proto:88

  Next tag: 2

  • string dataset_id = 1

    The id for the registered dataset.

• rpc GetSplit (GetSplitRequest, GetSplitResponse)

  dispatcher.proto:234

  Gets the next split for a given iteration.

  message GetSplitRequest

  dispatcher.proto:53

  Next tag: 4

  • int64 iteration_id = 1

  • int64 repetition = 2

  • int64 split_provider_index = 3

  message GetSplitResponse

  dispatcher.proto:60

  Next tag: 3

  • optional TensorProto split = 1

  • bool end_of_splits = 2

• rpc GetVersion (GetVersionRequest, GetVersionResponse)

  dispatcher.proto:237

  Returns the API version of the server.

  message GetVersionRequest

  dispatcher.proto:66

  Next tag: 1

  (message has no fields)

  message GetVersionResponse

  dispatcher.proto:69

  Next tag: 2

  • int64 version = 1

• rpc GetWorkers (GetWorkersRequest, GetWorkersResponse)

  dispatcher.proto:268

  Reports a list of all workers registered with the dispatcher.

  message GetWorkersRequest

  dispatcher.proto:215

  Next tag: 1

  (message has no fields)

  message GetWorkersResponse

  dispatcher.proto:218

  Next tag: 2

  • repeated WorkerInfo workers = 1

    A list of all workers.

• rpc MaybeRemoveTask (MaybeRemoveTaskRequest, MaybeRemoveTaskResponse)

  dispatcher.proto:255

  Attempts to remove a task from a round-robin read iteration.

  message MaybeRemoveTaskRequest

  dispatcher.proto:156

  Next tag: 4

  • int64 task_id = 1

  • int64 consumer_index = 2

  • int64 round = 3

  message MaybeRemoveTaskResponse

  dispatcher.proto:163

  Next tag: 2

  • bool removed = 1

• rpc ReleaseIterationClient (ReleaseIterationClientRequest, ReleaseIterationClientResponse)

  dispatcher.proto:259

  Releases an iteration client so that an iteration may eventually be cleaned up.

  message ReleaseIterationClientRequest

  dispatcher.proto:168

  Next tag: 2

  • int64 iteration_client_id = 1

  message ReleaseIterationClientResponse

  dispatcher.proto:173

  Next tag: 1

  (message has no fields)

• rpc WorkerHeartbeat (WorkerHeartbeatRequest, WorkerHeartbeatResponse)

  dispatcher.proto:225

  Performs a periodic worker heartbeat.

  message WorkerHeartbeatRequest

  dispatcher.proto:18

  Next tag: 6

  • string worker_address = 1

  • string transfer_address = 3

  • repeated string worker_tags = 4

  • int64 worker_uid = 5

    The UID of the worker Borg job, used for telemetry.

  • repeated int64 current_tasks = 2

  message WorkerHeartbeatResponse

  dispatcher.proto:28

  Next tag: 3

  • repeated TaskDef new_tasks = 1

  • repeated int64 tasks_to_delete = 2

• rpc WorkerUpdate (WorkerUpdateRequest, WorkerUpdateResponse)

  dispatcher.proto:228

  Updates the dispatcher with information about the worker's state.

  message WorkerUpdateRequest

  dispatcher.proto:34

  Next tag: 3

  • string worker_address = 1

  • repeated TaskProgress updates = 2

  message WorkerUpdateResponse

  dispatcher.proto:40

  Next tag: 1

  (message has no fields)

• rpc GetElement (GetElementRequest, GetElementResponse)

  worker.proto:63

  Gets the next dataset element.

  message GetElementRequest

  worker.proto:14

  • int64 task_id = 1

    The task to fetch an element from.

  • oneof optional_consumer_index

    Optional index to identify the consumer.

    • int64 consumer_index = 2

  • oneof optional_round_index

    Optional round index, indicating which round of round-robin the consumer wants to read from. This is used to keep consumers in sync.

    • int64 round_index = 3

  • bool skipped_previous_round = 4

    Whether the previous round was skipped. This information is needed by the worker to recover after restarts.

  • bool allow_skip = 5

    Whether to skip the round if data isn't ready fast enough.

  • string trainer_id = 6

    The trainer ID used to read elements from a multi-trainer cache. This cache enables sharing data across concurrent training iterations. If set, this request will read the data requested by other trainers, if available.

  message GetElementResponse

  worker.proto:37

  • oneof element

    The produced element.

    • CompressedElement compressed = 3

    • UncompressedElement uncompressed = 5

  • int64 element_index = 6

    The element's index within the task it came from.

  • bool end_of_sequence = 2

    Boolean to indicate whether the iterator has been exhausted.

  • bool skip_task = 4

    Indicates whether the round was skipped.

• rpc GetWorkerTasks (GetWorkerTasksRequest, GetWorkerTasksResponse)

  worker.proto:66

  Gets the tasks currently being executed by the worker.

  message GetWorkerTasksRequest

  worker.proto:52

  Named GetWorkerTasks to avoid conflicting with GetTasks in dispatcher.proto

  (message has no fields)

  message GetWorkerTasksResponse

  worker.proto:54

  • repeated TaskInfo tasks = 1

• rpc ProcessTask (ProcessTaskRequest, ProcessTaskResponse)

  worker.proto:60

  Processes a task for a dataset, making elements available to clients.

  message ProcessTaskRequest

  worker.proto:8

  • optional TaskDef task = 1

  message ProcessTaskResponse

  worker.proto:12

  (message has no fields)

Next tag: 3

Used in: Update

• int64 iteration_id = 1

• int64 iteration_client_id = 2

Represents the type of auto-sharding we enable.

Used in: DistributeOptions

• AUTO = 0

  AUTO: Attempts FILE-based sharding, falling back to DATA-based sharding.

• FILE = 1

  FILE: Shards by input files (i.e. each worker will get a set of files to process). When this option is selected, make sure that there is at least as many files as workers. If there are fewer input files than workers, a runtime error will be raised.

• DATA = 2

  DATA: Shards by elements produced by the dataset. Each worker will process the whole dataset and discard the portion that is not for itself. Note that for this mode to correctly partitions the dataset elements, the dataset needs to produce elements in a deterministic order.

• HINT = 3

  HINT: Looks for the presence of `shard(SHARD_HINT, ...)` which is treated as a placeholder to replace with `shard(num_workers, worker_index)`.

• OFF = -1

  OFF: No sharding will be performed.

next: 5

Used in: Options

• oneof optional_enabled

  Whether to automatically tune performance knobs.

  • bool enabled = 1

• oneof optional_cpu_budget

  When autotuning is enabled (through autotune), determines the CPU budget to use. Values greater than the number of schedulable CPU cores are allowed but may result in CPU contention.

  • int32 cpu_budget = 2

• oneof optional_ram_budget

  When autotuning is enabled (through autotune), determines the RAM budget to use. Values greater than the available RAM in bytes may result in OOM. If 0, defaults to half of the available RAM in bytes.

  • int64 ram_budget = 3

• oneof optional_autotune_algorithm

  When autotuning is enabled (through autotune), determines the algorithm to use. If not explicitly set by user, autotuning will follow HILL_CLIMB algorithm but has more flexibility to tune parameters more aggressively, in which case the behavior is implementation specific and may change over time.

  • model.AutotuneAlgorithm autotune_algorithm = 4

next: 2

• CardinalityOptions.ComputeLevel compute_level = 1

Used in: CardinalityOptions

• CARDINALITY_COMPUTE_UNSPECIFIED = 0

• CARDINALITY_COMPUTE_LOW = 1

  Cardinality will only be computed if it can be determined in a cheap manner (ie. without reading from file sources). If the cardinality would be nontrivial to compute, Cardinality() will return UNKNOWN_CARDINALITY.

• CARDINALITY_COMPUTE_MODERATE = 2

  Moderate effort will be made to determine cardinality, such as reading index data from source files. If significant work is needed to compute cardinality (e.g. reading entire source file contents or executing user defined functions), Cardinality() will return UNKNOWN_CARDINALITY.

Updates dispatcher state based on a client heartbeat. Next tag: 4

Used in: Update

• int64 iteration_client_id = 1

• bool task_accepted = 2

• optional TaskRejected task_rejected = 3

Metadata describing a compressed component of a dataset element.

Used in: CompressedElement

• DataType dtype = 1

  The dtype of the component tensor.

• optional TensorShapeProto tensor_shape = 2

  The shape of the component tensor.

• int64 tensor_size_bytes = 3

  Size of the uncompressed tensor bytes. For tensors serialized as TensorProtos, this is TensorProto::BytesAllocatedLong(). For raw Tensors, this is the size of the buffer underlying the Tensor.

Used in: GetElementResponse

• bytes data = 1

  Compressed tensor bytes for all components of the element.

• repeated CompressedComponentMetadata component_metadata = 2

  Metadata for the components of the element.

Next tag: 5

Used in: Update

• int64 iteration_id = 1

• int64 job_id = 2

• int64 repetition = 3

• int64 num_split_providers = 4

Next tag: 9

Used in: Update

• int64 job_id = 1

• string job_name = 2

• string dataset_id = 3

• optional ProcessingModeDef processing_mode_def = 4

• oneof optional_num_consumers

  Optional number of consumers. If set, the iteration's tasks will provide their elements to consumers round-robin.

  • int64 num_consumers = 6

• TargetWorkers target_workers = 7

  Specifies which workers the client of this iteration reads from.

• bool use_cross_trainer_cache = 8

  True if cross-trainer cache is enabled.

Next tag: 8

Used in: Update

• int64 task_id = 1

• int64 iteration_id = 2

• string worker_address = 3

• string transfer_address = 4

• repeated string worker_tags = 6

• int64 worker_uid = 7

• int64 starting_round = 5

Next tag: 9

Used in: Update

• int64 task_id = 1

• int64 iteration_id = 2

• string worker_address = 4

• string transfer_address = 6

• repeated string worker_tags = 7

• int64 worker_uid = 8

• string trainer_id = 1

Data service config available to the client through GetDataServiceConfig RPC. Next tag: 2

Used in: GetDataServiceConfigResponse

• DeploymentMode deployment_mode = 1

Metadata related to tf.data service datasets. Next tag: 4

Used in: GetDataServiceMetadataResponse, GetOrRegisterDatasetRequest, RegisterDatasetUpdate

• oneof optional_element_spec

  • bytes element_spec = 1

    Serialized element spec.

• DataServiceMetadata.Compression compression = 2

• int64 cardinality = 3

  Cardinality of the dataset.

Used in: DataServiceMetadata

• COMPRESSION_UNSPECIFIED = 0

• COMPRESSION_OFF = 1

  No compression.

• COMPRESSION_SNAPPY = 2

  Snappy compression as defined in tensorflow/core/platform/snappy.h.

Next tag: 2

Used in: GetDatasetDefResponse, GetOrRegisterDatasetRequest, TaskDef

• optional GraphDef graph = 1

  We represent datasets as tensorflow GraphDefs which define the operations needed to create a tf.data dataset.

tf.data service deployment mode.

Used in: ClientHeartbeatResponse, DataServiceConfig, experimental.DispatcherConfig

• DEPLOYMENT_MODE_UNSPECIFIED = 0

• DEPLOYMENT_MODE_COLOCATED = 1

  tf.data service workers colocate with TF workers.

• DEPLOYMENT_MODE_REMOTE = 2

  tf.data service workers run in dedicated tf.data hosts.

• DEPLOYMENT_MODE_HYBRID = 3

  tf.data service workers run in colocated TF hosts and dedicated tf.data hosts.

State of the dispatcher server, exported to improve debuggability. Next tag: 4

Used in: ServerStateExport

• optional experimental.DispatcherConfig dispatcher_config = 1

• repeated string worker_addresses = 2

• repeated DispatcherStateExport.Iteration iterations = 3

Used in: DispatcherStateExport

• string dataset_id = 1

• int64 iteration_id = 2

• optional IterationKeyDef iteration_key = 3

• optional ProcessingModeDef processing_mode = 4

• int64 num_consumers = 6

• int64 num_clients = 8

• bool finished = 10

• bool garbage_collected = 11

next: 3

Used in: Options

• AutoShardPolicy auto_shard_policy = 1

• oneof optional_num_devices

  The number of devices attached to this input pipeline.

  • int32 num_devices = 2

Represents how to handle external state during serialization.

Used in: Options

• POLICY_WARN = 0

• POLICY_IGNORE = 1

• POLICY_FAIL = 2

Next tag: 2

Used in: Update

• int64 task_id = 1

Next tag: 2

Used in: Update

• int64 iteration_id = 1

Next tag: 3

Used in: DispatcherStateExport.Iteration

• string name = 1

• int64 iteration = 2

next: 2

• bytes name = 1

next: 20

Used in: Options

• oneof optional_apply_default_optimizations

  Whether to apply default graph optimizations. If False, only graph optimizations that have been explicitly enabled will be applied.

  • bool apply_default_optimizations = 1

• oneof optional_filter_fusion

  Whether to fuse filter transformations.

  • bool filter_fusion = 6

• oneof optional_map_and_batch_fusion

  Whether to fuse map and batch transformations.

  • bool map_and_batch_fusion = 9

• oneof optional_map_and_filter_fusion

  Whether to fuse map and filter transformations.

  • bool map_and_filter_fusion = 10

• oneof optional_map_fusion

  Whether to fuse map transformations.

  • bool map_fusion = 11

• oneof optional_map_parallelization

  Whether to parallelize stateless map transformations.

  • bool map_parallelization = 12

• oneof optional_noop_elimination

  Whether to eliminate no-op transformations.

  • bool noop_elimination = 14

• oneof optional_parallel_batch

  Whether to parallelize copying of batch elements. This optimization is highly experimental and can cause performance degradation (e.g. when the parallelization overhead exceeds the benefits of performing the data copies in parallel). You should only enable this optimization if a) your input pipeline is bottlenecked on batching and b) you have validated that this optimization improves performance.

  • bool parallel_batch = 15

• oneof optional_shuffle_and_repeat_fusion

  Whether to fuse shuffle and repeat transformations.

  • bool shuffle_and_repeat_fusion = 17

• oneof optional_filter_parallelization

  Whether to parallelize stateless filter transformations.

  • bool filter_parallelization = 18

• oneof optional_inject_prefetch

  Whether to inject 'Prefetch' as the last transformation. Only takes effect if the last transformation is synchronous; otherwise does nothing.

  • bool inject_prefetch = 19

Message stored with Dataset objects to control how datasets are processed and optimized. next: 8

• oneof optional_deterministic

  Whether the outputs need to be produced in deterministic order.

  • bool deterministic = 1

• optional AutotuneOptions autotune_options = 7

  The distribution strategy options associated with the dataset.

• optional DistributeOptions distribute_options = 2

  The distribution strategy options associated with the dataset.

• optional OptimizationOptions optimization_options = 3

  The optimization options associated with the dataset.

• oneof optional_slack

  Whether to introduce 'slack' in the last `prefetch` of the input pipeline, if it exists. This may reduce CPU contention with accelerator host-side activity at the start of a step. The slack frequency is determined by the number of devices attached to this input pipeline.

  • bool slack = 4

• optional ThreadingOptions threading_options = 5

  The threading options associated with the dataset.

• oneof optional_external_state_policy

  This option can be used to override the default policy for how to handle external state when serializing a dataset or checkpointing its iterator. There are three settings available - IGNORE: External state is ignored without a warning; WARN: External state is ignored and a warning is logged; FAIL: External state results in an error.

  • ExternalStatePolicy external_state_policy = 6

Next tag: 2

Used in: CreateJobUpdate, DispatcherStateExport.Iteration, GetOrCreateJobRequest, TaskDef

• ProcessingModeDef.ShardingPolicy sharding_policy = 1

Specifies how data is sharded among tf.data service workers.

Used in: ProcessingModeDef

• OFF = 0

  No sharding will be performed. Each worker produces the entire dataset without any sharding. With this mode, the best practice is to shuffle the dataset nondeterministically so that workers process the dataset in different orders.

• DYNAMIC = 1

  The input dataset is dynamically split among workers at runtime. Each worker gets the next split when it reads data from the dispatcher. There is no fixed sharding with this mode.

• FILE = 2

  The following are static sharding policies. The semantics are similar to `tf.data.experimental.AutoShardPolicy`. These policies require: * The tf.data service cluster has a fixed size, and you need to specify the workers in DispatcherConfig. * Each client only reads from the local tf.data service worker. Shards by input files (each worker will get a set of files to process). When this option is selected, make sure that there is at least as many files as workers. If there are fewer input files than workers, a runtime error will be raised.

• DATA = 3

  Shards by elements produced by the dataset. Each worker will process the whole dataset and discard the portion that is not for itself. Note that for this mode to correctly partitions the dataset elements, the dataset needs to produce elements in a deterministic order.

• FILE_OR_DATA = 4

  Attempts FILE-based sharding, falling back to DATA-based sharding on failures.

• HINT = 5

  Looks for the presence of `shard(SHARD_HINT, ...)` which is treated as a placeholder to replace with `shard(num_workers, worker_index)`.

Next tag: 5

Used in: Update

• int64 iteration_id = 1

• int64 repetition = 2

• int64 split_provider_index = 4

• bool finished = 3

  Whether the split provider reached its end.

Next tag: 5

Used in: Update

• string dataset_id = 1

• uint64 fingerprint = 2

• optional DataServiceMetadata metadata = 3

• bool dedupe_by_dataset_id = 4

Next tag: 5

Used in: Update

• string worker_address = 1

• string transfer_address = 2

• repeated string worker_tags = 3

• int64 worker_uid = 4

Next tag: 3

Used in: Update

• int64 iteration_client_id = 1

• int64 time_micros = 2

  The time when the client was released, measured in microseconds since the epoch.

Next tag: 2

Used in: Update

• int64 task_id = 1

State of the tf.data service server, exported to improve debuggability. The dispatcher and worker servers will populate the corresponding fields. Next tag: 3

• optional DispatcherStateExport dispatcher_state_export = 1

• optional WorkerStateExport worker_state_export = 2

Specifies which tf.data service workers to read from.

Used in: CreateJobUpdate, GetOrCreateJobRequest

• TARGET_WORKERS_UNSPECIFIED = 0

• TARGET_WORKERS_AUTO = 1

  tf.data service runtime decides which workers to read from.

• TARGET_WORKERS_ANY = 2

  Reads from any available worker.

• TARGET_WORKERS_LOCAL = 3

  Only reads from local workers. If no local worker is found, it is an error.

Next tag: 14

Used in: ProcessTaskRequest, WorkerHeartbeatResponse, WorkerStateExport

• oneof dataset

  The dataset to iterate over.

  • DatasetDef dataset_def = 1

  • string path = 2

• string dataset_id = 3

• int64 task_id = 4

• int64 iteration_id = 5

• int64 num_split_providers = 9

  In distributed epoch processing mode, we use one split provider for each source that feeds into the dataset. In parallel_epochs mode, `num_split_providers` is always zero.

• string worker_address = 8

  Address of the worker that the task is assigned to.

• optional ProcessingModeDef processing_mode_def = 10

• oneof optional_num_consumers

  Optional number of consumers. If set, the results of the task will be provided to consumers round-robin.

  • int64 num_consumers = 7

• int64 num_workers = 11

  Number of workers and the worker index. These are only populated when the `processing_mode_def` specifies a static sharding policy.

• int64 worker_index = 12

• bool use_cross_trainer_cache = 13

  True if cross-trainer cache is enabled.

Next tag: 8

Used in: ClientHeartbeatResponse, GetWorkerTasksResponse

• string worker_address = 1

  The address of the worker processing the task.

• string transfer_address = 4

  The transfer address of the worker processing the task.

• repeated string worker_tags = 6

  Tags attached to the worker. This allows reading from selected workers. For example, by applying a "COLOCATED" tag, tf.data service is able to read from the local tf.data worker if one exists, then from off-TF-host workers, to avoid cross-TF-host reads.

• int64 task_id = 2

  The task id.

• int64 iteration_id = 3

  The id of the iteration that the task is part of.

• int64 worker_uid = 7

  The UID of the worker Borg job, used for telemetry.

• int64 starting_round = 5

  The round to start reading from the task in. For non-round-robin reads, this is always 0.

Next tag: 3

Used in: WorkerUpdateRequest

• int64 task_id = 1

  The task that this message is about.

• bool completed = 2

  Whether the task has completed.

Indicates that a client failed to block before reaching the target round. Next tag: 2

Used in: ClientHeartbeatUpdate

• int64 new_target_round = 1

  A new target round to try adding the task in.

next: 3

Used in: Options

• oneof optional_max_intra_op_parallelism

  If set, it overrides the maximum degree of intra-op parallelism.

  • int32 max_intra_op_parallelism = 1

• oneof optional_private_threadpool_size

  If set, the dataset will use a private threadpool of the given size.

  • int32 private_threadpool_size = 2

An uncompressed dataset element.

Used in: GetElementResponse

• repeated TensorProto components = 1

Message representing journaled dispatcher metadata updates. When we apply one of these changes to the dispatcher's in-memory state, we also write an Update message to the journal. Next tag: 15

• oneof update_type

  • RegisterDatasetUpdate register_dataset = 1

  • RegisterWorkerUpdate register_worker = 5

  • CreateJobUpdate create_job = 14

  • CreateIterationUpdate create_iteration = 2

  • ProduceSplitUpdate produce_split = 8

  • AcquireIterationClientUpdate acquire_iteration_client = 6

  • ReleaseIterationClientUpdate release_iteration_client = 7

  • GarbageCollectIterationUpdate garbage_collect_iteration = 12

  • RemoveTaskUpdate remove_task = 11

  • CreatePendingTaskUpdate create_pending_task = 9

  • ClientHeartbeatUpdate client_heartbeat = 10

  • CreateTaskUpdate create_task = 3

  • FinishTaskUpdate finish_task = 4

Next tag: 3

Used in: GetWorkersResponse

• string address = 1

State of the worker server, exported to improve debuggability. Next tag: 5

Used in: ServerStateExport

• optional experimental.WorkerConfig worker_config = 1

• repeated TaskDef tasks = 2

• repeated int64 finished_task_ids = 3

• repeated int64 deleted_task_ids = 4