package tensorflow.data

Get desktop application:
View/edit binary Protocol Buffers messages

• rpc CreateJob (CreateJobRequest, CreateJobResponse)

  dispatcher.proto:148

  Creates a job for reading from the tf.data service.

  message CreateJobRequest

  dispatcher.proto:60

  • int64 dataset_id = 1

    The id of the dataset to create a job for.

  • ProcessingModeDef processing_mode = 2

    A mode controlling how the tf.data service produces data for the job.

  message CreateJobResponse

  dispatcher.proto:67

  • int64 job_client_id = 1

    An id for the client that will read from the job. When the client is done with the job, they should call ReleaseJobClient with this id.

• rpc GetDatasetDef (GetDatasetDefRequest, GetDatasetDefResponse)

  dispatcher.proto:131

  Gets a dataset defintion.

  message GetDatasetDefRequest

  dispatcher.proto:32

  • int64 dataset_id = 1

  message GetDatasetDefResponse

  dispatcher.proto:36

  • optional DatasetDef dataset_def = 1

• rpc GetOrCreateJob (GetOrCreateJobRequest, GetOrCreateJobResponse)

  dispatcher.proto:145

  Gets a job if it already exists, otherwise creates it.

  message GetOrCreateJobRequest

  dispatcher.proto:73

  • int64 dataset_id = 1

    The id of the dataset to create a job for.

  • ProcessingModeDef processing_mode = 2

    A mode controlling how the tf.data service produces data for the job.

  • string job_name = 3

    A name for the job.

  • int64 job_name_index = 4

    An index for the job. Multiple jobs can be created for the same name, if they have different indices.

  message GetOrCreateJobResponse

  dispatcher.proto:85

  • int64 job_client_id = 1

    An id for the client that will read from the job. When the client is done with the job, they should call ReleaseJobClient with this id.

• rpc GetOrRegisterDataset (GetOrRegisterDatasetRequest, GetOrRegisterDatasetResponse)

  dispatcher.proto:141

  Registers a dataset with the server, or returns its id if it is already registered. The dataset is constructed in a new graph, so it must not refer to external resources or variables.

  message GetOrRegisterDatasetRequest

  dispatcher.proto:50

  • optional DatasetDef dataset = 1

    The dataset to register.

  message GetOrRegisterDatasetResponse

  dispatcher.proto:55

  • int64 dataset_id = 1

    The id for the registered dataset.

• rpc GetSplit (GetSplitRequest, GetSplitResponse)

  dispatcher.proto:134

  Gets the next split for a given job.

  message GetSplitRequest

  dispatcher.proto:40

  • int64 job_id = 1

  • int64 repetition = 2

  message GetSplitResponse

  dispatcher.proto:45

  • optional TensorProto split = 1

  • bool end_of_splits = 2

• rpc GetTasks (GetTasksRequest, GetTasksResponse)

  dispatcher.proto:155

  Reports a list of all tasks for a job.

  message GetTasksRequest

  dispatcher.proto:97

  • int64 job_client_id = 1

    The job client id to look up tasks for.

  message GetTasksResponse

  dispatcher.proto:102

  • repeated TaskInfo task_info = 1

    A list of all tasks for a job.

  • bool job_finished = 2

    Whether the job has finished. An empty `task_info` list could either mean that no tasks have been started yet, or that all tasks have finished. This field gives us a way to tell the difference.

• rpc GetWorkers (GetWorkersRequest, GetWorkersResponse)

  dispatcher.proto:158

  Reports a list of all workers registered with the dispatcher.

  message GetWorkersRequest

  dispatcher.proto:116

  (message has no fields)

  message GetWorkersResponse

  dispatcher.proto:118

  • repeated WorkerInfo workers = 1

    A list of all workers.

• rpc ReleaseJobClient (ReleaseJobClientRequest, ReleaseJobClientResponse)

  dispatcher.proto:151

  Releases a job client so that a job may eventually be cleaned up.

  message ReleaseJobClientRequest

  dispatcher.proto:91

  • int64 job_client_id = 1

  message ReleaseJobClientResponse

  dispatcher.proto:95

  (message has no fields)

• rpc WorkerHeartbeat (WorkerHeartbeatRequest, WorkerHeartbeatResponse)

  dispatcher.proto:125

  Performs a periodic worker heartbeat.

  message WorkerHeartbeatRequest

  dispatcher.proto:15

  • string worker_address = 1

  • repeated int64 current_tasks = 2

  message WorkerHeartbeatResponse

  dispatcher.proto:20

  • repeated TaskDef new_tasks = 1

  • repeated int64 tasks_to_delete = 2

• rpc WorkerUpdate (WorkerUpdateRequest, WorkerUpdateResponse)

  dispatcher.proto:128

  Updates the dispatcher with information about the worker's state.

  message WorkerUpdateRequest

  dispatcher.proto:25

  • string worker_address = 1

  • repeated TaskProgress updates = 2

  message WorkerUpdateResponse

  dispatcher.proto:30

  (message has no fields)

• rpc GetElement (GetElementRequest, GetElementResponse)

  worker.proto:38

  Gets the next dataset element.

  message GetElementRequest

  worker.proto:14

  • int64 task_id = 1

    The task to fetch an element from.

  message GetElementResponse

  worker.proto:19

  • optional CompressedElement compressed_element = 3

    The produced element.

  • bool end_of_sequence = 2

    Boolean to indicate whether the iterator has been exhausted.

• rpc GetWorkerTasks (GetWorkerTasksRequest, GetWorkerTasksResponse)

  worker.proto:41

  Gets the tasks currently being executed by the worker.

  message GetWorkerTasksRequest

  worker.proto:27

  Named GetWorkerTasks to avoid conflicting with GetTasks in dispatcher.proto

  (message has no fields)

  message GetWorkerTasksResponse

  worker.proto:29

  • repeated TaskInfo tasks = 1

• rpc ProcessTask (ProcessTaskRequest, ProcessTaskResponse)

  worker.proto:35

  Processes an task for a dataset, making elements available to clients.

  message ProcessTaskRequest

  worker.proto:8

  • optional TaskDef task = 1

  message ProcessTaskResponse

  worker.proto:12

  (message has no fields)

Used in: Update

• int64 job_id = 1

• int64 job_client_id = 2

Metadata describing a compressed component of a dataset element.

Used in: CompressedElement

• DataType dtype = 1

  The dtype of the component tensor.

• optional TensorShapeProto tensor_shape = 2

  The shape of the component tensor.

• int64 tensor_size_bytes = 3

  Size of the uncompressed tensor bytes. For tensors serialized as TensorProtos, this is TensorProto::BytesAllocatedLong(). For raw Tensors, this is the size of the buffer underlying the Tensor.

Used in: GetElementResponse

• bytes data = 1

  Compressed tensor bytes for all components of the element.

• repeated CompressedComponentMetadata component_metadata = 2

  Metadata for the components of the element.

Used in: Update

• int64 job_id = 1

• int64 dataset_id = 2

• ProcessingModeDef processing_mode = 3

• optional NamedJobKeyDef named_job_key = 4

  Only some jobs have names, so this may be unset.

Used in: Update

• int64 task_id = 1

• int64 job_id = 2

• int64 dataset_id = 3

• ProcessingModeDef processing_mode = 5

• string worker_address = 4

Used in: GetDatasetDefResponse, GetOrRegisterDatasetRequest, TaskDef

• optional GraphDef graph = 1

  We represent datasets as tensorflow GraphDefs which define the operations needed to create a tf.data dataset.

Used in: Update

• int64 task_id = 1

Used in: CreateJobUpdate

• string name = 1

• int64 index = 2

Used in: CreateJobRequest, CreateJobUpdate, CreateTaskUpdate, GetOrCreateJobRequest, TaskDef

• INVALID = 0

• PARALLEL_EPOCHS = 1

  Each tf.data worker processes an entire epoch.

• DISTRIBUTED_EPOCH = 2

  Processing of an epoch is distributed across all tf.data workers.

Used in: Update

• int64 dataset_id = 1

• uint64 fingerprint = 2

Used in: Update

• string worker_address = 1

Used in: Update

• int64 job_client_id = 1

• int64 time_micros = 2

  The time when the client was released, measured in microseconds since the epoch.

Used in: ProcessTaskRequest, WorkerHeartbeatResponse

• oneof dataset

  The dataset to iterate over.

  • DatasetDef dataset_def = 1

  • string path = 2

• int64 dataset_id = 3

• int64 task_id = 4

• int64 job_id = 5

• ProcessingModeDef processing_mode = 6

Used in: GetTasksResponse, GetWorkerTasksResponse

• string worker_address = 1

  The address of the worker processing the task.

• int64 task_id = 2

  The task id.

• int64 job_id = 3

  The id of the job that the task is part of.

Used in: WorkerUpdateRequest

• int64 task_id = 1

  The task that this message is about.

• bool completed = 2

  Whether the task has completed.

Message representing journaled dispatcher metadata updates. When we apply one of these changes to the dispatcher's in-memory state, we also write an Update message to the journal.

• oneof update_type

  • RegisterDatasetUpdate register_dataset = 1

  • RegisterWorkerUpdate register_worker = 5

  • CreateJobUpdate create_job = 2

  • AcquireJobClientUpdate acquire_job_client = 6

  • ReleaseJobClientUpdate release_job_client = 7

  • CreateTaskUpdate create_task = 3

  • FinishTaskUpdate finish_task = 4

Used in: GetWorkersResponse

• string address = 1

• int64 id = 2