package tensorflow.data

Get desktop application:
View/edit binary Protocol Buffers messages

• rpc CreateJob (CreateJobRequest, CreateJobResponse)

  master.proto:120

  Creates a job for reading from the tf.data service.

  message CreateJobRequest

  master.proto:51

  • int64 dataset_id = 1

    The id of the dataset to create a job for.

  • ProcessingModeDef processing_mode = 2

    A mode controlling how the tf.data service produces data for the job.

  message CreateJobResponse

  master.proto:58

  • int64 job_id = 1

    An id for the created job.

• rpc GetOrCreateJob (GetOrCreateJobRequest, GetOrCreateJobResponse)

  master.proto:117

  Gets a job if it already exists, otherwise creates it.

  message GetOrCreateJobRequest

  master.proto:63

  • int64 dataset_id = 1

    The id of the dataset to create a job for.

  • ProcessingModeDef processing_mode = 2

    A mode controlling how the tf.data service produces data for the job.

  • string job_name = 3

    A name for the job.

  • int64 job_name_index = 4

    An index for the job. Multiple jobs can be created for the same name, if they have different indices.

  message GetOrCreateJobResponse

  master.proto:75

  • int64 job_id = 1

    The id of the (potentially newly created) job.

• rpc GetOrRegisterDataset (GetOrRegisterDatasetRequest, GetOrRegisterDatasetResponse)

  master.proto:113

  Registers a dataset with the server, or returns its id if it is already registered. The dataset is constructed in a new graph, so it must not refer to external resources or variables.

  message GetOrRegisterDatasetRequest

  master.proto:34

  • optional DatasetDef dataset = 1

    The dataset to register.

  message GetOrRegisterDatasetResponse

  master.proto:39

  • int64 dataset_id = 1

    The id for the registered dataset.

• rpc GetTasks (GetTasksRequest, GetTasksResponse)

  master.proto:123

  Reports a list of all tasks for a job.

  message GetTasksRequest

  master.proto:80

  • int64 job_id = 1

    The job to look up tasks for.

  message GetTasksResponse

  master.proto:92

  • repeated TaskInfo task_info = 1

    A list of all tasks for a job.

  • bool job_finished = 2

    Whether the job has finished. An empty `task_info` list could either mean that no tasks have been started yet, or that all tasks have finished. This field gives us a way to tell the difference.

• rpc RegisterWorker (RegisterWorkerRequest, RegisterWorkerResponse)

  master.proto:103

  Registers a worker with the master.

  message RegisterWorkerRequest

  master.proto:7

  • string worker_address = 1

    The address of the registering worker.

  message RegisterWorkerResponse

  master.proto:12

  • int64 worker_id = 1

    An id for the worker.

  • repeated TaskDef tasks = 2

    Tasks to begin processing.

• rpc WorkerUpdate (WorkerUpdateRequest, WorkerUpdateResponse)

  master.proto:106

  Updates the master with information about the worker's state.

  message WorkerUpdateRequest

  master.proto:26

  • int64 worker_id = 1

    The worker id that the update is for.

  • repeated TaskProgress updates = 2

  message WorkerUpdateResponse

  master.proto:32

  (message has no fields)

• rpc GetElement (GetElementRequest, GetElementResponse)

  worker.proto:30

  Gets the next dataset element.

  message GetElementRequest

  worker.proto:13

  • int64 task_id = 1

    The task to fetch an element from.

  message GetElementResponse

  worker.proto:18

  • optional CompressedElement compressed_element = 3

    The produced element.

  • bool end_of_sequence = 2

    Boolean to indicate whether the iterator has been exhausted.

• rpc ProcessTask (ProcessTaskRequest, ProcessTaskResponse)

  worker.proto:27

  Processes an task for a dataset, making elements available to clients.

  message ProcessTaskRequest

  worker.proto:7

  • optional TaskDef task = 1

  message ProcessTaskResponse

  worker.proto:11

  (message has no fields)

Used in: CompressedElement

• DataType dtype = 1

  The dtype of the component tensor.

• optional TensorShapeProto tensor_shape = 2

  The shape of the component tensor.

• int64 tensor_size_bytes = 3

  Size of the uncompressed tensor bytes. For tensors serialized as TensorProtos, this is TensorProto::BytesAllocatedLong(). For raw Tensors, this is the size of the buffer underlying the Tensor.

Used in: GetElementResponse

• bytes data = 1

  Compressed tensor bytes for all components of the element.

• repeated ComponentMetadata component_metadata = 2

  Metadata for the components of the element.

Used in: GetOrRegisterDatasetRequest, TaskDef

• optional GraphDef graph = 1

  We represent datasets as tensorflow GraphDefs which define the operations needed to create a tf.data dataset.

Used in: CreateJobRequest, GetOrCreateJobRequest

• PARALLEL_EPOCHS = 0

  Each tf.data worker processes an entire epoch.

• ONE_EPOCH = 1

  Processing of an epoch is distributed across all tf.data workers.

Used in: ProcessTaskRequest, RegisterWorkerResponse

• optional DatasetDef dataset = 1

  The dataset to iterate over. TODO(aaudibert): load the dataset from disk instead of passing it here.

• int64 dataset_id = 2

• int64 task_id = 3

• int64 job_id = 4

Used in: GetTasksResponse

• string worker_address = 1

  The address of the worker processing the task.

• int64 id = 2

  The task id.

Used in: WorkerUpdateRequest

• int64 task_id = 1

  The task that this message is about.

• bool completed = 2

  Whether the task has completed.