package tws.proto.jobmaster

Get desktop application:
View/edit binary Protocol Buffers messages

Used in: JobEvent

• oneof required

  • int32 numberOfWorkers = 1

• repeated WorkerInfo workerInfo = 2

Used in: JobEvent

• oneof typeRequired

  • BarrierType barrierType = 1

• oneof resultRequired

  • BarrierResult result = 2

All workers in a job wants to wait in a Barrier point When BarrierRequest message is received from all workers in the job, Job Master will send response messages to all previous request messages are queued until the last request message is received failures: if the barrier times out, barrier fails if a worker fails in the job, the default barrier fails if there is any

• oneof idRequired

  • int32 workerID = 1

• oneof typeRequired

  • BarrierType barrierType = 2

• oneof timeoutRequired

  • int64 timeout = 3

• oneof required

  • int32 workerID = 1

• oneof typeRequired

  • BarrierType barrierType = 2

• oneof resultRequired

  • BarrierResult result = 3

possible results for barriers

Used in: BarrierDone, BarrierResponse

• SUCCESS = 0

• TIMED_OUT = 1

• JOB_FAULTY = 2

Used in: BarrierDone, BarrierRequest, BarrierResponse

• DEFAULT = 0

• INIT = 1

Driver to workers communication related messages let the the workers know that Driver to worker message Driver can send any byte array to workers including serialized protocol buffer messages It is encoded as a byte array

• bytes data = 1

• oneof onlyOneEvent

  • WorkerFailed failed = 1

  • WorkerRestarted restarted = 2

  • AllJoined allJoined = 3

  • BarrierDone barrierDone = 4

  • JobMasterRestarted jmRestarted = 5

  • JobScaled jobScaled = 6

Used in: JobEvent

• int32 numberOfWorkers = 1

• string jmAddress = 2

job master can be any one of these states job master does not go from STARTING to RUNNING, it just goes to STARTED since enum types must be unique in a proto, I added JM as a prefix to this enum.

• JM_STARTED = 0

• JM_RESTARTED = 1

• JM_COMPLETED = 2

• JM_FAILED = 3

• JM_KILLED = 4

the number of workers in the job has hanged if change is positive, new workers added if change is negative, some worker instances removed numberOfWorkers shows the updated numberOfWorkers in the job

Used in: JobEvent

• oneof changeRequired

  • int32 change = 1

• oneof numberOfWorkersRequired

  • int32 numberOfWorkers = 2

Get the list of wokers in a job from the job master

• oneof idRequired

  • int32 workerID = 1

• oneof typeRequired

  • ListWorkersRequest.RequestType requestType = 2

Used in: ListWorkersRequest

• IMMEDIATE_RESPONSE = 0

• RESPONSE_AFTER_ALL_JOINED = 1

Job master returns the list of workers in a job to a worker

• int32 numberOfWorkers = 1

• repeated WorkerInfo worker = 2

Used in: WorkerInfo

• string nodeIP = 1

• string rackName = 2

• string dataCenterName = 3

this message is sent when the worker is about to start it registers itself with the job master its status in job master becomes: STARTING when this message is received

• oneof idRequired

  • int32 workerID = 1

• oneof restartCountRequired

  how many times this worker is restarted zero means it is starting for the first time 1 means it is restarted one time

  • int32 restartCount = 2

• oneof workerInfoRequired

  • WorkerInfo workerInfo = 3

• oneof idRequired

  • int32 workerID = 1

• oneof resultRequired

  • bool result = 2

• string reason = 3

Used in: JobEvent

• oneof idRequired

  • int32 workerID = 1

Information about a worker in a job

Used in: htgjob.SubGraph, AllJoined, ListWorkersResponse, RegisterWorker, WorkerRestarted

• oneof required

  • int32 workerID = 1

• string workerIP = 2

• int32 port = 3

• optional NodeInfo nodeInfo = 4

• optional job.ComputeResource computeResource = 5

• map<string, int32> additionalPort = 6

Worker to Driver message Worker can send any byte array to the driver including serialized protocol buffer messages

• bytes data = 1

• oneof required

  • int32 workerID = 2

WorkerToDriver response message if it fails, reason will have a string explaining the problem

• oneof required

  • bool succeeded = 1

• string reason = 2

Used in: JobEvent

• oneof workerInfoRequired

  • WorkerInfo workerInfo = 1

a worker can be any one of these states

Used in: WorkerStateChange, WorkerStateChangeResponse

• STARTED = 0

• RESTARTED = 1

• COMPLETED = 2

• FAILED = 3

• ERROR = 4

• UNASSIGNED = 5

• KILLED = 6

• FULLY_FAILED = 7

  fully failed. will not try to restart. job should fail.

a worker informs the master that its state has changed this message is not sent for STARTING state, it is covered by RegisterWorker message this message is sent for states: RUNNING and COMPLETED later on, we may add more states

• oneof idRequired

  • int32 workerID = 1

• oneof stateRequired

  • WorkerState state = 2

sent by the master to the worker to inform that its state change message has been received

• oneof idRequired

  • int32 workerID = 1

• oneof stateRequired

  • WorkerState state = 2