package uber.data.kafka.datatransfer

Get desktop application:
View/edit binary Protocol Buffers messages

MasterAdminService is implemented by the Master for managing the cluster. This allows an external service to control the state of the system.

• rpc AddJobGroup (AddJobGroupRequest, AddJobGroupResponse)

  master_admin_service.proto:20

  AddJobGroup creates a new JobGroup. An error is thrown if this job group already exist.

  message AddJobGroupRequest

  master_admin_service.proto:40

  AddJobGroupRequest runs a new job group with job group definition and configuration provided in the request. A JobGroup is a group of Jobs. e.g., for Kafka, JobGroup = Topic, Job = Partition.

  • optional JobGroup job_group = 1

  • JobState job_group_state = 2

  • optional ScaleStatus scale_status = 3

  message AddJobGroupResponse

  master_admin_service.proto:47

  The JobGroup that is registered to the system.

  • optional StoredJobGroup group = 1

• rpc DeleteJobGroup (DeleteJobGroupRequest, DeleteJobGroupResponse)

  master_admin_service.proto:28

  DeleteJobGroup deletes the JobGroup from storage. This differs from updating state to CANCEL which stops the job but retains the job group information in storage. Deleted Job Groups are not recoverable!

  message DeleteJobGroupRequest

  master_admin_service.proto:75

  DeleteJobGroupRequest is sent to delete job group from storage.

  • string id = 1

  message DeleteJobGroupResponse

  master_admin_service.proto:81

  DeleteJobGroupResponse is sent back in response to a delete request. Any non-error GRPC response is considered success.

  (message has no fields)

• rpc GetAllJobGroups (GetAllJobGroupsRequest, stream GetAllJobGroupsResponse)

  master_admin_service.proto:33

  GetAllJobGroups is a server stream that returns all job groups. We use streams b/c gRPC has a 1MB single payload limit for large clusters a unary request may not work.

  message GetAllJobGroupsRequest

  master_admin_service.proto:94

  GetAllJobGroupsRequest is sent to request all job groups.

  (message has no fields)

  message GetAllJobGroupsResponse

  master_admin_service.proto:98

  GetAllJobGroupsResponse is sent back with the job groups.

  • optional JobGroup job_group = 1

  • JobState job_group_state = 2

  • optional ScaleStatus scale_status = 3

• rpc GetClusterScaleStatus (GetClusterScaleStatusRequest, GetClusterScaleStatusResponse)

  master_admin_service.proto:35

  GetClusterScaleStatus returns the scale status for the entire cluster.

  message GetClusterScaleStatusRequest

  master_admin_service.proto:105

  GetClusterScaleStatusRequest is sent to get the scale status of the cluster.

  (message has no fields)

  message GetClusterScaleStatusResponse

  master_admin_service.proto:108

  GetClusterScaleStatusResponse is sent back with the scale status of the cluster.

  • uint32 total_jobgroup_count = 1

    the total number of job groups in the cluster

  • uint32 total_worker_count = 2

    the total number of workers in the cluster

  • double total_allowed_messages_per_sec = 3

    the combined messages per second of all the workers in the cluster

  • double total_allowed_bytes_per_sec = 4

    the combined bytes per second of all the workers in the cluster

  • double total_used_messages_per_sec = 5

    the combined messages per second of all the job groups in the cluster

  • double total_used_bytes_per_sec = 6

    the combined bytes per second of all the job groups in the cluster

• rpc GetJobGroup (GetJobGroupRequest, GetJobGroupResponse)

  master_admin_service.proto:30

  GetJobGroup returns the JobGroup for the specified id.

  message GetJobGroupRequest

  master_admin_service.proto:84

  GetJobGroupRequest is sent to get a job group from storage.

  • string id = 1

  message GetJobGroupResponse

  master_admin_service.proto:89

  GetJobGroupResponse is sent back in response to a get request.

  • optional StoredJobGroup group = 1

• rpc UpdateJobGroup (UpdateJobGroupRequest, UpdateJobGroupResponse)

  master_admin_service.proto:22

  UpdateJobGroupConfiguration updates the mutable JobConfiguration for a JobGroup.

  message UpdateJobGroupRequest

  master_admin_service.proto:52

  UpdateJobGroupRequest updates the (mutable) Job configuration for the specified JobGroup.

  • optional JobGroup job_group = 1

  message UpdateJobGroupResponse

  master_admin_service.proto:57

  UpdateJobGroupConfigurationResponse contains the update JobGroup.

  • optional StoredJobGroup group = 1

• rpc UpdateJobGroupState (UpdateJobGroupStateRequest, UpdateJobGroupStateResponse)

  master_admin_service.proto:24

  UpdateJobGroupState updates the expected state for the JobGroup.

  message UpdateJobGroupStateRequest

  master_admin_service.proto:64

  UpdateJobGroupStateRequest changes the expected state for the specified job group. Note: this updates the expected state but does not necessarily block until the distributed workers pick up the jobs.

  • string id = 1

  • JobState state = 2

  message UpdateJobGroupStateResponse

  master_admin_service.proto:70

  UpdateJobGroupStateResponse is the job group after the expected state has been changed.

  • optional StoredJobGroup group = 1

MasterWorkerService is the GRPC service exposed by the master for the worker to contact. Each worker is required to register itself and maintain an active heartbeat for liveness and to receive work assignment.

• rpc Heartbeat (HeartbeatRequest, HeartbeatResponse)

  master_worker_service.proto:24

  Heartbeat must be sent by a worker regularly to maintain liveness and lease.

  message HeartbeatRequest

  master_worker_service.proto:47

  HeartbeatRequest is sent by the worker to the master to indicate liveness. If the master does not receive a successful heartbeat request in a configurable amount of time, it should consider the worker dead and rebalance the job to other workers as necessary. The HeartbeatRequest also contains a progress report on all running job that the worker has previously received. The master may use this information to rebalance the workload as it sees fit.

  • optional Participants participants = 1

  • repeated JobStatus job_status = 2

  message HeartbeatResponse

  master_worker_service.proto:55

  HeartbeatResponse is sent by the controller to the worker to tell how it should command the jobs running at the worker. The controller compares how it expects the jobs on the worker to run and how they actually run, and derives the commands for the jobs according to the matrix of CommandType.

  • optional Participants participants = 1

  • repeated Command commands = 2

• rpc RegisterWorker (RegisterWorkerRequest, RegisterWorkerResponse)

  master_worker_service.proto:22

  RegisterWorker is used by a new worker to register itself to receive work.

  message RegisterWorkerRequest

  master_worker_service.proto:30

  RegisterWorkerRequest is sent by the worker when it believes that it is not an active member of the cluster. A worker should set the Participants set as it best understands them. The master will validate that the Participants set is in a valid state when responding.

  • optional Participants participants = 1

  message RegisterWorkerResponse

  master_worker_service.proto:38

  RegisterWorkerResponse is sent by the master to the worker to acknowledge that it is successfully registered as a worker in a cluster The participants set may have modified data (specifically the node id). The worker is required to update its internal participants information to match the updated one provided by the master. The correct participants set must be sent in all subsequent requests.

  • optional Participants participants = 1

Audit configuration contains the configurations for individual audit tasks

Used in: AuditMetaData

• AuditType audit_type = 1

  Type of audit to be executed as part of the task

• int32 audit_interval_in_seconds = 2

  Frequency of the audit in seconds

• oneof config

  Config that will be populated based on the audit type

  • CrossDCAuditConfig cross_dc_audit_config = 3

    Configurations for cross DC auditing

  • ReplicationAuditConfig replication_audit_config = 4

    Configurations for auditing replication pipelines

  • RetentionAuditConfig retention_audit_config = 5

    Configurations for retention auditing

Audit metadata contains the configurations for the different types of auditing to be performed

Used in: AuditTask, KafkaConsumerTask, KafkaConsumerTaskGroup

• repeated AuditConfig audit_configs = 1

  a list of audit configs

AuditTask is a specification for audit tasks in Uatu

Used in: Job

• string cluster = 1

  cluster to audit

• string topic = 2

  topic to audit

• int32 partition = 3

  partition to audit

• optional AuditMetaData audit_metadata = 4

  Auditing related metadata

AuditTaskGroup is a specification for audit tasks in Uatu

Used in: JobGroup

• string cluster = 1

  cluster to audit

• string topic = 2

  topic to audit

Represents the different types of audit that can be done in KafkaAuditTask

Used in: AuditConfig

• AUDIT_TYPE_INVALID = 0

  0 enum must be invalid per t.uber.com/protobuf-style

• AUDIT_TYPE_EXACT_UNIQ = 1

  Type of audit where Kafka message count and latency are computed for a given duration. In addition, unique message count is calculated precisely by de-duping the messages

• AUDIT_TYPE_APPROX_UNIQ = 2

  Type of audit where Kafka message count and latency are computed for a given duration. In addition, unique message count is calculated approximately

• AUDIT_TYPE_RETENTION = 3

  Type of audit where the topic retention is audited for any violations

• AUDIT_TYPE_AVAILABILITY = 4

  Type of audit where topic availability is monitored. e.g offline partitions

• AUDIT_TYPE_SERVICE_AUDIT_AGGREGATION = 5

  Audit type for aggregating service local audits

• AUDIT_TYPE_KRP_AUDIT_AGGREGATION = 6

  Audit type for aggregating KRP local audits

• AUDIT_TYPE_XDC = 7

  Audit type for comparing cross dc message count

• AUDIT_TYPE_REPLICATION = 8

  Audit type for monitoring replication pipelines - e.g regional to agg

AutoOffsetResetPolicy defines which offset to start to read when 1. the specified start_offset does not exist on the Kafka server, or 2. start_offset is not specified and there is no previously-committed offset on the Kafka server.

Used in: KafkaConsumerTask, KafkaConsumerTaskGroup

• AUTO_OFFSET_RESET_POLICY_INVALID = 0

  0 enum must be invalid per t.uber.com/protobuf-style

• AUTO_OFFSET_RESET_POLICY_EARLIEST = 1

  automatically reset the offset to the earliest offset.

• AUTO_OFFSET_RESET_POLICY_LATEST = 2

  automatically reset the offset to the latest offset.

Used in: AvailabilityTask, AvailabilityTaskGroup

• AVAILABILITY_JOB_TYPE_INVALID = 0

  0 enum must be invalid per t.uber.com/protobuf-style

• AVAILABILITY_JOB_TYPE_NATIVE_PRODUCER = 1

  the type of job to monitor by producing natively

• AVAILABILITY_JOB_TYPE_NATIVE_CONSUMER = 2

  the type of job to monitor by consuming natively

• AVAILABILITY_JOB_TYPE_PROXY_PRODUCER = 3

  the type of job to monitor by producing via proxy

• AVAILABILITY_JOB_TYPE_PROXY_CONSUMER = 4

  the type of job to monitor by consuming via proxy

Used in: Job

• AvailabilityJobType availability_job_type = 1

  the type of job Kafka Availability to perform to monitor a service

• bool zone_isolated = 2

  whether the availability task should be zone aware. should be set for proxy producer or proxy consumer with zone isolation

AvailabilityTaskGroup is a task group spec for Kafka Availability

Used in: JobGroup

• AvailabilityJobType availability_job_type = 1

  the type of job Kafka Availability to perform to monitor a service

• bool zone_isolated = 2

  whether the availability task should be zone aware. should be set for proxy producer or proxy consumer with zone isolation

Command is an instruction that is sent from master to worker to change the worker behavior.

Used in: HeartbeatResponse

• CommandType type = 1

  The type of command that is being issued.

• optional Job job = 2

  Details for the job to run.

CommandType of the command that is being issued. The matrix of commands is: +---------------+----------------+----------------+----------------+----------------+----------------+----------------+----------------------------------------+ | - | Expected State | null | INVALID | UNIMPLEMENTED | FAILED | CANCELED | RUNNING | +---------------+----------------+----------------+----------------+----------------+----------------+----------------+----------------------------------------+ | Actual State | - | - | - | - | - | - | - | +---------------+----------------+----------------+----------------+----------------+----------------+----------------+----------------------------------------+ | null | - | N/A | N/A | N/A | N/A | N/A | COMMAND_RUN | +---------------+----------------+----------------+----------------+----------------+----------------+----------------+----------------------------------------+ | INVALID | - | N/A | N/A | N/A | N/A | N/A | COMMAND_RUN | +---------------+----------------+----------------+----------------+----------------+----------------+----------------+----------------------------------------+ | UNIMPLEMENTED | - | N/A | N/A | N/A | N/A | N/A | COMMAND_RUN | +---------------+----------------+----------------+----------------+----------------+----------------+----------------+----------------------------------------+ | FAILED | - | N/A | N/A | N/A | N/A | N/A | COMMAND_RUN | +---------------+----------------+----------------+----------------+----------------+----------------+----------------+----------------------------------------+ | CANCELED | - | N/A | N/A | N/A | N/A | N/A | COMMAND_RUN | +---------------+----------------+----------------+----------------+----------------+----------------+----------------+----------------------------------------+ | RUNNING | - | COMMAND_CANCEL | COMMAND_CANCEL | COMMAND_CANCEL | COMMAND_CANCEL | COMMAND_CANCEL | COMMAND_UPDATE | | | | | | | | | if configuration change | +---------------+----------------+----------------+----------------+----------------+----------------+----------------+----------------------------------------+

Used in: Command

• COMMAND_TYPE_INVALID = 0

  0 enum must be invalid per t.uber.com/protobuf-style

• COMMAND_TYPE_RUN_JOB = 1

  This is a command to run a job.

• COMMAND_TYPE_CANCEL_JOB = 2

  This command cancels a job. A CANCELED job cannot be restarted.

• COMMAND_TYPE_UPDATE_JOB = 3

  This command updates a job.

Represents specific audit configurations for cross DC auditing

Used in: AuditConfig

• optional TopicsToCompare cross_dc_checks = 1

• bool send_to_udq = 2

• double failure_threshold = 3

• string udq_test_name = 4

Used in: DebugJobsTable

• optional StoredJob job = 1

• string job_group_id = 2

• string pipeline_id = 3

• string rpc_debug_url = 4

• string job_partition = 5

Used in: DebugJobStatusTable

• string pipeline_id = 1

• optional JobStatus job_status = 2

• repeated DebugJobStatusRow data = 1

• repeated DebugJobRow data = 1

Used in: DebugMasterTable

• string host_port = 1

• string role = 2

• repeated DebugMasterRow data = 1

Used in: DebugMessageStubTable

• string pipeline_id = 1

• int32 partition = 2

• int64 offset = 3

• repeated string events = 4

• repeated DebugMessageStubRow data = 1

• string host = 1

• string role = 2

• string service = 3

Used in: DebugWorkersTable

• optional StoredWorker worker = 1

• int64 expected_job_count = 2

• string url = 3

• string routing_destination = 4

• double total_message_per_sec = 5

• double total_load = 6

• repeated DebugWorkerRow data = 1

EncodedFormatInfo configures the encoded format of the messages

Used in: KafkaDispatcherTask, KafkaDispatcherTaskGroup

• EncodedFormatType encoded_format_type = 1

  encoded format type

• int32 schema_version = 2

  schema version of the encoded format

EncodedFormatType defines the type to encode data

Used in: EncodedFormatInfo

• ENCODED_FORMAT_TYPE_INVALID = 0

  Per http://t.uber.com/protobuf-style, enum must start with INVALID=0

• ENCODED_FORMAT_TYPE_AVRO = 1

• ENCODED_FORMAT_TYPE_PROTOBUF = 2

Flow describes how the data should flow.

Used in: Job, JobGroup, ResqConfig

• double messages_per_sec = 1

  messages_per_sec constraint.

• double bytes_per_sec = 2

  bytes_per_sec constraint.

• double max_inflight_messages = 3

  max_inflight_messages constraint.

To do exactly-once delivery in multiple partitions, Kafka guarantees atomic transactions. 1. read uncommitted messages 2. only read committed messages

Used in: KafkaConsumerTask, KafkaConsumerTaskGroup

• ISOLATION_LEVEL_INVALID = 0

  0 enum must be invalid per t.uber.com/protobuf-style

• ISOLATION_LEVEL_UNSET = 1

  no value present

• ISOLATION_LEVEL_READ_UNCOMMITTED = 2

  read all messages including uncommitted

• ISOLATION_LEVEL_READ_COMMITTED = 3

  read only transaction committed messages

The Job is the unit of work that needs to be done.

Used in: Command, JobStatus, StoredJob

• int64 job_id = 1

  job_id is a globally unique id for the job item. This is set by the master when job is created.

• JobType type = 2

  JobType is an enumeration for the type of job to be run. This allows the master-worker protocol to generalize to multiple types of workload that needs sharding. JobType shall be used to determine the set of valid tasks that contains specific information about the job. The mapping of JobType to Tasks is: - JOB_TYPE_KAFKA_CONSUMER_TO_RPC_DISPATCHER: KafkaConsumerTask and RpcDispatcherTask.

• optional FlowControl flow_control = 3

  FlowControl defines how the data should flow (msgs/s, bytes/s inflight_msgs).

• optional KafkaConsumerTask kafka_consumer_task = 4

  KafkaConsumerTask is populated for job that is involves Kafka Consumption. Specifically, this is non-null for: - JOB_TYPE_KAFKA_CONSUMER_TO_RPC_DISPATCHER

• optional RpcDispatcherTask rpc_dispatcher_task = 5

  RpcDispatcherTask is populated for job that is involves RPC dispatching. Specifically, this is non null for: - JOB_TYPE_KAFKA_CONSUMER_TO_RPC_DISPATCHER

• optional SecurityConfig security_config = 6

  SecurityConfig is populated for job that intends to use message security feature It contains service identity that will be used for authz check. It also stores the flag which decides if the job should be run securely.

• optional RetryConfig retry_config = 7

  RetryConfig is populated for job that involves one or more retry queues. It contains instructions on entry conditions for retry queues. It also stores the flag controlling if this feature is turned on or not, and how the retry count is calculated.

• optional KafkaDispatcherTask kafka_dispatcher_task = 8

  KafkaDispatcherTask is populated for job that involves Kafka replication. Specifically, this is non-null for: - JOB_TYPE_KAFKA_REPLICATION and - JOB_TYPE_LOAD_GEN_PRODUCE

• optional MiscConfig misc_config = 9

  MiscConfig contains the non-work-related, miscellaneous configs of a job

• optional AuditTask audit_task = 10

  Audit task is populated for job that involves auditing in Uatu SLA monitor. Whenever audit task is populated, the job type would be set to JOB_TYPE_KAFKA_AUDIT

• optional ResqConfig resq_config = 11

  ResqConfig is populated for a job's resilience configuration It contains resilience queue configuration and a flag controlling if resilience queue should be enabled or not

• optional AvailabilityTask availability_task = 12

  AvailabilityTask should be populated for Kafka Availability jobs

• optional ReplicationTask replication_task = 13

  ReplicationTask should be populated for Kafka Replication/Ingestion jobs

• optional google.protobuf.Any extension = 14

  The extensible field, used for 3rdParty application

Used in: AddJobGroupRequest, GetAllJobGroupsResponse, StoredJobGroup, UpdateJobGroupRequest

• string job_group_id = 1

  Job group id is a unique id for this job group. We use string b/c we want the znode to be /jobgroup/<string> in order to leverage zookeeper optimistic locking to ensure uniqueness of logical job group.

• JobType type = 2

  JobType is an enumeration for the type of job to be run. This allows the master-worker protocol to generalize to multiple types of workload that needs sharding. JobType shall be used to determine the set of valid tasks that contains specific information about the job. The mapping of JobType to Tasks is: - JOB_TYPE_KAFKA_CONSUMER_TO_RPC_DISPATCHER: KafkaConsumerTask and RpcDispatcherTask.

• optional FlowControl flow_control = 3

  Flow defines how the data should flow (msgs/s, bytes/s inflight_msgs).

• optional KafkaConsumerTaskGroup kafka_consumer_task_group = 4

  KafkaConsumerTaskGroup is populated for job that is involves Kafka Consumption. Specifically, this is non-null for: - JOB_TYPE_KAFKA_CONSUMER_TO_RPC_DISPATCHER

• optional RpcDispatcherTaskGroup rpc_dispatcher_task_group = 5

  RpcDispatcherTaskGroup is populated for job that is involves RPC dispatching. Specifically, this is non null for: - JOB_TYPE_KAFKA_CONSUMER_TO_RPC_DISPATCHER

• optional SecurityConfig security_config = 6

  SecurityConfig is populated for job that intends to use message security feature. It contains service identity that will be used for authz check. It also stores the flag which decides if the job should be run securely.

• optional RetryConfig retry_config = 7

  RetryConfig is populated for job that involves one or more retry queues. It contains instructions on entry conditions for retry queues. It also stores the flag controlling if this feature is turned on or not, and how the retry count is calculated.

• optional KafkaDispatcherTaskGroup kafka_dispatcher_task_group = 8

  KafkaDispatcherTaskGroup is populated for job that involves Kafka produce Specifically, this is non-null for: - JOB_TYPE_KAFKA_REPLICATION and - JOB_TYPE_LOAD_GEN_PRODUCE

• optional MiscConfig misc_config = 9

  MiscConfig contains the miscellaneous configs of a jobGroup

• optional AuditTaskGroup audit_task_group = 10

  Audit task is populated for job that involves auditing in Uatu SLA monitor. Whenever audit task is populated, the job type would be set to JOB_TYPE_KAFKA_AUDIT

• optional ResqConfig resq_config = 11

  ResqConfig is populated for a job's resilience configuration It contains resilience queue configuration and a flag controlling if resilience queue should be enabled or not

• optional AvailabilityTaskGroup availability_task_group = 12

  AvailabilityTaskGroup is populated for jobs related to Kafka Availability service Its job type should be JOB_TYPE_KAFKA_AVAILABILITY when populated

• optional ReplicationTaskGroup replication_task_group = 13

  ReplicationTaskGroup is populated for job groups that involve Kafka replication/ingestion Its job type should be JOB_TYPE_KAFKA_REPLICATION when populated

• optional google.protobuf.Any extension = 14

  The extensible field, used for 3rdParty application

Used in: ScaleStoreSnapshot

• string cluster = 1

  cluster to read from.

• string topic = 2

  topic to read from.

• string consumer_group = 3

  consumer_group to read on behalf of and/or checkpoint for.

• optional ScaleStateSnapshot scale_state_snapshot = 4

  scale state snapshots

JobSnapshot is an aggregated view of the job from both the master and worker's perspective.

• optional StoredJob expected_job = 1

  Job is the expected definition/configuration for this job from the master's perspective.

• optional StoredJobStatus actual_job_status = 2

  JobStatus is the last reported configuration of the job from the worker's perspective.

JobState is the current state of the worker. If you add or remove any state, please update the matrix in master_worker_service.proto and CommandListBuilder.java.

Used in: AddJobGroupRequest, GetAllJobGroupsResponse, JobStatus, StoredJob, StoredJobGroup, UpdateJobGroupStateRequest

• JOB_STATE_INVALID = 0

  0 enum must be invalid per t.uber.com/protobuf-style

• JOB_STATE_UNIMPLEMENTED = 1

  UNIMPLEMENTED signals that the command or job type is not supported by this worker.

• JOB_STATE_FAILED = 2

  FAILED signals a worker side failure of the job. A worker may determine that a job is failed independent of any command issued by the master. This is the ONLY JobState the worker may determine independently of a COMMAND from the master. All other JobStates must be in response to a COMMAND from the master. This is a terminal state.

• JOB_STATE_RUNNING = 3

  RUNNING is used for jobs that actively running in a stable state.

• JOB_STATE_CANCELED = 4

  CANCELED is used for jobs for which COMMAND_TYPE_CANCEL was received by master. This is a terminal state. A canceled job does not necessarily imply cancelation due to failure.

JobStatus is used by the worker to report the current status of job it was assigned.

Used in: DebugJobStatusRow, HeartbeatRequest, StoredJobStatus

• optional Job job = 1

  Job that is registered.

• JobState state = 2

  JobState is the current state of the job. See comments above each enumeration for the description of each JobState.

• optional KafkaConsumerTaskStatus kafka_consumer_task_status = 3

  Consumer Task Status contains task state when job_type = JOB_TYPE_KAFKA_CONSUMER*.

• optional KafkaDispatcherTaskStatus kafka_dispatcher_task_status = 4

  Dispatcher Task Status contains task state when job_type = JOB_TYPE_LOAD_GEN_PRODUCE

JobType is an enumeration for the type of job to be run. This allows the master-worker protocol to generalize to multiple types of workload that needs sharding.

Used in: Job, JobGroup

• JOB_TYPE_INVALID = 0

  0 enum must be invalid per t.uber.com/protobuf-style

• JOB_TYPE_KAFKA_CONSUMER_TO_RPC_DISPATCHER = 1

  KAFKA_CONSUMER_TO_RPC_DISPATHCER is a job item that should consume from Kafka and dispatch to RPC. This is used by Consumer Proxy.

• JOB_TYPE_KAFKA_AUDIT = 2

  KAFKA_AUDIT is a job that consumes messages or other metadata from Kafka for auditing

• JOB_TYPE_KAFKA_REPLICATION = 3

  KAFKA_REPLICATION is a job item that should consume from one Kafka queue and produce to another queue. This is used by the Replication Service.

• JOB_TYPE_LOAD_GEN_PRODUCE = 4

  Load generator producer job to produce records to the broker for stress/spike test

• JOB_TYPE_LOAD_GEN_CONSUME = 5

  Load generator consumer job to consume records from the broker for stress/spike test

• JOB_TYPE_KAFKA_AVAILABILITY = 6

  Indicates a Kafka Availability job

KafkaConsumerTask is a specification for a Kafka Consumption task.

Used in: Job

• string cluster = 1

  cluster to read from.

• string topic = 2

  topic to read from.

• int32 partition = 3

  partition to read from.

• string consumer_group = 4

  consumer_group to read on behalf of and/or checkpoint for.

• AutoOffsetResetPolicy auto_offset_reset_policy = 5

  the auto offset reset policy.

• int64 start_offset = 6

  start_offset to start at when reading. 1. When start_offset >= 0, the consumer tries to read from this offset. If iet does not exist on the Kafka server, then use auto_offset_reset_policy to reset the starting offset. 2. When start_offset < 0, the consumer tries to read from last committed offset. If it does not exist on the Kafka server, then use auto_offset_reset_policy to reset the starting offset.

• int64 end_offset = 7

  end_offset to stop at when reading. end_offset follows Kafka zero-offset semantics so actually refers to the first offset to **not** consume. For example, if end_offset = 100, then the last offset to read and process is 99. 0 for unbounded consumption.

• int32 processing_delay_ms = 8

  a delay processing time for the kafka topic. This is a configuration for the original topic. for retry topics, use processing_delay_ms in RetryConfig.

• IsolationLevel isolation_level = 9

  isolation level could be read_uncommitted or read_committed, default read_uncommitted

• optional AuditMetaData audit_metadata = 10

  Auditing related metadata. It will be set if the job type is `JOB_TYPE_KAFKA_AUDIT`

• int64 committed_offset = 11

  committed_offset is the committed offset of a consumer group to a topic

KafkaConsumerTaskGroup is a specification for a Kafka Consumption Group task.

Used in: JobGroup

• string cluster = 1

  cluster to read from.

• string topic = 2

  topic to read from.

• string consumer_group = 3

  consumer_group to read on behalf of and/or checkpoint for.

• AutoOffsetResetPolicy auto_offset_reset_policy = 4

  the auto offset reset policy.

• optional google.protobuf.Timestamp start_timestamp = 5

  start_timestamp to start at when reading. 1. When start_timestamp >= 0, the consumer tries to read from the earliest offset whose timestamp is greater than or equal to the start_timestamp. If it does not exist on the Kafka server, then use auto_offset_reset_policy to reset the starting offset. 2. When start_timestamp < 0, the consumer tries to read from last committed offset. If it does not exist on the Kafka server, then use auto_offset_reset_policy to reset the starting offset.

• optional google.protobuf.Timestamp end_timestamp = 6

  end_timestamp to stop at when reading. end_timestamp follows Kafka zero-offset semantics so actually refers to the first timestamp to **not** consume.

• int32 processing_delay_ms = 7

  a delay processing time for the kafka topic. This is a configuration for the original topic. for retry topics, use processing_delay_ms in RetryConfig.

• IsolationLevel isolation_level = 8

  isolation level could be read_uncommitted or read_committed, default read_uncommitted

• optional AuditMetaData audit_metadata = 9

  Auditing related metadata. It will be set if the job type is `JOB_TYPE_KAFKA_AUDIT`

• optional PartitionOffsetRanges partition_offset_ranges = 10

  start/end offset for each partition. The partition_offset_ranges, if available, will make the controller ignore the start_timestamp and end_timestamp. That means only the partitions in the partition_offset_ranges will be consumed. New partitions after expansion will not be consumed either.

KafkaConsumerTaskStatus represents the state of a consumer task.

Used in: JobStatus

• int64 read_offset = 1

  Read offset, following Kafka conventions, is the next offset to be read.

• int64 commit_offset = 2

  Commit offset, following Kafka convention, is the next offset to be committed.

• double messages_per_sec = 3

  number of messages per second delivered.

• double bytes_per_sec = 4

  number of bytes per second delivered.

• double messages_inflight = 5

  deprecated, use total_messages_inflight instead

• double bytes_inflight = 6

  deprecated, use total_bytes_inflight instead

• int64 total_messages_inflight = 7

  number of messages inflight to be processed

• int64 total_bytes_inflight = 8

  number of bytes of messages inflight to be processed

• double cpu_usage = 9

  computed cpu usage, in terms of percentage per core

KafkaDispatcherTask is a specification for a Kafka dispatcher task.

Used in: Job

• string cluster = 1

  cluster to produce to.

• string topic = 2

  topic to produce to.

• bool dedup_enabled = 3

  whether deduplication with uDeduplify is enabled or not.

• int32 partition = 4

  partition to produce to.

• bool is_secure = 5

  whether to produce securely

• bool is_acks_one = 6

  true when acks=1/lossy, false when acks=all/lossless

• optional EncodedFormatInfo encoded_format_info = 7

  optional encoded format information

Used in: JobGroup

• string cluster = 1

  cluster to produce to

• string topic = 2

  topic to produce to

• bool dedup_enabled = 3

  whether deduplication with uDeduplify is enabled or not.

• bool is_secure = 4

  whether to produce securely

• bool is_acks_one = 5

  true when acks=1/lossy, false when acks=all/lossless

• optional EncodedFormatInfo encoded_format_info = 6

  optional encoded format information

KafkaDispatcherTaskStatus represents the state of the dispatcher task.

Used in: JobStatus

• int64 num_records_produced = 1

MiscConfig contains the miscellaneous configs of a jobGroup/job

Used in: Job, JobGroup

• bool enable_debug = 1

  enables extra logging for debugging purpose

• string owner_service_name = 2

  the service name which owns this consumer

• bool scale_reset_enabled = 3

  enables job scale reset based on quota. If enabled, each flow control quota update will reset the job scale based on the quota values, otherwise quota update will not reset the job scale.

Node identifies a member of a cluster.

Used in: Participants, StoredWorker

• int64 id = 1

  id is a UUID assigned to each node by the master.

• string host = 2

  host name for the node.

• int32 port = 3

  port for the node if it is reachable over the network. This may be 0 to indicate if it is unset.

• string zone = 4

  zone where the node is located

Participants is a master-worker pair of participants. Participants is included in all master-worker requests so that both parties are clear about who they are communicating and the Node id can be used to as a fencing mechanism.

Used in: HeartbeatRequest, HeartbeatResponse, RegisterWorkerRequest, RegisterWorkerResponse

• optional Node master = 1

• optional Node worker = 2

PartitionOffsetRange is a specification for a single partition offset range.

Used in: PartitionOffsetRanges

• uint32 partition = 1

  the partition index

• uint64 start_offset = 2

  the start offset of the partition

• uint64 end_offset = 3

  the end offset of the partition

PartitionOffsetRanges is a specification for a set of partition offset ranges.

Used in: KafkaConsumerTaskGroup

• repeated PartitionOffsetRange partition_offset_range = 1

  start/end offset for each partition

Represents specific audit configurations for auditing replication pipeline

Used in: AuditConfig

• repeated TopicsToCompare replication_checks = 1

ReplicationTask is the task spec for Kafka Replication/Ingestion jobs

Used in: Job

• bool is_continuous = 1

  whether the source end_offset should be ignored and keep running continuously

ReplicationTaskGroup is a task group spec for Kafka Replication/Ingestion

Used in: JobGroup

• bool is_continuous = 1

  whether the source end_offset should be ignored and keep running continuously

ResqConfig configures the underlying resilience queue for a topic in the consumer

Used in: Job, JobGroup

• string resq_topic = 1

  the resilience queue topic name. an empty resq_topic indicates that the resilience queue feature is disabled.

• string resq_cluster = 2

  the cluster to find the resilience queue topic on.

• optional FlowControl flow_control = 3

  Optional to specify flow control for resilience queue topic If left unspecified, resilience queue will inherit flow control from main topic

• bool resq_enabled = 4

  whether the resilience queue is enabled or not

• bool acks_one_producer = 5

  whether the resilience queue uses acks=1 producer or not

Used in: AuditConfig

• int32 partition_count = 1

RetryConfig configures the retry policy for retriable errors. It's the updated version of RetryConfig(see below).

Used in: Job, JobGroup

• repeated RetryQueue retry_queues = 1

  a list of retry queues

• bool retry_enabled = 2

  whether the whole policy is enabled or not;

RetryQueue configures one retry queue in a set of retry queues

Used in: RetryConfig

• string retry_queue_topic = 1

  retry queue topic. when the recipient service indicates the consumer proxy to retry a message, the message should be produced to the retry queue topic. an empty retry queue topic indicates that it is disabled.

• string retry_cluster = 2

  the cluster to write the retry_queue_topic to.

• int32 processing_delay_ms = 3

  a user configurable delay that is applied to redelivering of retried requests. when retry topic is enabled, the retry delay must fail into [0, 86400000]. i.e, 0ms ~ 1day.

• int32 max_retry_count = 4

  The max retry of the retry queue

RpcDispatcherTask is a specification for a Rpc Dispatcher task.

Used in: Job

• string uri = 1

  uri for resolving the destination service that will receive RPC push messages. We support a subset of standard Uber URI schemes from dev-platform/frameworks/jfx/library/grpc/client/src/main/java/com/uber/jfx/grpc/GrpcChannelBuilder.java#L56: 1. muttley://muttley-routing-key[@service-name] (this is a KCP internal format which is different from standard Muttley format)

• string procedure = 2

  This indicates the procedure name that is used to identify the handler on the callee server. e.g., kafka.consumerproxy.consumer_group/topic.

• int32 rpc_timeout_ms = 3

  when a rpc call take more than rpc_time_out_ms to finish, it is considered failed.

• string retry_queue_topic = 4

  DEPRECATED. Use retry_queue_topic in RetryConfig instead.

• string retry_cluster = 5

  DEPRECATED. Use retry_cluster in RetryConfig instead.

• string dlq_topic = 6

  dlq topic. when the recipient service indicates the consumer proxy to stash a message, the message should be produced to the dlq topic. an empty dlq topic indicates that it is disabled.

• string dlq_cluster = 7

  the cluster to write the dlq_topic to.

• int32 max_rpc_timeouts = 8

  when rpc timeout times exceeds the limit, send message to DLQ if configured the option should be disabled by default, only enable when consumer stuck with rpc timeout

• int32 rpc_timeout_limit_ms = 9

  the max allowed timeout in backoff retry. Even with exponential backoff, the timeout value should not exceed rpc_timeout_limit_ms, if it is configured.

Used in: JobGroup

• string uri = 1

  uri for resolving the destination service that will receive RPC push messages. We support a subset of standard Uber URI schemes from dev-platform/frameworks/jfx/library/grpc/client/src/main/java/com/uber/jfx/grpc/GrpcChannelBuilder.java#L56: 1. muttley://muttley-routing-key[@service-name] (this is a KCP internal format which is different from standard Muttley format)

• string procedure = 2

  This indicates the procedure name that is used to identify the handler on the callee server. e.g., kafka.consumerproxy.consumer_group/topic.

• int32 rpc_timeout_ms = 3

  when a rpc call take more than rpc_time_out_ms to finish, it is considered failed.

• string retry_queue_topic = 4

  DEPRECATED, use the retry_queue_topic for each retry queue in RetryConfig instead

• string retry_cluster = 5

  DEPRECATED, use the retry_cluster for each retry queue in RetryConfig instead

• string dlq_topic = 6

  dlq topic. when the recipient service indicates the consumer proxy to stash a message, the message should be produced to the dlq topic. an empty dlq topic indicates that it is disabled.

• string dlq_cluster = 7

  the cluster to write the dlq_topic to.

• int32 max_rpc_timeouts = 8

  when rpc timeout times exceeds the limit, send message to DLQ if configured the option should be disabled by default, only enable when consumer stuck with rpc timeout

• int32 rpc_timeout_limit_ms = 9

  the max allowed timeout in backoff retry. Even with exponential backoff, the timeout value should not exceed rpc_timeout_limit_ms, if it is configured.

Used in: ScaleStateSnapshot

• oneof impl

  • WindowedComputerSnapshot windowed_computer_snapshot = 1

Used in: JobGroupScaleStatusSnapshot

• repeated ScaleComputerSnapshot scale_computer_snapshots = 1

  scale computers for up/down/hibernate/bootstrap

• double scale = 2

• string name = 3

scale status of job group

Used in: AddJobGroupRequest, GetAllJobGroupsResponse, StoredJobGroup

• double scale = 1

  scale of the job group which indicates number of works needed

• double total_messages_per_sec = 2

  the combined messages per second of all the jobs in the job group

• double total_bytes_per_sec = 3

  the combined bytes per second of all the jobs in the job group

• repeated JobGroupScaleStatusSnapshot job_group_snapshot = 1

• int64 timestamp_nanos = 2

SecurityConfig stores details of the user service identities. It also holds the flag which is used to decide if a job will be run securely.

Used in: Job, JobGroup

• repeated string service_identities = 1

  service_identities will store the SPIFFE Id(principal name) of the user service this will be used to check if the user has the necessary permissions to consume from a topic

• bool is_secure = 2

  is_secure flag will be used to decide if the job should be run securely the job will still be run unsecured(normal flow) if identity is provided but this flag is set to false

• bool multi_tenancy_enabled = 3

  the flag which determines if the job is multi-tenancy enabled

StoredJob is the internal representation of job within storage.

Used in: DebugJobRow, JobSnapshot, StoredJobGroup

• optional google.protobuf.Timestamp last_updated = 1

  The last updated timestamp for this record.

• optional Job job = 2

  Job information.

• JobState state = 3

  The expected state of the job.

• int64 worker_id = 4

  The exepcted worker id for this job.

• double scale = 5

  scale of the job

• string job_pod = 6

  this indicates the pod that the job consumes from

JobGroup is a grouping of jobs that represent a single logical resource. In Kafka, JobGroup -> topic and Job -> topic-partition.

Used in: AddJobGroupResponse, GetJobGroupResponse, UpdateJobGroupResponse, UpdateJobGroupStateResponse

• optional google.protobuf.Timestamp last_updated = 1

  The last updated timestamp for this record.

• optional JobGroup job_group = 2

  The grouped job provides the aggregate expected state for the job group. Note: the partition and job_id fields within the aggregate_job are unusued and must always be set to 0 to indicate unuse.

• JobState state = 3

  state of the job group.

• repeated StoredJob jobs = 4

  The specific (partitioned) job for this job group. Implementations should guarantee uniqueness of jobs.

• optional ScaleStatus scale_status = 5

  scale status of job group for autoScaler

StoredJobStatus is the internal representation of JobStatus report from worker

Used in: JobSnapshot

• optional google.protobuf.Timestamp last_updated = 1

  The last updated timestamp for this record.

• optional JobStatus job_status = 2

  The status of the job as reported by the worker in the last heartbeat.

• int64 worker_id = 3

  The id for the worker current working on this job.

StoreWorker is the internal representation of a worker on the master within storage. We choose to use protobuf encoded Worker so that schema evolution for data written to storage can be seamlessly handled between master upgrades. This shall only be used internally in the master's implementation and shall not be exposed to the worker.

Used in: DebugWorkerRow

• optional google.protobuf.Timestamp last_updated = 1

  The last updated timestamp for this record.

• optional Node node = 2

  The node information for this worker.

• WorkerState state = 3

  The state of this worker.

TopicInfo represents information about a topic

Used in: TopicsToCompare

• string cluster = 1

  Cluster name

• string topic = 2

  Topic name

TopicsToCompare represents the source(LHS) and the destination(RHS) topics that must be compared against

Used in: CrossDCAuditConfig, ReplicationAuditConfig

• repeated TopicInfo source = 1

  List of source topics / clusters - LHS

• repeated TopicInfo destination = 2

  List of destination topics / clusters - RHS

Used in: WindowedComputerSnapshot

• int64 size_in_seconds = 1

• int64 min_size_in_seconds = 2

• int32 size_in_samples = 3

• int32 min_size_in_samples = 4

Used in: ScaleComputerSnapshot

• double lower_boundary = 1

• double upper_boundary = 2

• double percentile = 3

• double percentile_scale = 4

• double base_scale = 5

• optional WindowSnapshot window_snapshot = 6

WorkerState is the state of the worker.

Used in: StoredWorker

• WORKER_STATE_INVALID = 0

  0 enum must be invalid per t.uber.com/protobuf-style

• WORKER_STATE_REGISTERING = 1

  REGISTERING state represents the a worker node that has/is sending RegisterWorker, but has yet to send Heartbeat.

• WORKER_STATE_WORKING = 2

  WORKING state represents workers that has/is stably sending Heartbeat.