package uber.data.kafka.consumer

Get desktop application:
View/edit binary Protocol Buffers messages

ConsumerOffsetInfo contains information about consumer offsets.

Used in: ConsumerOffsetInfoResponse

• optional OffsetInfo offset_info = 1

  offset_info contains information about the offsets for a given topic.

• string consumer_group = 2

  consumer_group is the consumer group name that this ConsumerGroupInfo is being reported for.

• int64 committed_offset = 3

  committedOffset is the last committed offset for this consumer group. This may be -1 if there are no consumer offsets checkpointed.

• int64 uncommitted_count = 4

  uncommitted_count is the number of messages for this topic-partition that has yet to be committed. On worker rebalance or restart, this is the number of message that will be received. This is calculated as highwatermark - committedOffset - 1.

Request and Response objects. ConsumerOffsetInfoRequest is the request that will be made to POST /getConsumerOffsetInfo in aakafkaoffsetmgmt.

• string cluster_name = 1

• string group_name = 2

• string topic_name = 3

• ConsumerOffsetStore offset_store = 4

ConsumerOffsetInfoResponse is the response that will be returned on POST /getConsumerOffsetInfo in aakafkaoffsetmgmt.

• repeated ConsumerOffsetInfo consumer_offset_info = 1

ConsumerOffsetStore is an enum for place where consumer offsets are stored.

Used in: ConsumerOffsetInfoRequest, GroupInfo, GroupSubscriptionRequest, OffsetSyncRequest, OffsetSyncResponse

• CONSUMER_OFFSET_STORE_INVALID = 0

• CONSUMER_OFFSET_STORE_KAFKA_8 = 1

• CONSUMER_OFFSET_STORE_KAFKA_10 = 2

• CONSUMER_OFFSET_STORE_SELF_MANAGE = 3

  consumer self manage their kafka offset store

DLQMetadata contains metadata from the original kafka message used in messaging-consumer-go lib and /data/messaging/clients:consumer The metadata will be encoded and decoded when sending or receiving messages from the DLQ cluster in order to present the library user a seamless logical topic.

• int64 retry_count = 1

  retry_count is an incrementing value denoting the number of times a message has been redelivered. It will be 0 on first delivery.

• string topic = 2

  topic is the original kafka topic the message was received on. This is analogous to the logical topic name.

• int32 partition = 3

  partition is the original kafka partition the message was received on.

• int64 offset = 4

  offset is the record offset of the original message in the original topic-partition.

• int64 timestamp_ns = 5

  timestamp_ns is the original record timestamp of the original message.

• bytes data = 6

  data is a byte buffer for storing arbitrary information. This is useful if the Kafka Broker version used is < 0.11 and hence Kafka native record headers (KAFKA-4208) are unavailable so the DLQ metadata must be stored in the record Key or Value.

• int64 timeout_count = 7

  timeout_count is an incrementing value denoting the number of times a message has been redelivered due to timeout. It will be 0 on first delivery.

Used in: ListGroupResponse

• string group_name = 1

  group_name is the consumer group id

• string cluster_name = 2

• ConsumerOffsetStore offset_store = 3

  the offset store (KAFAK10, KAFKA8) that the group is shown

GroupSubscriptionRequest contains the parameters for making a group subscription curl call to offset management

• string cluster_name = 1

  cluster is the kafka cluster.

• string group_name = 2

  group is the consumer group

• ConsumerOffsetStore offset_store = 3

GroupSubscriptionResponse contains the group subscription response object returned by rest endpoint of offset management.

• string cluster_name = 1

  cluster is the kafka cluster.

• string group_name = 2

  group is the consumer group

• repeated string topics = 3

  topics is an array of topics subscribed by the group

ListGroupRequest contains the parameters for making a listGroup curl call to offset management

• string cluster_name = 1

  cluster is the kafka cluster.

ListGroupResponse contains listGroup response object returned by rest endpoint of offset management.

• repeated GroupInfo group_list = 1

  an array of groupAndOffsetStore instances

Used in: OffsetSyncResponse

• MAPPING_STATE_INVALID = 0

• MAPPING_STATE_STALE_SRC = 1

  low watermark = high watermark or offset not in [lowwatermark, highwatermark] for all partition

• MAPPING_STATE_NO_SNAPSHOT_MAPPING = 2

  no offset snapshots to utilize for mapping

• MAPPING_STATE_SMART_MERGE = 3

  existing dst offsets > src mapped offsets for all partitions

• MAPPING_STATE_NO_ERROR = 4

  no error

OffsetInfo contains information about offsets.

Used in: ConsumerOffsetInfo

• string cluster = 1

  cluster is the kafka cluster.

• string topic = 2

  topic is the kafka topic.

• int32 partition = 3

  partition is the kafka partition.

• int64 low_watermark = 4

  low_watermark is the offset of the first message that is currently available on the broker.

• int64 high_watermark = 5

  high_watermark is the offset of the next message produced to the broker. note: the offset of the latest mesage available on the broker is highwatermark - 1.

• string src_cluster_name = 1

  src kloak cluster

• string dst_cluster_name = 2

  dst kloak cluster

• string src_group_name = 3

  consumer group name on the src cluster

• string dst_group_name = 4

  consumer group name on the dst cluster

• ConsumerOffsetStore src_offset_store = 5

  offset store from the src cluster

• ConsumerOffsetStore dst_offset_store = 6

  offset store from the dst cluster

• string topic_name = 7

  the topic to migrate.

• repeated PartitionOffsetInfo src_offsets = 8

  user provided partition offsets from the src cluster.

• bool disable_smart_merge = 9

  disable checking existing offsets on the dst cluster and simply use the translated values from offset snapshot

OffsetSyncResponse contains the offset sync response returned by rest endpoint of offset management. the purpose is to make the user aware of what offset values are being used at src and dst cluster respectively.

• string src_cluster_name = 1

• string dst_cluster_name = 2

• string src_group_name = 3

  consumer group name from the srcCluster

• string dst_group_name = 4

  consumer group name from the dstCluster

• string topic_name = 5

  topic to sync

• ConsumerOffsetStore src_offset_store = 6

  consumer kafka version from the srcCluster

• ConsumerOffsetStore dst_offset_store = 7

  consumer kafka version from the dstCluster

• MappingState status = 8

  mapping status indicates the offset state from the src cluster and snapshots

• map<int32, PartitionOffsetInfo> src_offset_infos = 9

  partition offset_infos from the src cluster

• map<int32, PartitionOffsetInfo> dst_offset_infos = 10

  partition offsets from the dst cluster

partition and its associated offset info

Used in: OffsetSyncRequest, OffsetSyncResponse

• int64 offset = 1

  by default UNSET commit offset value is -1. should be explictly set in the builder since proto default value is 0

• int64 low_watermark = 2

  partition beginning offset

• int64 high_watermark = 3

  partition ending offset

• int64 lag = 4

  message lags between offset and highWatermark