Proto commits in uber/uForwarder

These 15 commits are when the Protocol Buffers files have changed:

2025-10-07

[kafka] Refactor BatchJobCreator to support job creation with partition offsets Summary: So the job can be precisely created with partition offset. Timestamps will be ignored with the existance of the partition offsets. Reviewers: #ldap_owner_kafka_monorepo_blocking, #ldap_owner_kafka_monorepo_nonblocking, ureview, sil Reviewed By: #ldap_owner_kafka_monorepo_blocking, #ldap_owner_kafka_monorepo_nonblocking, sil Subscribers: sil Tags: #auto-approver-analyzed JIRA Issues: KAFEP-7884 Differential Revision: https://code.uberinternal.com/D17493893

KCP report issue retry without retry queue Summary: KCP report issue retry without retry queue Test Plan: unit test Reviewers: #ldap_owner_kafka_monorepo_blocking, #ldap_owner_kafka_monorepo_nonblocking, hf Reviewed By: #ldap_owner_kafka_monorepo_blocking, #ldap_owner_kafka_monorepo_nonblocking, hf JIRA Issues: KAFEP-7630 Differential Revision: https://code.uberinternal.com/D17458615

fix precision problem in latencyTracker Summary: TimeUnit.SECONDS.toNanos() has bad precision that caused health check accuracy problem Test Plan: unit test Reviewers: #ldap_owner_kafka_monorepo_blocking, #ldap_owner_kafka_monorepo_nonblocking, ureview, qichao Reviewed By: #ldap_owner_kafka_monorepo_blocking, #ldap_owner_kafka_monorepo_nonblocking, qichao Subscribers: qichao JIRA Issues: KAFEP-7631 Differential Revision: https://code.uberinternal.com/D17973201

export auto-scalar snapshot Summary: Scalar of data-transfer framework support snapshot() method, and autoscalar implemented the method to export internal state for analysis purpose the internal state can be analyzed in order to optimize performance/efficiency of auto-scalar Test Plan: unit test dump snapshot on dev Reviewers: #ldap_owner_kafka_monorepo_blocking, #ldap_owner_kafka_monorepo_nonblocking, ureview, hf Reviewed By: #ldap_owner_kafka_monorepo_blocking, #ldap_owner_kafka_monorepo_nonblocking, hf Tags: #autoland JIRA Issues: KAFEP-8114 Differential Revision: https://code.uberinternal.com/D18401477

Autoscalar support calibration of new jobs Summary: When job was new added or quota updated, KCP will reset job's scale by quota. this job scale usually is not correctly sized, so over time, KCP autoscaler will resize it but it will take days of time. incorrect size of workload put capacity under risk. so this change introduced a new calibration state which will last about 30 mins after job creation or quota update. during the 30mins, job scale will be calibrated fast to find correct size test shows scale calibration finished in about 25 mins after quota update {F1178824079} Test Plan: unit test dev test Reviewers: #ldap_owner_kafka_monorepo_blocking, #ldap_owner_kafka_monorepo_nonblocking, ureview, sil Reviewed By: #ldap_owner_kafka_monorepo_blocking, #ldap_owner_kafka_monorepo_nonblocking, sil JIRA Issues: KAFEP-8323 Differential Revision: https://code.uberinternal.com/D19497093

The documentation is generated from this commit.

[kafka/KCP] Add ScaleStatus to AddJobGroupRequest Summary: We are going to revert the change in D17310079 because it's very weird to have 2 ScaleStatus in the StoredJobGroup. We combine the code and IDL in the single diff so we know how it works, and it also guarantees that the IDL change won't break. Test Plan: UT Add Request in Studio: https://api-explorer.uberinternal.com/b1ea5804-8e3a-44fb-847b-4979acdeccb8 Get response after add {F1072387239} Reviewers: #ldap_owner_kafka_monorepo_blocking, #ldap_owner_kafka_monorepo_nonblocking, ureview, hf Reviewed By: #ldap_owner_kafka_monorepo_blocking, #ldap_owner_kafka_monorepo_nonblocking, hf Tags: #backward_incompatible_idl JIRA Issues: KAFEP-7768 Differential Revision: https://code.uberinternal.com/D17311729

KCP report CPU usage metrics through heartbeat Summary: KCP will switch workload scale compute model from throughput based to CPU usage based. this commit just allow worker to send CPU usage info to controller through heartbeat next phase is control use the metrics to compute scale Test Plan: unit test Reviewers: #ldap_owner_kafka_monorepo_blocking, #ldap_owner_kafka_monorepo_nonblocking, ureview, hf Reviewed By: #ldap_owner_kafka_monorepo_blocking, #ldap_owner_kafka_monorepo_nonblocking, hf Subscribers: hf JIRA Issues: KAFEP-8187 Differential Revision: https://code.uberinternal.com/D18733329

KCP calculateDownScaleWindowDuration Summary: implement calculateDownScaleWindowDuration of ReactiveScaleWindowCalculator.java use AIMD method to adjust down scale window. when system load is low, increase down scale window duration additively with nature time slowly when system load is high, reduce down scale window duration aggressively with multiply function. the reduction rate depends on prediction of down scale window. - prediction was done by collecting all potential down scale windows, and sort by mature time form low to high, - find the smallest mature time that can meet the need of load reduction - subtract the search result from down scale window size for adjustment Note, the feature is still under development, won't actually functional yet Test Plan: unit test Reviewers: #ldap_owner_kafka_monorepo_blocking, #ldap_owner_kafka_monorepo_nonblocking, ureview, yulan Reviewed By: #ldap_owner_kafka_monorepo_blocking, #ldap_owner_kafka_monorepo_nonblocking, yulan JIRA Issues: KAFEP-8199 Differential Revision: https://code.uberinternal.com/D18870943

2024-12-06

[Kafka/uForwarder]Add job_pod to indicate the pod where the job consumes from update commit id

pull latest proto

2024-04-23

Add consumer service Up/uOwn name in metrics Summary: This diff adds the owning service name into throughput and quota metrics. This Up/uOwn service name is for UAE to identify the service for sending alerts to. It is not related to the RPC service name, which is solely used by the Go consumer services to identify the requests to its GoFx gRPC server.

2024-03-06

integrate from fievel commit da1ae6f5b764d6ad31936a058dac73aff7ac557e JIRA Issues: KAFEP-437 Differential Revision: https://code.uberinternal.com/D13148325

2024-01-08

Chenz/migration (#3) * config maven repos * cleanup build files * buildKite relase to artifactory job * integrate from fievel uforworwarder gradle repo * integrate from fievel commit 7e777eb6a5f481faed9920af89cb880cd477dd91 * cleanup fievel bash

integrate from fievel commit 7e777eb6a5f481faed9920af89cb880cd477dd91

2023-12-01

Copy uforwarder from microrepo to monorepo