Proto commits in google/clusterfuzz

These commits are when the Protocol Buffers files have changed: (only the last 100 relevant commits are shown)

2026-06-16

[Swarming] Adds Response proto & discards tasks with missing dimensions (#5289) ## Overview It was request for me to add a log to easily debug when swarming scheduled a task, but i noticed we didnt had the protos for parsing the response, hence i added them. Also this PR handles an edge case in which swarming will not return a task_id if the tasks wasn't scheduled due to swarming related issues(not request, nor pc contract). #### Background This PR comes from a discussion & agreement to a required change from this [PR](https://github.com/google/clusterfuzz/pull/5277) ## Changes - Adds the missing swarming_task_response proto - Ran `protos/generate.sh` which updated a ton of files - The swarming service now marks tasks as not scheduled when no task_id is found, task_id cannot be found if swarming didn't queue the task, because of wrong dimensions or other swarming internal reasons - As the `count_tasks()` api method, now the `push_task()` will also only raise `SwarmingAPIErrors` --------- Co-authored-by: Titouan Rigoudy <titouan@chromium.org>

The documentation is generated from this commit.

2026-06-10

Rename EngineOutput to FuzzerRunOutput in uworker_msg and fuzz_task (#5310) This PR is related to #5311, which contains the core implementation and full context. Bug: b/487905050 ## Changes Renames the `EngineOutput` protobuf message to `FuzzerRunOutput` and the `engine_outputs` field to `fuzzer_run_outputs`. This generalizes the uworker interface to support deferred logging for both engine-based and blackbox fuzzers. Updates the internal helper functions and variables in `fuzz_task.py` to compile against the new types. This is a syntactic change that only updates the protobuf structure and its references. It does not change any execution or upload logic.

2026-05-26

[blackbox fuzzer logs] defer the blackbox fuzzer logs to postprocess

2026-05-20

Swarming: Add TasksCountRequest and TasksCount to swarming.proto (#5276) As part of the initiative of creating a backpressure logic for the swarming integration we are adding first the required protobuf definitions for a upcoming(in another pr) `/prpc/swarmig.v2.tasks/CountTasks` prpc endpoint. ### Background For more context on the backpressure initative see more info in internal fuzz docs ## Changes - Adds the following protobuf definitions to the `swarming.proto` file: - CountTask message - TasksCountRequest message - StateQuery enum - Updated the reference to the chromium repo in the swarming.proto file - Ran the `src/clusterfuzz/_internal/protos/generate.sh` script, which added a bunch of changes to the generated files. ## Tests No unit tests nor actual tests in the CF code are required for this one, as we are just adding more protobuf definitions not doing nothing new with them(yet),

2026-04-15

Add additional JobRun stats for blackbox fuzzers (#5238) This PR aims to improve our ability to benchmark blackbox fuzzers effectiveness and monitor their health. ## Context When measuring fuzzer effectiveness, we’re interested in the rate at which the fuzzer can execute testcases and find bugs. This requires a baseline definition of total fuzzing hours across all types of fuzzers. Some of these stats are already computed for Monarch monitoring in `monitoring_metrics.py`, but we'd like to get them into BigQuery to have a consistent data source for analysis ## Changes Adds `testcases_generated`, `testcase_execution_duration`, `testcase_generation_duration`, and `fuzzing_duration` to the uworker `FuzzTaskOutput` proto and write those stats to the JobRun BigQuery table. ## Notes This only adds metrics for blackbox fuzzers. We will need to aggregate the fuzzing session hours and execution metrics we already store in the TestcaseRun tables for engine guided fuzzers if we want comparisons across fuzzer types. This either means we need to aggregate those tables and write to the JobRun tables, or do the aggregation in our plx workflows/scripts. ## Testing - [x] Ran the server locally, uploaded a fuzzer, and ran a bot locally with additional logs and local bigquery writes enabled. Verified that the stats were added to the `FuzzTaskOutput` proto and written to BigQuery logs from running a local bot: https://paste.googleplex.com/4551083230887936 local BigQuery stats: https://paste.googleplex.com/6697329928306688 - [x] Test on dev and verify the stats are successfully written to the BigQuery tables dev BigQuery json: https://paste.googleplex.com/4627780642930688 dev BigQuery table https://paste.googleplex.com/6251944436957184 Added unit tests for the stats

2026-04-07

Add total_fuzzing_time_seconds to FuzzTaskOutput

2026-01-18

implement max cpu monitor Signed-off-by: Javan Lacerda <javanlacerda@google.com>

2026-01-08

Copy fuzzer script logs to correct buckets (#5099) Handler for `store_fuzzer_run_results` in postprocess (#3406) was not adding the fuzzer script logs to the correct structured buckets, as expected by the fuzzer page on the UI. This PR tries to fix it by copying the data from the blob store to the expected buckets during postprocess. b/361867455

2025-12-22

Download databundles for non engine jobs (#5085) This fixes b/446692241 by downloading databundles on blacbox fuzzer runs. It does so without hurting the performance of non-blackbox batch tasks. Since downloading the default databundle takes ~5 minutes. It also optimizes the unzip by about 50% by using the linux unzip binary instead of Python's slow implementation. --------- Co-authored-by: google-labs-jules[bot] <161369871+google-labs-jules[bot]@users.noreply.github.com> Co-authored-by: jonathanmetzman <31354670+jonathanmetzman@users.noreply.github.com> Co-authored-by: Jonathan Metzman <metzman@chromium.org>

2025-12-15

tmp

2025-12-08

tmp

2025-11-04

[Fix] Correctly set testcase build url and gn args for fuzzing on batch (#5013) Fix missing testcase metadata when fuzz task is running on batch (or any non-local execution). #### Context Some build-related environment variables (`build_url`, `build_key` and `gn_args_path`) are set during the build setup on fuzz task main stage. These should be retrieved during postprocess to set the testcase's metadata, which is used to display the build url and the gn args config section on the clusterfuzz UI. However, for non-local execution, these variables are not propagated through the `uworker_output` proto and are missing in postprocess, so any testcase created during fuzzing on batch is not displaying these values. #### Changes * Added the `build_url`, `build_key` and `gn_args` fields to the fuzz task output message. * Added the logic to store these fields and use them to set the initial testcase metadata. #### Tests * Testing on development environment is ineffective as it is not currently running fuzz tasks on batch. I deployed to dev anyway and confirmed that, at least, the local execution of fuzz tasks is still working. * Testing on prod (deployed this PR to internal) * Fuzzing hours stable after deploy: https://screenshot.googleplex.com/7JfregzBR8F6MRA.png * Logs for fuzz tasks executing on batch: https://cloudlogging.app.goo.gl/5xXpLjUdwzK6wTeR9 (main) and https://cloudlogging.app.goo.gl/nX832G1782asdMDXA (postprocess) * Logs for fuzz tasks executing on GCE VMs: https://cloudlogging.app.goo.gl/61nvxR1BwQrQnvXr8 (main) and https://cloudlogging.app.goo.gl/6yoQSvU8r1mUVxsN7 (postprocess) * Evidence of testcases creates after the deploy containing the build url and GN config args: * GCE VMs (local exec): https://clusterfuzz.com/testcase-detail/6746169883426816 * Batch: https://clusterfuzz.com/testcase-detail/6451774202249216 * Build URL: https://screenshot.googleplex.com/45n7M75hEPoEsNZ.png * GN Configs: https://screenshot.googleplex.com/9r3SxRPRQRyC4tv.png Bug: b/441128474 Related issue: https://github.com/google/clusterfuzz/issues/4914

2025-10-31

Add build metadata to fuzz task output.

2025-10-20

feat: Implement gRPC API for Test Case Reproduction This commit introduces a new gRPC API for asynchronously reproducing test cases in ClusterFuzz. The API is designed to be a robust and scalable standard for exposing ClusterFuzz capabilities. It includes the following features: - **Asynchronous Job Management:** Clients can start reproduction jobs for multiple test cases and receive a job ID for tracking. - **Persistent State:** Job status is persisted using a simple `dbm` database to survive server restarts. - **Real-time Updates & Polling:** Supports both a server-streaming endpoint (`SubscribeToJobUpdates`) for real-time progress and a polling endpoint (`GetJobStatus`). - **Final Metrics:** A dedicated endpoint (`GetJobResults`) calculates and returns final metrics, such as success and failure percentages. - **Secure by Default:** The API is secured using an API key interceptor that reads valid keys from an environment variable (`VALID_API_KEYS`). The implementation is separated into modules for the server, job management, authentication, and integration with the existing `butler` reproduction logic. A new public function was added to the `butler` script to provide a clean API for this integration.

2025-08-04

[batch] Support trials. (#4890) The old method requires db access in main, which is no longer allowed. Trials are a feature allowing experimental flags to be used in certain chrome and v8 builds. CC @mi-ac

2025-08-01

[batch] Support trials.

2025-07-30

fix

Refactor trial selection and implement hybrid approach with lists

I've moved the trial selection to the preprocessing step.

2025-06-11

Merge OSS-Fuzz branch into master (#4797) This merges the OSS-Fuzz branch into master. The branches diverged when support for fuzzing on batch was added to OSS-Fuzz. Among other things this: Added scheduling of fuzz tasks and made a lot of optimizations to it, in order to deal with challenges filling up our capacity with fuzzing. Optimized things like corpus signing, downloading and pruning as previously these had not been run at OSS-Fuzz scale and levels of diversity. --------- Co-authored-by: Vitor Guidi <vitorguidi@gmail.com> Co-authored-by: Ali HIJAZI <ahijazi@google.com> Co-authored-by: Vinicius da Costa <vtcosta@google.com>

2025-03-13

Use corpus backup as base for corpus. (#4721) Use corpus backup as base for corpus when fuzzing. This reduces time to start fuzzing (including signing the URLs in preprocess and downloading the urls in main) from 5 minutes to 20 seconds in tests I did. It should make downloads even faster than they were before the batch migration. TODO: Do this for pruning.

2024-11-14

Upload crashes to logs bucket in permissionless way (#4400)

2024-10-23

Reapply "Make fuzz task work on a uworker. (#4342)" (#4351) (#4352) This reverts commit https://github.com/google/clusterfuzz/commit/90b87101cee2778bf1cb5a90f606f7273d10571c. 1. Don't assume there is a data bundle. 2. Fix typo.

Revert "Make fuzz task work on a uworker. (#4342)" (#4351) This reverts commit 4a752d6e40bdb06ecfe56cc5b94fe9e6f12b9c9c. Causes these errors: https://pantheon.corp.google.com/errors/detail/CIa_hZWh6JKCHA;locations=global;time=P30D?project=google.com:clusterfuzz&utm_source=error-reporting-notification&utm_medium=email&utm_content=new-error&e=-13802955&mods=logs_tg_prod https://pantheon.corp.google.com/errors/detail/CIa_hZWh6JKCHA;locations=global;time=P30D?project=google.com:clusterfuzz&utm_source=error-reporting-notification&utm_medium=email&utm_content=new-error&e=-13802955&mods=logs_tg_prod

Make fuzz task work on a uworker. (#4342) This has been verified locally. This involved: * Get data bundle directory in an untrusted manner. * Upload fuzzing engine output in an untrusted manner. * Not checking for last corpus sync on a uworker. This is a trusted operation and is uneeded because uworkers do not outlive the task and thus will never have the corpus. * Refactor upload_log so callers aren't exposed to its internals.

Fix permissions issue in analyze uworker_main (#4348) Close testcases in postprocess. Fixes: crbug.com/374738561

2024-10-14

Make sure we serialize objects before putting them in a proto. (#4323) Follow up on: https://github.com/google/clusterfuzz/pull/4316 Fixes: https://pantheon.corp.google.com/errors/detail/CKnhiqX14viPPw;time=PT1H;locations=global?project=google.com:clusterfuzz

Harmonize trusted host and normal corpus pruning code. (#4308) Get rid of CorpusCrash. Use CrashInfo everywhere. Using CrashInfo in some places breaks oss-fuzz corpus pruning when there is a crash in the corpus, because we expect to be using the unit_name which CrashInfo doesn't have. It was kind of crazy to have two totally different objects passed to the same function (only similarity is basically ducktyped).

2024-09-19

Reland get fuzz target in preprocess and fix simple issue assuming all jobs have fuzz target (#4259)

Reapply "[fuzz_task] Get fuzz target weights in preprocess (#4192)" (#4258) This reverts commit ab0ebe16e627caeb4692c0aed70b276afe690082.

Revert "[fuzz_task] Get fuzz target weights in preprocess (#4192)" (#4258) This reverts commit b765e4ad88758522350c8e7c451dbc4be5e139e0. Reverting due to exceptions.

[fuzz_task] Get fuzz target weights in preprocess (#4192)

2024-09-12

Revert "Revert rearch PRs from today. (#4209)" (#4227) This reverts commit 1b8ca0d94266d741bfd1da8eaaba52c40f768f19. This is safe to land because I am around to fix it.

2024-08-26

Add the possibility to push tasks to swarming (#3996)

2024-08-23

Revert rearch PRs from today. (#4209) They should be working, but I be around to fix them.

Fix exception in build_data (#4206)

[fuzz_task][rearch] Handle bad builds in postprocess (#4201)

2024-08-16

Handle issue metadata properly (#4174) Since it's a heterogenous dictionary, we must store it as a string representation of a json dict (as unwieldy as this is, it's the only correct way to do this). Fixes: https://pantheon.corp.google.com/errors/detail/CODtycKwtN-pGA;filter=%5B%22Type%22%5D;time=P30D;locations=global?e=-13802955&invt=AbZgCw&mods=logs_tg_prod&project=google.com:cluster-fuzz ``` Bar chart Sample stack trace Parsed Raw TypeError: bad argument type for built-in operation at .update ( <frozen _collections_abc>:949 ) at .find_fixed_range ( /mnt/scratch0/clusterfuzz/src/clusterfuzz/_internal/bot/tasks/utasks/progression_task.py:567 ) at .utask_main ( /mnt/scratch0/clusterfuzz/src/clusterfuzz/_internal/bot/tasks/utasks/progression_task.py:671 ) at .uworker_main_no_io ( /mnt/scratch0/clusterfuzz/src/clusterfuzz/_internal/bot/tasks/utasks/__init__.py:211 ) at .execute_locally ( /mnt/scratch0/clusterfuzz/src/clusterfuzz/_internal/bot/tasks/task_types.py:63 ) at .execute ( /mnt/scratch0/clusterfuzz/src/clusterfuzz/_internal/bot/tasks/task_types.py:127 ) at .run_command ( /mnt/scratch0/clusterfuzz/src/clusterfuzz/_internal/bot/tasks/commands.py:218 ) at .process_command_impl ( /mnt/scratch0/clusterfuzz/src/clusterfuzz/_internal/bot/tasks/commands.py:429 ) at .wrapper ( /mnt/scratch0/clusterfuzz/src/clusterfuzz/_internal/bot/tasks/commands.py:159 ) at .process_command ( /mnt/scratch0/clusterfuzz/src/clusterfuzz/_internal/bot/tasks/commands.py:248 ) at .task_loop ( /mnt/scratch0/clusterfuzz/src/python/bot/startup/run_bot.py:146 ) ```

Prevent integer overflow by using int64 instead of int32 (#4184) Fixes: ``` ValueError: Value out of range: 2148226322 at ._extract_coverage_information ( /mnt/scratch0/bots/oss-fuzz-linux-zone5-host-qp4x-2/clusterfuzz/src/clusterfuzz/_internal/bot/tasks/utasks/corpus_pruning_task.py:978 ) at .utask_main ( /mnt/scratch0/bots/oss-fuzz-linux-zone5-host-qp4x-2/clusterfuzz/src/clusterfuzz/_internal/bot/tasks/utasks/corpus_pruning_task.py:1029 ) ```

Ensure archived field is sent to utask_main (#4180) Follow up to https://github.com/google/clusterfuzz/pull/4173 Fixes https://pantheon.corp.google.com/errors/detail/CNDympvhu7XrzwE;locations=global;time=P30D?project=clusterfuzz-external&utm_source=error-reporting-notification&utm_medium=email&utm_content=new-error&e=-13802955&mods=logs_tg_prod

Ensure archived field is sent to utask_main (#4180) Follow up to https://github.com/google/clusterfuzz/pull/4173 Fixes https://pantheon.corp.google.com/errors/detail/CNDympvhu7XrzwE;locations=global;time=P30D?project=clusterfuzz-external&utm_source=error-reporting-notification&utm_medium=email&utm_content=new-error&e=-13802955&mods=logs_tg_prod

2024-08-15

Fix AttributeError: 'NoneType' object has no attribute 'corpus_pruning_task_input' (#4168) This was caused by the roll forward of oss-fuzz.

Fix AttributeError: 'NoneType' object has no attribute 'corpus_pruning_task_input' (#4168) This was caused by the roll forward of oss-fuzz.

Propagate fuzz_targets for RemoteRegularBuild. (#4166) This is required for OSS-Fuzz after the uworker refactor. This fixes the following exception: ``` Error occurred while working on task. Traceback (most recent call last): File "/mnt/scratch0/bots/oss-fuzz-linux-zone1-host-bsw9-13/clusterfuzz/src/python/bot/startup/run_bot.py", line 146, in task_loop commands.process_command(task) File "/mnt/scratch0/bots/oss-fuzz-linux-zone1-host-bsw9-13/clusterfuzz/src/clusterfuzz/_internal/bot/tasks/commands.py", line 249, in process_command task.high_end, task.is_command_override) File "/mnt/scratch0/bots/oss-fuzz-linux-zone1-host-bsw9-13/clusterfuzz/src/clusterfuzz/_internal/bot/tasks/commands.py", line 159, in wrapper return func(task_name, task_argument, job_name, *args, **kwargs) File "/mnt/scratch0/bots/oss-fuzz-linux-zone1-host-bsw9-13/clusterfuzz/src/clusterfuzz/_internal/bot/tasks/commands.py", line 433, in process_command_impl preprocess) File "/mnt/scratch0/bots/oss-fuzz-linux-zone1-host-bsw9-13/clusterfuzz/src/clusterfuzz/_internal/bot/tasks/commands.py", line 218, in run_command result = task.execute(task_argument, job_name, uworker_env) File "/mnt/scratch0/bots/oss-fuzz-linux-zone1-host-bsw9-13/clusterfuzz/src/clusterfuzz/_internal/bot/tasks/task_types.py", line 105, in execute self.execute_locally(task_argument, job_type, uworker_env) File "/mnt/scratch0/bots/oss-fuzz-linux-zone1-host-bsw9-13/clusterfuzz/src/clusterfuzz/_internal/bot/tasks/task_types.py", line 63, in execute_locally uworker_output = utasks.uworker_main_no_io(self.module, uworker_input) File "/mnt/scratch0/bots/oss-fuzz-linux-zone1-host-bsw9-13/clusterfuzz/src/clusterfuzz/_internal/bot/tasks/utasks/__init__.py", line 211, in uworker_main_no_io uworker_output = utask_module.utask_main(uworker_input) File "/mnt/scratch0/bots/oss-fuzz-linux-zone1-host-bsw9-13/clusterfuzz/src/clusterfuzz/_internal/bot/tasks/utasks/fuzz_task.py", line 1919, in utask_main return session.run() File "/mnt/scratch0/bots/oss-fuzz-linux-zone1-host-bsw9-13/clusterfuzz/src/clusterfuzz/_internal/bot/tasks/utasks/fuzz_task.py", line 1731, in run self.fuzz_task_output.fuzz_targets.extend(build_setup_result.fuzz_targets) TypeError: Value must be iterable ```

Propagate fuzz_targets for RemoteRegularBuild. (#4166) This is required for OSS-Fuzz after the uworker refactor. This fixes the following exception: ``` Error occurred while working on task. Traceback (most recent call last): File "/mnt/scratch0/bots/oss-fuzz-linux-zone1-host-bsw9-13/clusterfuzz/src/python/bot/startup/run_bot.py", line 146, in task_loop commands.process_command(task) File "/mnt/scratch0/bots/oss-fuzz-linux-zone1-host-bsw9-13/clusterfuzz/src/clusterfuzz/_internal/bot/tasks/commands.py", line 249, in process_command task.high_end, task.is_command_override) File "/mnt/scratch0/bots/oss-fuzz-linux-zone1-host-bsw9-13/clusterfuzz/src/clusterfuzz/_internal/bot/tasks/commands.py", line 159, in wrapper return func(task_name, task_argument, job_name, *args, **kwargs) File "/mnt/scratch0/bots/oss-fuzz-linux-zone1-host-bsw9-13/clusterfuzz/src/clusterfuzz/_internal/bot/tasks/commands.py", line 433, in process_command_impl preprocess) File "/mnt/scratch0/bots/oss-fuzz-linux-zone1-host-bsw9-13/clusterfuzz/src/clusterfuzz/_internal/bot/tasks/commands.py", line 218, in run_command result = task.execute(task_argument, job_name, uworker_env) File "/mnt/scratch0/bots/oss-fuzz-linux-zone1-host-bsw9-13/clusterfuzz/src/clusterfuzz/_internal/bot/tasks/task_types.py", line 105, in execute self.execute_locally(task_argument, job_type, uworker_env) File "/mnt/scratch0/bots/oss-fuzz-linux-zone1-host-bsw9-13/clusterfuzz/src/clusterfuzz/_internal/bot/tasks/task_types.py", line 63, in execute_locally uworker_output = utasks.uworker_main_no_io(self.module, uworker_input) File "/mnt/scratch0/bots/oss-fuzz-linux-zone1-host-bsw9-13/clusterfuzz/src/clusterfuzz/_internal/bot/tasks/utasks/__init__.py", line 211, in uworker_main_no_io uworker_output = utask_module.utask_main(uworker_input) File "/mnt/scratch0/bots/oss-fuzz-linux-zone1-host-bsw9-13/clusterfuzz/src/clusterfuzz/_internal/bot/tasks/utasks/fuzz_task.py", line 1919, in utask_main return session.run() File "/mnt/scratch0/bots/oss-fuzz-linux-zone1-host-bsw9-13/clusterfuzz/src/clusterfuzz/_internal/bot/tasks/utasks/fuzz_task.py", line 1731, in run self.fuzz_task_output.fuzz_targets.extend(build_setup_result.fuzz_targets) TypeError: Value must be iterable ```

2024-08-05

Remove unused proto field (#4150)

Reland "Making crashes uworker safe (#4102)" (#4137) This reverts commit 8c2f3bcebd061dfdd40470ee2e2ddab6bf6637e1.

2024-07-05

Revert making crashes uworker safe (#4102)

2024-07-03

Add back used field (#4101)

Fix mismatch between Crash and proto version (#4097)

Remove unused field (#4096)

Dont serialize unneeded field (#4092)

Revert "Make fuzz crashes uworker safe (#4000)" This reverts commit 4a17b2f82cbfcfef979991df635251d9c22ac4b9.

2024-07-02

Make fuzz crashes uworker safe (#4000) This is probably the most important part of migrating fuzz_task. This superseeds: #3834 and #3816

2024-06-28

Attempting to bump proto/grpcio in Python 3.7 - 3rd attempt (#4066) ### Motivation Last attempted was [reverted](https://github.com/google/clusterfuzz/pull/4064) due to an issue in utask main scheduler, related to protos and gcp batch This PR attempts to fix the following issue: ``` AttributeError: Unknown field for InstancePolicy: boot_disk at .__setattr__ ( /mnt/scratch0/clusterfuzz/src/third_party/proto/message.py:932 ) at ._get_allocation_policy ( /mnt/scratch0/clusterfuzz/src/clusterfuzz/_internal/google_cloud_utils/batch.py:182 ) at ._create_job ( /mnt/scratch0/clusterfuzz/src/clusterfuzz/_internal/google_cloud_utils/batch.py:225 ) at .create_uworker_main_batch_jobs ( /mnt/scratch0/clusterfuzz/src/clusterfuzz/_internal/google_cloud_utils/batch.py:139 ) at .schedule_utask_mains ( /mnt/scratch0/clusterfuzz/src/python/bot/startup/run_bot.py:105 ) at .task_loop ( /mnt/scratch0/clusterfuzz/src/python/bot/startup/run_bot.py:136 ) ``` ### Testing strategy This issue was caused because batch was set to 0.2.0, and the boot_disk proto field was only [introduced in 0.6.0](https://cloud.google.com/python/docs/reference/batch/latest/changelog). Returning the dependency to 0.11.0 solved the issue. This was tested manually through [use_batch.py](https://github.com/google/clusterfuzz/blob/bcae6b19f399a74ca105404284a2cad8e174627e/src/clusterfuzz/_internal/tests/core/google_cloud_utils/use_batch.py#L4), pointing to our internal gcp service. Evidence that a batch job is successfully created:  

2024-06-27

Revert "Chore/py37 proto bump reattempt (#4062)" (#4064) This reverts commit 7e57e8821d35ec2b12c957239ae75c9bf33964f7. New error: ``` AttributeError: Unknown field for InstancePolicy: boot_disk at .__setattr__ ( /mnt/scratch0/clusterfuzz/src/third_party/proto/message.py:932 ) at ._get_allocation_policy ( /mnt/scratch0/clusterfuzz/src/clusterfuzz/_internal/google_cloud_utils/batch.py:182 ) at ._create_job ( /mnt/scratch0/clusterfuzz/src/clusterfuzz/_internal/google_cloud_utils/batch.py:225 ) at .create_uworker_main_batch_jobs ( /mnt/scratch0/clusterfuzz/src/clusterfuzz/_internal/google_cloud_utils/batch.py:139 ) at .schedule_utask_mains ( /mnt/scratch0/clusterfuzz/src/python/bot/startup/run_bot.py:105 ) at .task_loop ( /mnt/scratch0/clusterfuzz/src/python/bot/startup/run_bot.py:136 ) ```

2024-06-26

Chore/py37 proto bump reattempt (#4062) Reattempt at #4029 This PR belongs to [this initiative](https://github.com/google/clusterfuzz/issues/4059)

2024-06-24

Revert changes to upgrade packages. (#4061) They are broken in prod causing this error: https://pantheon.corp.google.com/errors/detail/CPv0x-n-iqv-jQE?e=-13802955&mods=logs_tg_prod&project=google.com:clusterfuzz ``` Unable to update heartbeat. Traceback (most recent call last): File "/mnt/scratch0/clusterfuzz/src/clusterfuzz/_internal/datastore/data_handler.py", line 1130, in update_heartbeat heartbeat = ndb.Key(data_types.Heartbeat, bot_name).get() File "/mnt/scratch0/clusterfuzz/src/third_party/google/cloud/ndb/_options.py", line 102, in wrapper return wrapped(*pass_args, **kwargs) File "/mnt/scratch0/clusterfuzz/src/third_party/google/cloud/ndb/utils.py", line 150, in positional_wrapper return wrapped(*args, **kwds) File "/mnt/scratch0/clusterfuzz/src/third_party/google/cloud/ndb/key.py", line 859, in get return self.get_async(_options=_options).result() File "/mnt/scratch0/clusterfuzz/src/third_party/google/cloud/ndb/tasklets.py", line 210, in result self.check_success() File "/mnt/scratch0/clusterfuzz/src/third_party/google/cloud/ndb/tasklets.py", line 157, in check_success raise self._exception File "/mnt/scratch0/clusterfuzz/src/third_party/google/cloud/ndb/tasklets.py", line 319, in _advance_tasklet yielded = self.generator.throw(type(error), error, traceback) File "/mnt/scratch0/clusterfuzz/src/third_party/google/cloud/ndb/key.py", line 944, in get entity_pb = yield _datastore_api.lookup(self._key, _options) File "/mnt/scratch0/clusterfuzz/src/third_party/google/cloud/ndb/tasklets.py", line 323, in _advance_tasklet yielded = self.generator.send(send_value) File "/mnt/scratch0/clusterfuzz/src/third_party/google/cloud/ndb/_datastore_api.py", line 165, in lookup entity_pb = yield batch.add(key) File "/mnt/scratch0/clusterfuzz/src/third_party/google/cloud/ndb/_datastore_api.py", line 217, in add todo_key = key.to_protobuf()._pb.SerializeToString() File "/mnt/scratch0/clusterfuzz/src/third_party/google/cloud/datastore/key.py", line 324, in to_protobuf key.path.append(element) TypeError: Parameter to MergeFrom() must be instance of same class: expected google.datastore.v1.Key.PathElement got PathElement. ``` Vitor says this isn't a deterministic bug.

2024-06-17

Chore/bump py37 deps (#4029) *** Motivation This PR complements #4005 by bumping the gcp clients, grpcio and protobuf, and fixing the code to accomodate for breaking changes.

2024-06-10

Dropping entity proto headers for protobuf.Any, recompiling

2024-05-13

Make crash_revision an int64 (#3990) Fixes: https://pantheon.corp.google.com/errors/detail/CPquw-3h_ueTvQE;service=;version=?project=google.com:cluster-fuzz

2024-04-26

Store the backup corpus using a signed url (#3946)

2024-04-03

Migrate corpus pruning [3/N] (#3905) Move `process_corpus_crashes` to postprocess: 1- From `utask_main`: Pack the crash.unit_path into a zip file and upload using a signed url 2- From `utask_postprocess`: get the zip, unpack individual files and upload necessary ones to blob storage, store the blob key in the created Testcase's `fuzzed_keys` attribute.

2024-03-28

Fix

2024-03-19

Handle fuzz target metadata correctly and in an untrusted manner. (#3894) Previously analyze task was not retrieving the fuzz target metadata properly (this needs the build, analyze was doing it in postprocess) and progression was doing it in an unprivileged way (using the db in utask_main). Change this so we can retrieve the metadata in utask_main and save it in postprocess. Leave a TODO in corpus_pruning which isn't migrated yet.

Move privileged operation in minimize to preprocess. (#3896) Move argument retreival to preprocess. Related: #3008 Fixes: #3888

2024-03-15

Set data bundles properly in preprocess.

Reland 00c0b532ef5a2ccd4b390304db11cb62414b378d (#3870) Most of the failures were either unrelated or temporary due to change in proto and updated clients not using the new proto.

Revert "Make data bundles uworker safe (#3868) Causing exceptions.

Make data bundles uworker safe (#3862)

Make test_for_crash_with_retries uworker_safe (#3865)

Make test_for_reproducibility uworker safe. (#3864) All tasks that take a testcase as an argument use this function.

2024-03-13

[rearchitecture] Upload more stats in postprocess (#3829)

2024-03-12

Make LSAN blacklist uworker safe (#3844)

2024-03-11

fuzz_task: Migrate crashes to uworker. - Add proto definitions for crash groups. - Have uworkers return serialized CrashGroups, with the input testcases uploaded via signed URLs. - Handle checking if crash is new, testcase creation etc in postprocess. - Remove old uworker_msg.CrashInfo conversion and move the job_run crash group computation to postprocess, as it relies on querying the Datastore to know if the crash is new or not. - Also handle uploading file handles in storage.upload_signed_url - Add bot_name, platform_id to uworker output. TODO: tests.

2024-03-06

fix

2024-03-05

Dont make proto backwards incompatible. (#3806)

Migrate corpus pruning [2/N] (#3763)

2024-02-26

Revert "Use signed URLs for corpus (#3651)" This reverts commit d65125cd0e15c9786e0c2421eff8880a8a777395.

2024-02-23

Improve logging so we can see tasks postprocess and execute (#3774)

2024-02-22

Do fuzzer selection in preprocess (#3760) Handle fuzzer selection in a way that is compatible with the fact that corpus access doesn't happen during uworker_main. This means we will not be able to record fuzz targets during main or access the corpus then, so fuzzer selection must happen first.

2024-02-21

[Minimize] Use signed urls within utask_main and cleanup unused blobs in (#3766) postprocess. --------- Co-authored-by: jonathanmetzman <31354670+jonathanmetzman@users.noreply.github.com>

2024-02-12

Use signed URLs for corpus (#3651)

[Progression] call filter_stacktrace with a signed_url within utask_main (#3730)

2024-01-31

[Minimize] upload testcase using signed url (#3701)

2024-01-30

Migrate corpus pruning 2 (#3675) Co-authored-by: jonathanmetzman <31354670+jonathanmetzman@users.noreply.github.com>

merge

[Progression] Store testcase for regression testing using a signed url. (#3695) The testcase is now stored under a randomly generated filename instead of the testcase file hash of contents.

2024-01-26

Migrate corpus pruning 2

[rearchitecture] Record how long utasks take (#3679) Record how long utasks take in two ways: 1. From start to finish 2. How long each part takes. So we can figure out why batch seems to fall behind the queue-based system.

2024-01-24

Migrate corpus pruning task [1/N] (#3670)

2024-01-23

Migrate regression task. (#3652)

Update to python3.11

Handle trusted parts of store_fuzzer_run_results in pre/postprocess (#3406)

Migrate Minimize task (#3600)

2024-01-03

[Progression] Move update_build_metadata to postprocess (#3595)

2023-12-19

Revert "[Progression] Move update_build_metadata to postprocess (#3463)" (#3593) This reverts commit d6742919725dd09c6f11e2ae9b2d874412418de8.

[Progression] Move update_build_metadata to postprocess (#3463)

2023-12-12

[Symbolize] Move get_stacktrace to preprocess (#3556)

2023-12-11

Use protos directly in utasks (#3536) This change removes the uworker_io wrappers and uses protos directly within the utasks. This change also removes the Json user defined proto.