payload-dumper-go

An android OTA payload dumper written in Go.

Features

screenshot

See how fast payload-dumper-go is: https://imgur.com/a/X6HKJT4. (MacBook Pro 16-inch 2019 i9-9750H, 16G)

Incredibly fast decompression. All decompression progresses are executed in parallel.
Incremental OTA (delta) payload support. Applied on top of the base images (-old), bit-exact output.
Verifies everything: operation data, source images, and final images (sha256). Fails loudly with a non-zero exit code.
Support original zip package that contains payload.bin, read in place without a temp copy.
Usable as a Go library. (github.com/ssut/payload-dumper-go/payload)

Cautions

There's just one dependency you need to install on your system: xz. (The reason I didn't use the pure Go implementation is written in the Performance section below.)
Working on a SSD is highly recommended for performance reasons, a HDD could be a bottle-neck.

Limitations

PUFFDIFF, ZUCCHINI and LZ4DIFF_* delta operations are not supported yet. Affected partitions (usually only system/product/system_ext) fail with a clear error; extract the rest with -p.

Installation

Linux and macOS (From releases, recommended)

Download the latest binary for your platform from here and extract the contents of the downloaded file to a directory on your system.
Make sure the extracted binary file has executable permissions. You can use the following command to set the permissions if necessary:

chmod +x payload-dumper-go

Run the following command to add the directory path to your system's PATH environment variable:

export PATH=$PATH:/path/to/payload-dumper-go

Note: This command sets the PATH environment variable only for the current terminal session. To make it permanent, you need to add the command to your system's profile file (e.g. .bashrc or .zshrc for Linux/Unix systems).

macOS (Homebrew)

Just simply run:

$ brew install payload-dumper-go

Windows

Download the latest binary for your platform from here and extract the contents of the downloaded file to a directory on your system.
Open the Start menu and search for "Environment Variables".
Click on "Edit the system environment variables".
Click on the "Environment Variables" button at the bottom right corner of the "System Properties" window.
Under "System Variables", scroll down and click on the "Path" variable, then click on "Edit".
Click "New" and add the path to the directory where the extracted binary is located.
Click "OK" on all the windows to save the changes.

Usage

Run the following command in your terminal:

payload-dumper-go /path/to/payload.bin

The input can be a raw payload.bin or an OTA zip, detected by content.

Options:
  -c, -concurrency       Number of workers to extract concurrently (default: number of CPUs)
  -l, -list              Show list of partitions in payload.bin
  -o, -output            Set output directory
  -p, -partitions        Dump only selected partitions (comma-separated)
  -old                   Directory with base images, required for incremental OTA
  -q, -quiet             Quiet mode - suppress non-essential output
  -m, -machine-readable  Machine-readable output format
  -no-verify             Skip sha256 verification

Incremental (delta) OTA

Extract the base (previous) full OTA first, then pass it via -old:

payload-dumper-go -o base_images base_full_ota.zip
payload-dumper-go -old base_images -o new_images incremental_ota.zip

Library usage

import "github.com/ssut/payload-dumper-go/payload"

p, _ := payload.Open("ota.zip")
defer p.Close()

err := p.Extract(context.Background(), payload.ExtractOptions{
    OutputDir: "out",
    SourceDir: "base_images",
})

Performance

Machine: MacBook Pro 16-inch 2021 (Apple M1 Max, 64G), OS: macOS Sonoma 14.5, Go: 1.22.4.

Tested with a 2.31GB payload.bin file from https://developers.google.com/android/ota (akita).

payload.bin: payload.bin
Payload Version: 2
Payload Manifest Length: 154250
Payload Manifest Signature Length: 523
Found partitions:
abl (1.8 MB), bl1 (16 kB), bl2 (537 kB), bl31 (106 kB), boot (67 MB), dtbo (17 MB), gcf (8.2 kB), gsa (348 kB), gsa_bl1 (33 kB), init_boot (8.4 MB), ldfw (2.4 MB), modem (102 MB), pbl (49 kB), product (3.4 GB), pvmfw (1.0 MB), system (821 MB), system_dlkm (11 MB), system_ext (288 MB), tzsw (7.9 MB), vbmeta (12 kB), vbmeta_system (8.2 kB), vbmeta_vendor (4.1 kB), vendor (693 MB), vendor_boot (67 MB), vendor_dlkm (28 MB), vendor_kernel_boot (67 MB)
Number of workers: 4
abl (1.8 MB)                [===================================================================================================================] 100 %
bl2 (537 kB)                [===================================================================================================================] 100 %
bl1 (16 kB)                 [===================================================================================================================] 100 %
bl31 (106 kB)               [===================================================================================================================] 100 %
boot (67 MB)                [===================================================================================================================] 100 %
dtbo (17 MB)                [===================================================================================================================] 100 %
gcf (8.2 kB)                [===================================================================================================================] 100 %
gsa (348 kB)                [===================================================================================================================] 100 %
gsa_bl1 (33 kB)             [===================================================================================================================] 100 %
init_boot (8.4 MB)          [===================================================================================================================] 100 %
ldfw (2.4 MB)               [===================================================================================================================] 100 %
modem (102 MB)              [===================================================================================================================] 100 %
pbl (49 kB)                 [===================================================================================================================] 100 %
product (3.4 GB)            [===================================================================================================================] 100 %
pvmfw (1.0 MB)              [===================================================================================================================] 100 %
system (821 MB)             [===================================================================================================================] 100 %
system_dlkm (11 MB)         [===================================================================================================================] 100 %
system_ext (288 MB)         [===================================================================================================================] 100 %
tzsw (7.9 MB)               [===================================================================================================================] 100 %
vbmeta (12 kB)              [===================================================================================================================] 100 %
vbmeta_system (8.2 kB)      [===================================================================================================================] 100 %
vbmeta_vendor (4.1 kB)      [===================================================================================================================] 100 %
vendor (693 MB)             [===================================================================================================================] 100 %
vendor_boot (67 MB)         [===================================================================================================================] 100 %
vendor_dlkm (28 MB)         [===================================================================================================================] 100 %
vendor_kernel_boot (67 MB)  [===================================================================================================================] 100 %
go run *.go payload.bin  87.93s user 3.51s system 145% cpu 1:02.99 total

Why not use the pure Go implementation for xz decompression?

The pure Go implementation of xz is very slow compared to the C implementation used with CGO. Here's the result with the same payload.bin file on the same conditions:

payload.bin: payload.bin
Payload Version: 2
Payload Manifest Length: 154250
Payload Manifest Signature Length: 523
Found partitions:
abl (1.8 MB), bl1 (16 kB), bl2 (537 kB), bl31 (106 kB), boot (67 MB), dtbo (17 MB), gcf (8.2 kB), gsa (348 kB), gsa_bl1 (33 kB), init_boot (8.4 MB), ldfw (2.4 MB), modem (102 MB), pbl (49 kB), product (3.4 GB), pvmfw (1.0 MB), system (821 MB), system_dlkm (11 MB), system_ext (288 MB), tzsw (7.9 MB), vbmeta (12 kB), vbmeta_system (8.2 kB), vbmeta_vendor (4.1 kB), vendor (693 MB), vendor_boot (67 MB), vendor_dlkm (28 MB), vendor_kernel_boot (67 MB)
Number of workers: 4
abl (1.8 MB)                [===================================================================================================================] 100 %
bl1 (16 kB)                 [===================================================================================================================] 100 %
bl2 (537 kB)                [===================================================================================================================] 100 %
bl31 (106 kB)               [===================================================================================================================] 100 %
boot (67 MB)                [===================================================================================================================] 100 %
dtbo (17 MB)                [===================================================================================================================] 100 %
gcf (8.2 kB)                [===================================================================================================================] 100 %
gsa (348 kB)                [===================================================================================================================] 100 %
gsa_bl1 (33 kB)             [===================================================================================================================] 100 %
init_boot (8.4 MB)          [===================================================================================================================] 100 %
ldfw (2.4 MB)               [===================================================================================================================] 100 %
modem (102 MB)              [===================================================================================================================] 100 %
pbl (49 kB)                 [===================================================================================================================] 100 %
product (3.4 GB)            [===================================================================================================================] 100 %
pvmfw (1.0 MB)              [===================================================================================================================] 100 %
system (821 MB)             [===================================================================================================================] 100 %
system_dlkm (11 MB)         [===================================================================================================================] 100 %
system_ext (288 MB)         [===================================================================================================================] 100 %
tzsw (7.9 MB)               [===================================================================================================================] 100 %
vbmeta (12 kB)              [===================================================================================================================] 100 %
vbmeta_system (8.2 kB)      [===================================================================================================================] 100 %
vbmeta_vendor (4.1 kB)      [===================================================================================================================] 100 %
vendor (693 MB)             [===================================================================================================================] 100 %
vendor_boot (67 MB)         [===================================================================================================================] 100 %
vendor_dlkm (28 MB)         [===================================================================================================================] 100 %
vendor_kernel_boot (67 MB)  [===================================================================================================================] 100 %
go run *.go payload.bin  587.89s user 2428.69s system 248% cpu 20:12.19 total

As you can see, the pure Go implementation is about 6~ times slower than the C implementation.

Sources

https://android.googlesource.com/platform/system/update_engine/+/master/update_metadata.proto

License

This source code is licensed under the Apache License 2.0 as described in the LICENSE file.