EZIO Developer and User Guide

build test

Introduction

EZIO is a tool for rapid server disk image cloning/deployment within local area network. We utilize BitTorrent protocol to speed up the data distribution. Also, we use partclone to dump used filesystem blocks, and EZIO receiver can directly write received blocks to raw disk, which greatly improves performance.

Motivation

EZIO is inspired by Clonezilla and BTsync (Resilio) for its idea to transfer data and massive deployment. The issue of Clonezilla is that, it is too slow in real world due to multicast feature. In real world, all clients must register to Clonezilla server before starting deployment, which cost too much time. In addition, whenever there is a client that doesn't get data or gets incorrect one and need to be re-transferred, it causes server a lot of effort. Most importantly, in most case, the clients which cannot get data correctly may be broken, it will make server to re-transfer data again and again until it reaches its re-transfer limit and quit. Due to above issues of Clonezilla, EZIO make a difference by changing transfer mechanism. EZIO implement transfer function on top of BitTorrent, and make a lot of progress on deployment speed.

Feature

Installation

Minimum System Requirements

Dependencies

sudo apt install build-essential cmake libboost-all-dev libtorrent-rasterbar-dev libgrpc-dev libgrpc++-dev libprotobuf-dev protobuf-compiler-grpc libspdlog-dev

Build and Install

mkdir build
cd build
cmake ../
make
sudo make install

We also provide a Dockerfile for the ease of installation and CI testing. To build the image type this:

docker build . -t ezio-latest-img

Usage

Partclone

Partclone provides utilities to save and restore used filesystem blocks (and skips the unused blocks) from/to a partition.

The newest partclone will support dump your disk to EZIO image, and generate torrent.info simultaneously.

sudo partclone.extfs -c -T -s /dev/sda1 -O target/ --buffer_size 16777216

or you want generate torrent, but don't want BT image.

sudo partclone.extfs -c -t -s /dev/sda1 -O target/ --buffer_size 16777216

When finishing to dump disk, you will see the file like the picture. And using utils/partclone_create_torrent.py to generate torrent for deploy. 

utils/partclone_create_torrent.py -c CloneZilla -p sda1 -i <some_path>/torrent.info -o sda1.torrent -t 'http://<some tracker>:6969/announce'

EZIO

When you have a sda1.torrent you can deploy or clone your disk via Network.

Help

Allowed Options:
  -h [ --help ]          some help
  -F [ --file ]          read data from file rather than raw disk
  --listen arg           gRPC service listen address and port, default is

Seeding

./ezio -F
./utils/create_proto_py.sh
./utils/add_torrent_seed.py sda1.torrent /some/path/to/sda1

./ezio
./utils/create_proto_py.sh
./utils/add_torrent_seed.py sda1.torrent /dev/sda1

Downloading

./ezio
./utils/create_proto_py.sh
./utils/add_torrent.py sda1.torrent /dev/sda1

./ezio -F
./utils/create_proto_py.sh
./utils/add_torrent.py sda1.torrent /some/path/to/save/sda1

Proxy

If you want to deploy over Internet or some bottleneck, you can proxy the torrent via regular BT software like qBittorrent. And don't let internal peer connect outside directly.

Easy Usage to Deploy Disk or OS via EZIO

Using CloneZilla Live (version>=testing-2.6.0-31). CloneZilla contains EZIO in its Lite Server Mode. It will be most easy way to deploy your disk or OS via BT.

Design

In raw_storage.cpp implements a libtorrent custom storage, to allow the receiver to write received blocks directly to raw disk.

We store the "offset" in hex into torrent, the "length" into file attribute. so BT will know where the block is, and it can use the offset to seek in the disk

{
    'announce': 'http://tracker.site1.com/announce',
    'info':
    {
        'name': 'root',
        'piece length': 262144,
        'files':
        [
            {'path': ['0000000000000000'], 'length': 4096}, // store offset and length of blocks
            {'path': ['0000000000020000'], 'length': 8192},
            ...
        ],
        'pieces': 'some piece hash here'
    }
}

Benchmark

Compare with CloneZilla Multicast Mode with EZIO Mode.

Experimental environment

Result

Time in second

Number of client Time (Unicast) Time (EZIO) Time (Multicast) Ratio (BT/Multicast)
1 474 675 390 1.731
2 948 1273 474 2.686
4 1896 1331 638 2.086
8 3792 1412 980 1.441
16 7584 1005 1454 0.691
24 11376 1048 1992 0.526
32 15168 1143 2203 0.519

Open Access Journal

More details about EZIO design and benchmark are in A Novel Massive Deployment Solution Based on the Peer-to-Peer Protocol.

Limitation

Future

Contribute

Support

If you are having issues, please let us know. EZIO main developer email is located at: tjjh89017@hotmail.com

Special Thanks

License

The project is licensed under the GNU General Public License v2.0 license.