Tony Bhimani’s Blog

The goal of this article is to help you create multi-volume archives and generate checksums to validate integrity. Why? You have data larger than any single CD-R disc or DVD and you need to split it into pieces, or you have to transfer gigabytes of data over the net and would rather send smaller segments instead of a giant glob. As an example, I’ll create a multi-volume gzip archive of /home with a MD5 checksum using tar, gzip, split, and md5sum.

Creating Volumes from your Data

1. Create a single TAR archive of all your data using tar to preserve permissions, directory structures, etc.

[root@linux archive]# tar -cf home.tar /home
tar: Removing leading `/' from member names
[root@linux archive]# ls -la home.tar
-rw-r--r-- 1 root root 304220160 Apr 30 13:34 home.tar

2. Compress your TAR archive using gzip (or any other compressing program of your choice).

[root@linux archive]# gzip home.tar
[root@linux archive]# ls -la home.tar.gz
-rw-r--r-- 1 root root 284859091 Apr 30 13:34 home.tar.gz

3. Use the split command to chop the compressed archive into smaller segments (I’ll be using 100MB pieces).

[root@linux archive]# split -d -b100m home.tar.gz home.tar.gz.
[root@linux archive]# ls -la
total 556940
drwxr-xr-x 2 root root      4096 Apr 30 13:56 .
drwxr-x--- 6 root root      4096 Apr 30 13:31 ..
-rw-r--r-- 1 root root 284859091 Apr 30 13:34 home.tar.gz
-rw-r--r-- 1 root root 104857600 Apr 30 13:56 home.tar.gz.00
-rw-r--r-- 1 root root 104857600 Apr 30 13:56 home.tar.gz.01
-rw-r--r-- 1 root root  75143891 Apr 30 13:57 home.tar.gz.02

4. Create a MD5 checksum (or a SHA1 checksum).

[root@linux archive]# md5sum home.tar.gz* > MD5SUM
[root@linux archive]# cat MD5SUM
cb16175f4acad02f977f74d5c142879b  home.tar.gz
33c745ca49ab6e63b727658ec148cf67  home.tar.gz.00
14e6952b632fbb7f4c0731067afdb46c  home.tar.gz.01
386655357f8553c7730fd792c22fde2a  home.tar.gz.02

Same thing but creating a SHA1 checksum instead (you don’t need two checksums, I just illustrate to use both types — pick one).

[root@linux archive]# sha1sum home.tar.gz* > SHA1SUM
[root@linux archive]# cat SHA1SUM
3858b51622dc9135c192a7c98dec24ccd35c63d6  home.tar.gz
6bc12b26dc1388d70d1a7cc0290dc6c9e8e0f97e  home.tar.gz.00
0683a44538ac65330fe103440e4f2a4a3a652be5  home.tar.gz.01
eb0f65fd0f4b3d98221e3ae8600f1691b536ad1d  home.tar.gz.02

Restoring your Data from the Volumes

You’ve burned or transferred your volumes and now want to restore them to the original. Here are the steps.

1. Verify the checksum against the volumes (ignore the error on the original file).

[root@linux resurrection]# md5sum --check MD5SUM
md5sum: home.tar.gz: No such file or directory
home.tar.gz: FAILED open or read
home.tar.gz.00: OK
home.tar.gz.01: OK
home.tar.gz.02: OK
md5sum: WARNING: 1 of 4 listed files could not be read

Once again, same deal but with the SHA1SUM file.

[root@linux resurrection]# sha1sum --check SHA1SUM
sha1sum: home.tar.gz: No such file or directory
home.tar.gz: FAILED open or read
home.tar.gz.00: OK
home.tar.gz.01: OK
home.tar.gz.02: OK
sha1sum: WARNING: 1 of 4 listed files could not be read

2. Join the volume pieces together using cat (after you finish you can validate the checksum *again* to see if the original file passes an integrity check).

[root@node2 resurrection]# cat home.tar.gz.* > home.tar.gz
[root@node2 resurrection]# ls -la home.tar.gz
-rw-r--r-- 1 root root 284859091 Apr 30 15:23 home.tar.gz

3. Decompress and extract the tar.gz file contents and you’re done.

[root@linux resurrection]# tar zxvf home.tar.gz
... verbose file list ...
[root@linux resurrection]# ls -la
total 556952
drwxr-xr-x 3 root root      4096 Apr 30 15:33 .
drwxr-x--- 7 root root      4096 Apr 30 15:02 ..
drwxr-xr-x 4 root root      4096 Apr 30 13:06 home
-rw-r--r-- 1 root root 284859091 Apr 30 15:23 home.tar.gz
-rw-r--r-- 1 root root 104857600 Apr 30 15:04 home.tar.gz.00
-rw-r--r-- 1 root root 104857600 Apr 30 15:04 home.tar.gz.01
-rw-r--r-- 1 root root  75143891 Apr 30 15:04 home.tar.gz.02
-rw-r--r-- 1 root root       193 Apr 30 15:05 MD5SUM
-rw-r--r-- 1 root root       225 Apr 30 15:05 SHA1SUM

Contents extracted and there is the home directory. The End.

I got the idea of using the split command from this post on the Ubuntu Forum (how to create multi zip files) because I couldn’t get tar or gzip to create multi-volume archives.