web analytics

Blog

Duplicity – secure incremental backup

Tags: cPanel server backupDuplicityincremental backup

Published on: July 18, 2009 by George K.

Duplicity – secure incremental backup

Scenario:

Duplicity is a software suite, which helps in backup management services by providing encrypted, digitally signed, versioned, remote backup of files requiring little of the remote server.

Overview:

Duplicity is a tool to create GPG-encrypted (this way you can store your backups on remote servers without having to worry about who has access to your data) incremental backups to remote servers. Its a quite handy and secure method, which makes the backup management services much easier.

Installation

The steps to install duplicity is as follows

wget http://code.launchpad.net/duplicity/trunk/0.6.02/+download/duplicity-0.6.02.tar.gz

tar -xvf duplicity-0.6.02.tar.gz

cd duplicity-0.6.02.tar.gz

python setup.py install

If you come across any librsync.so errors . You can resolve it by using the following steps

wget http://sourceforge.net/projects/librsync/files/librsync/0.9.7/librsync-0.9.7.tar.gz/download

tar -xzvf librsync-0.9.7.tar.gz
cd librsync-0.9.7

./configure

make

make install

Now we got duplicity installed 🙂

Create a GPG key

In order to be able to encrypt your backups, you have to create a GPG key. Open a second shell and run the following command (this generates some “randomness” on your system, which will be useful to create a secure key). Kill the command with CTRL+C when you are done with key generation.

while /bin/true; do cat /var/log/messages > ~/temp.txt; sleep 1; done;

On your other shell, create your GPG key. Be sure to use a secure passphrase and to copy/write down the key ID which is displayed at the end of the generation process (we’ll need it for ftplicity). Also, make sure to backup the key to a secure location outside your server. As all your backups will be encrypted, they will be worthless if your server crashes and you lose the key.

gpg –gen-key

Default options should be fine. This will create your key in ~/.gnupg/. Once its done you can verify the existence of your key using the command

gpg –list-keys

The next step is to prepare an off-site location to receive the backup files.
The software supports different protocols like FTP,RSYNC,SCpP.
I am restricting myself with SCP here

Simple unEncrypted Backup over SCP

Setup ssh keys on the backup server allowing root to seamlessly login to the backup server.

duplicity /home/me scp://uname@other.host/usr/backup

  • If the above command is run repeatedly, the first session will be a full backup, and subsequent ones will be incremental.

    The full option can be used to force a full backup. The next command also excludes the /tmp directory.

    duplicity full –exclude /tmp /home/me scp://uname@other.host/usr/backup

  • Basic restore command—restore the /home/me directory backed up with scp above to directory restored_dir:
  • duplicity scp://uid@other.host//usr/backup restored_dir
  • To enable verbose mode use the option -v<level>
    Specify verbosity level (0 is total silent, 4 is the default, and 9 is noisiest)
    The command would look like

    duplicity -v5 /home/me scp://uid@other.host/some_dir

    Encrypted Backup over SCP

    Here we use the GPG key generated earlier
    The format would be look like this

    duplicity
        --encrypt-key=${GPG_KEY} \
        --sign-key=${GPG_KEY} \
        --include=/boot \
        --include=/etc \
        --include=/home \
        --include=/root \
        --include=/var/lib/mysql \
        --exclude=/** \
        ${SOURCE} ${DEST}

    Needless to say the include and exclude options are for specifying the backup criteria.

    duplicity –encrypt-key=”FFF7730B” –sign-key=”FFF7730B” -v5 /home/me scp://uid@other.host/some_dir

    you will be asked for a GnuPG passphrase. You can type in any password you like; this has to be done everytime you run duplicity. The backup will be encrypted with the help of GnuPG. Permissions and ownerships will be preserved in the backup.

    To avoid this issue , you can simply set the passphrase as environment variable using the command

    export PASSPHRASE=gpgpassphrase

    Backup Format & Explanation

    Once it is executed , you can see the backup in the server and it would look like the following way

    duplicity-full-signatures.2005-11-27T01:00:01-05:00.sigtar.gpg
    duplicity-full.2005-11-27T01:00:01-05:00.manifest.gpg
    duplicity-full.2005-11-27T01:00:01-05:00.vol1.difftar.gpg
    duplicity-full.2005-11-27T01:00:01-05:00.vol2.difftar.gpg

    The signatures file contains, signatures of each file that is backed up so that Duplicity can figure out which part of a file has changed. With that information it can upload only the missing part to complete a new backup set.

    The manifest file contains a listing of all the files in the backup set and a SHA1 hash of each file, probably so Duplicity can tell very quickly whether a file has been changed or not since the last backup.

    The volume files (vol1 and vol2) contain the actual file data. It appears that Duplicity volumes are at most 5MB. That’s helpful during restores so the entire backup set does’t not need to be downloaded to retrieve a single file. Duplicity will only download the volume containing that file.

    Common Options:

    Depending on the parameters and order of the parameters in the duplicity command, different functions can be performed. For example, an archive can be verified to see if a complete backup was made and what files, if any, have changed since the last backup.

    duplicity verify [options] source_url target_directory

    duplicity verify -v4 scp://user@bakuphost/etc /etc

    Listfiles

    It’s sometimes handy to check which files are in the latest backup set.

    duplicity list-current-files [options] target_url

    The command would look like

    duplicity list-current-files –archive-dir /root/test/ scp://user@backupserver/some_dir

    Restore

    The main purpose of backup is to restore data which has been lost. The following is the common format for restoring the data from the latest backup

    duplicity scp://uid@other.host/some_dir /home/me

    Duplicity enters restore mode because the URL comes before the local directory. If we wanted to restore just the file “Mail/article” in /home/me as it was three days ago into /home/me/restored_file:

    duplicity -t 3D –file-to-restore Mail/article scp://uid@other.host/some_dir /home/me/restored_file

    The following command compares the files we backed up, so see what has changed since then:

    duplicity verify scp://uid@other.host/some_dir /home/me

    The following command can be used to retrieve a single file from backup

    duplicity –encrypt-key “” –sign-key “” –file-to-restore home/sburke/file.txt scp://user@server.com/backup/ /var/tmp/file.txt

    1. The path to the file that is to be restored is relative to the directory on which the backup set is based. So in the command above, home/sburke/file.txt plus the directory on which we based our backup (/backup) equals /backup/home/sburke/file.txt/. It would not work to put /backup/home/sburke/file.txt as the source path because the backup will not recognize /backup as a valid path. The last portion in the above command is the location where the file will be restored.

    To delete old backups, we can use the following command

     duplicity --full --remove-older-than 1Y /media/data/backup scp://uid@server/personal

    To automate the tasks, you can write a shell script

    Category : Howtos, Linux

    George K.

    George K.

    George started his career in web hosting and Linux technical support in the year 2004 and is with SupportSages since 2009. He has keen interest in server optimizations, custom security solutions, hacked server recovery, cyber forensic and high availability fail over system design and implementation. George loves long drives and is passionate about art and literature.

    You may also read:

    Comments

    Add new commentSIGN IN

    Let's Connect

    Get new updates

    Categories

    $0.000 items