Raspberry Pi: Using rdiff-backup with my automated Python script

So you’ve followed my Raspberry Pi: Publicly accessible file server and you want to be a bit smarter on how you back up your data. People tell you to back up your data all the time, and you’re probably like me and grunt at the thought of the manual burden of physically copying and pasting your data.

Do you want your backup to be automated?
Do you want to be able to retrieve older revisions of your files?
Do you want to choose the files and folders that you want to back up with no limitation?
Are you a bit of a security nut and don’t really trust leaving all (or some) of your data in the cloud?
Do you want to keep costs down to an absolute minimum?

I have put together a Python script which you can use that helps you with all of the above.

I’m getting all salesy because this is a bit of a long process but it’s definitely worth it. I’ve simplified the script so you only have to edit 2 lines of code to make it work with your set up.

Prerequisites

  • Permissions with rdiff-backup seem to work differently compared to rsync, so to ensure rdiff-backup can write to the external drive on your RPi properly, make sure you have gid and uid defined in your /etc/fstab. See a working example in my Raspberry Pi: Network file sharing with Samba post.
  • You will need the Microsoft Visual C++ 2008 redistributables. If these are not installed on your system, rdiff-backup will be unable to run and Windows will display a message such as “The system cannot execute the specified program”. To install the redistributables for all users, install the package available from Microsoft.com (search for “visual c 2008 redistributable”).

1. Install Python

  1. Go to Python’s website and download either Windows x86 MSI Installer or Windows x86-64 MSI Installer for Python 2.7.x, depending if you’re running a 64bit system or not.
  2. Run through the installation, but at the “Customize Python 2.7.x” screen ensure you select to install “Add python.exe to Path“.
  3. Finish off the installation and you’re done. You will now notice if you open a command prompt Window and type “python“, you will be within a Python shell.

2. Install Cygwin, SSH and rdiff-backup and set Path variable

  1. Go to Cygwin’s website and download either setup-x86.exe or setup-x86_64.exe, depending if you’re running a 64bit system or not.
  2. Run through the installation, Install from Internet, keep directory locations as is, enter proxy settings if you’re using one, and at “Choose A Download Site” enter the following URL in the bottom text bod and press Add. Make sure you then have this URL selected and highlighted from the list once added. You will then be asked if you’re sure you want to use an unofficial mirror; this is an official mirror, it’s the same origin as http://mirrors.kernel.org/ but the location of the cygwin files changed to the below, the default list just hasn’t been updated yet as of July 2015. Click Yes.
http://mirrors.kernel.org/sourceware/cygwin/
  1. In the search bar, search for “SSH” and collapse the Net list and select to install the openssh: The OpenSSH server and client programs package from only the Bin? and not Src?.
  2. Click Next and Cygwin will begin to download the files for those packages.
  3. Click Finish and installation is complete.
  4. Go to rdiff-backup’s website and download the latest stable version with “-win32.zip” extension – this contains a 64bit compatible Windows executable of rdiff-backup.
  5. Extract only the .exe file within the archive and play it within Cygwin’s bin directory, default it is “C:\cygwin64\bin“.
  6. Add Cygwin’s directory to your system’s PATH variable:
    1. Windows 7: Select Computer from the Start menu, choose System Properties beneath the address bar, click Advanced system settings and go to the Advanced tab, click Environment Variables… and find the Path variable under System VariablesEdit and append “;C:\cygwin64\bin” to the end of the variable value. Click OK twice to apply changes, and you’ll now able to use various basic linux commands within CMD but most importantly ssh and rdiff-backup.
    2. Windows 8/8.1: Drag the mouse pointer to the right bottom corner of your screen, click Settings and then Control Panel. Click System and Security then System, and Advanced system settings from the left panel. Go to the Advanced tab, click Environment Variables… and find the Path variable under System VariablesEdit and append “;C:\cygwin64\bin” to the end of the variable value. Click OK twice to apply changes, and you’ll now able to use various basic linux commands within CMD but most importantly ssh and rdiff-backup.

3. Convert Putty private key to OpenSSH format

  1. Make sure you have downloaded PuTTYgen
  2. Run the executable and Load the Putty private key (enter passphrase if required)
  3. Go to Conversions from the toolbar and select Export OpenSSH key
  4. Name and save accordingly

4. Configure the Python script

  1. Click here to download my Python backup script; right click in whitespace and click “Save As“. Save anywhere you like, though C:\ would be easiest.
  2. Open it up in your favourite code editor, otherwise Notepad will do.
  3. Make the following changes on line #35 and # 40 and save the file:
    1. The IP address of the target machine, the Raspberry Pi, i.e. 192.168.0.125
    2. The port for SSH, i.e. 1021
    3. The full path location of the SSH private key (must be in OpenSSH format, use PuTTYgen to convert), i.e. C:\Users\JoeBloggs\.ssh\privatekey
    4. The source directory of which you want to backup, i.e. C:\Users\JoeBloggs\Documents
    5. The IP address of the target machine, the Raspberry Pi, i.e. 192.168.0.125
    6. The target directory on the remote machine of which you wish to store the backup, i.e. ::/media/exthdd1/backup – notice use of the double colon between the target IP and target directory

5. Change permissions of the SSH keys

SSH will only work if your keys have the appropriate permission settings. Public keys should be 644 (-rw-r–r–) and private keys should be 600 (-rw——).

  1. Open command prompt and change directory to where your private key resides
  2. Execute the following command to change its permissions:

chmod 600 <name of private key here>

6. Test the Python script

  1. Locate the Python script in Windows File Explorer and double click, if prompted to choose an application then select CMD.
  2. Run through the script. You might be prompted to confirm the fingerprint of the machine you’re connecting to if this is the first time you’re connecting to it. Type yes only if you’re sure that fingerprint is correct, read more here.
  3. The file transfer should begin giving you all sorts of feedback and statistics throughout the transfer, read my Comparison of rsync vs. rdiff-backup or the rdiff-backup manpage which explains all the details.

7. Schedule the Python script

If the test was successful, you can configure it to run at a particular or multiple times of the day for as long as your computer is on.

  1. Go to Control Panel
  2. Search for and select “Schedule tasks
  3. Click Create Task… name it whatever you like, mine is simply “Python rdiff-backup” and ensure to select the appropriate OS family in the “Configure for” dropdown at the bottom, otherwise everything else is fine as is
  4. Move over to the Triggers tab and select New
  5. From the left hand radio buttons select whichever frequency you like, and start date & time.
  6. Move over to the Actions tab and select New
  7. Click Browse and select the python.exe which is located at C:\Python27
  8. In the Add arguments field, give the full path location of the Python script. If you placed the script at the root of the C:\ drive, then it would simply be C:\desktopScript-rdiff-backup.py
  9. Press OK and all is now scheduled

Scripts for systems that go outside of your LAN (i.e. laptops) will be coming soon. There currently is a “laptop” script in my Git repo but I’m yet to tinker and retest.

How to restore an old archive using rdiff-backup

So how do you restore a file to a given date or interval period? Open up command prompt…

rdiff-backup -r 3D --remote-schema "ssh %s -p1019 -i C:/Users/Adam/.ssh/private-passphrase rdiff-backup --server" 192.168.0.125::/media/backup/exthdd1/Documents/file.txt C:/Users/JoeBloggs/Documents/temp/file3days.txt

The above restores file.txt from the remote server, from 3 days ago, and stores it in a local folder. The usual connection via SSH with public keys is also here.

rdiff-backup -r 2015/07/01 --remote-schema "ssh %s -p1019 -i C:/Users/Adam/.ssh/private-passphrase rdiff-backup --server"  192.168.0.125::/media/backup/exthdd1/Documents/file.txt C:/Users/JoeBloggs/Documents/temp/file-01072015.txt

The above restores file.txt from the remote server, from the 1st July 2015, and stores in a local folder named “file-01072015.txt”. The usual connection via SSH with public keys is also here.

More examples on restoring can be found here.

As mentioned in the last section, intervals are typically expressed as: s, m, h, D, W, M, and Y – seconds, months, hours, days, weeks, months and years respectively. They can even be used in a series of pairs, e.g. 4W2D1h30m5s equates to 4 weeks 2 days 1 hour 30 minutes and 5 seconds.

Not only can intervals be used, but date formats can also be expressed too. So if you wanted to delete up to a particular date, this can be expressed in any of the date formats: YYYY/MM/DD, YYYY-MM-DD, MM/DD/YYYY, or MM-DD-YYYY. The times of these dates will default to midnight in the machine’s relative timezone. Strings can be used to explicitly define date and time: “2015-07-01T06:00:00+01:00” 1st July 2015 at 6am BST (because my relative timezone is GMT).

More information can be found under the Time Formats section in rdiff-backup’s manual page.

Delete old archives that are x seconds/minutes/hours/days/weeks/months/years old

As rdiff-backup enables you to keep old revisions of your data in compressed archives, you can imagine that the space required to keep this going after a while can get quite large. Using the –remove-older-than option  you can ensure you keep archives no older than your desired age.

rdiff-backup --remove-older-than 2W /media/exthdd1/backup

The command is pretty self explanatory. 2W tells rdiff-backup to delete archives that are older than 2 weeks that are located in /media/exthdd1/backup. Other intervals like this are expressed as follows: s, m, h, D, W, M, and Y – seconds, months, hours, days, weeks, months and years respectively.

This can be utilised most effectively as a cron job where the Raspberry Pi will periodically (as frequently as you like) flush out old archives. Click here to read more about setting up cron jobs.

Note: rdiff-backup cannot remove-older-than and back up or restore in a single session. In order to both backup a directory and remove old files in it, you must run rdiff-backup twice.

This post is part of the Public Accessible Raspberry Pi File Server tutorial post.

Leave a Reply

Your email address will not be published. Required fields are marked *