Raspberry Pi: Using rsync with my automated Python script

So you’ve followed my Raspberry Pi: Publicly accessible file server and you want to be a bit smarter on how you back up your data. People tell you to back up your data all the time, and you’re probably like me and grunt at the thought of the manual burden of physically copying and pasting your data.

Do you want your backup to be automated?
Do you want to choose the files and folders that you want to back up with no limitation?
Are you a bit of a security nut and don’t really trust leaving all (or some) of your data in the cloud?
Do you want to keep costs down to an absolute minimum?

I have put together a Python script which you can use that helps you with all of the above.

I’m getting all salesy because this is a bit of a long process but it’s definitely worth it. I’ve simplified the script so you only have to edit 2 lines of code to make it work with your set up.

1. Install Python

  1. Go to Python’s website and download either Windows x86 MSI Installer or Windows x86-64 MSI Installer for Python 2.7.x, depending if you’re running a 64bit system or not.
  2. Run through the installation, but at the “Customize Python 2.7.x” screen ensure you select to install “Add python.exe to Path“.
  3. Finish off the installation and you’re done. You will now notice if you open a command prompt Window and type “python“, you will be within a Python shell.

2. Install Cygwin, SSH and rsync and set Path variable

  1. Go to Cygwin’s website and download either setup-x86.exe or setup-x86_64.exe, depending if you’re running a 64bit system or not.
  2. Run through the installation, Install from Internet, keep directory locations as is, enter proxy settings if you’re using one, and at “Choose A Download Site” enter the following URL in the bottom text bod and press Add. Make sure you then have this URL selected and highlighted from the list once added. You will then be asked if you’re sure you want to use an unofficial mirror; this is an official mirror, it’s the same origin as http://mirrors.kernel.org/ but the location of the cygwin files changed to the below, the default list just hasn’t been updated yet as of July 2015. Click Yes.
http://mirrors.kernel.org/sourceware/cygwin/
  1. In the search bar, search for “SSH” and collapse the Net list and select to install the openssh: The OpenSSH server and client programs package from only the Bin? and not Src?.
  2. In the search bar, search for “rsync” and collapse the Net list and select to install the rsync: Fast remote file transfer program (can use excisting data to minimize transfer) package from only the Bin? and not Src?.
  3. Click Next and Cygwin will begin to download the files for those packages.
  4. Click Finish and installation is complete.
  5. Add Cygwin’s directory to your system’s PATH variable:
    1. Windows 7: Select Computer from the Start menu, choose System Properties beneath the address bar, click Advanced system settings and go to the Advanced tab, click Environment Variables… and find the Path variable under System VariablesEdit and append “;C:\cygwin64\bin” to the end of the variable value. Click OK twice to apply changes, and you’ll now able to use various basic linux commands within CMD but most importantly ssh and rsync.
    2. Windows 8/8.1: Drag the mouse pointer to the right bottom corner of your screen, click Settings and then Control Panel. Click System and Security then System, and Advanced system settings from the left panel. Go to the Advanced tab, click Environment Variables… and find the Path variable under System VariablesEdit and append “;C:\cygwin64\bin” to the end of the variable value. Click OK twice to apply changes, and you’ll now able to use various basic linux commands within CMD but most importantly ssh and rsync.

3. Convert Putty private key to OpenSSH format

  1. Make sure you have downloaded PuTTYgen
  2. Run the executable and Load the Putty private key (enter passphrase if required)
  3. Go to Conversions from the toolbar and select Export OpenSSH key
  4. Name and save accordingly

4. Configure the Python script

  1. Click here to download my Python backup script; right click in whitespace and click “Save As“. Save anywhere you like, though C:\ would be easiest.
  2. Open it up in your favourite code editor, otherwise Notepad will do.
  3. Make the following changes on line #35 and # 40 and save the file:
    1. The IP address of the target machine, the Raspberry Pi, i.e. 192.168.0.125
    2. The full path of which you want to save the rsync.log file in, i.e. /media/exthdd1/rsync.log
    3. The port for SSH, i.e. 1021
    4. The full path location of the SSH private key (must be in OpenSSH format, use PuTTYgen to convert), i.e. /cygdrive/c/Users/JoeBloggs/.ssh/privatekey – notice use of /cygdrive/c/ instead of C:\
    5. The source directory of which you want to backup, i.e. /cygdrive/c/Users/JoeBloggs/Documents – notice use of /cygdrive/c/ instead of C:\
    6. The IP address of the target machine, the Raspberry Pi, i.e. 192.168.0.125
    7. The target directory on the remote machine of which you wish to store the backup, i.e. :/media/exthdd1/backup – notice use of single colon between the target IP and target directory

5. Change permissions of the SSH keys

SSH will only work if your keys have the appropriate permission settings. Public keys should be 644 (-rw-r–r–) and private keys should be 600 (-rw——).

  1. Open command prompt and change directory to where your private key resides
  2. Execute the following command to change its permissions:

chmod 600 <name of private key here>

6. Test the Python script

  1. Locate the Python script in Windows File Explorer and double click, if prompted to choose an application then select CMD.
  2. Run through the script. You might be prompted to confirm the fingerprint of the machine you’re connecting to if this is the first time you’re connecting to it. Type yes only if you’re sure that fingerprint is correct, read more here.
  3. The file transfer should begin giving you all sorts of feedback and statistics throughout the transfer, read my Comparison of rsync vs. rdiff-backup or the rsync manpage which explains all the details.

7. Schedule the Python script

If the test was successful, you can configure it to run at a particular or multiple times of the day for as long as your computer is on.

  1. Go to Control Panel
  2. Search for and select “Schedule tasks
  3. Click Create Task… name it whatever you like, mine is simply “Python rsync” and ensure to select the appropriate OS family in the “Configure for” dropdown at the bottom, otherwise everything else is fine as is
  4. Move over to the Triggers tab and select New
  5. From the left hand radio buttons select whichever frequency you like, and start date & time.
  6. Move over to the Actions tab and select New
  7. Click Browse and select the python.exe which is located at C:\Python27
  8. In the Add arguments field, give the full path location of the Python script. If you placed the script at the root of the C:\ drive, then it would simply be C:\desktopScript-rsync.py
  9. Press OK and all is now scheduled

Scripts for systems that go outside of your LAN (i.e. laptops) will be coming soon. There currently is a “laptop” script in my Git repo but I’m yet to tinker and retest.

This post is part of the Public Accessible Raspberry Pi File Server tutorial post.

Leave a Reply

Your email address will not be published. Required fields are marked *