Using remote trigger files to start a file transfer

The Concept

Sometimes you want to start file transfers when you’re ready. You’re generating some files and you want the transfer to start immediately after the generated files are done.

The Server

The script to sit on the server is simple. Run the command then touch the trigger.

user@Server:~$ generate_files.sh; touch ./trigger-file

This will run and then touch the trigger file. The trigger file starts the next half of the process.

The Workstation (or second server)

Now all we need to do is wait for that trigger to be created and then start to run the transfer.


#!/bin/bash

while [ 1 ]
do
    rsync --remove-source-files --times --timeout 180 --partial --progress -e "ssh -p 22" user@server.dom:/home/user/trigger-file /home/user/
    if [ "$?" = "0" ] ; then
        echo "Starting rsync based on trigger file."
        break
    else
        echo "Backing off, waiting for trigger file."
        sleep 5m
    fi
done

while [ 1 ]
do
    rsync --recursive --times --timeout 180 --partial --progress -e "ssh -p 22" user@server.dom:/home/user/source-files/ /home/user/destination-files/
    if [ "$?" = "0" ] ; then
        echo "Done, rsync completed normally"
        break
    else
        echo "Rsync failed. Backing off and retrying..."
        sleep 5m
    fi
done

Basically this loops through until the file exist then continues on to finish the transfer. The whole process uses rsync over ssh which means that you get the wonderful resuming capabilities of rsync along with the confidentiality, integrity and authentication of ssh.

This is the basis for many scripts which I use on a daily basis to automate file transfers and it’s proven most reliable.

Regards,
Robert.

Advertisements
This entry was posted in Uncategorized. Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s