The Concept
Sometimes you want to start file transfers when you’re ready. You’re generating some files and you want the transfer to start immediately after the generated files are done.
The Server
The script to sit on the server is simple. Run the command then touch the trigger.
user@Server:~$ generate_files.sh; touch ./trigger-file
This will run and then touch the trigger file. The trigger file starts the next half of the process.
The Workstation (or second server)
Now all we need to do is wait for that trigger to be created and then start to run the transfer.
#!/bin/bash
while [ 1 ]
do
rsync --remove-source-files --times --timeout 180 --partial --progress -e "ssh -p 22" user@server.dom:/home/user/trigger-file /home/user/
if [ "$?" = "0" ] ; then
echo "Starting rsync based on trigger file."
break
else
echo "Backing off, waiting for trigger file."
sleep 5m
fi
done
while [ 1 ]
do
rsync --recursive --times --timeout 180 --partial --progress -e "ssh -p 22" user@server.dom:/home/user/source-files/ /home/user/destination-files/
if [ "$?" = "0" ] ; then
echo "Done, rsync completed normally"
break
else
echo "Rsync failed. Backing off and retrying..."
sleep 5m
fi
done
Basically this loops through until the file exist then continues on to finish the transfer. The whole process uses rsync over ssh which means that you get the wonderful resuming capabilities of rsync along with the confidentiality, integrity and authentication of ssh.
This is the basis for many scripts which I use on a daily basis to automate file transfers and it’s proven most reliable.
Regards,
Robert.
Robert Small, The blog thereof. 