Posts tagged linux

If you have a remote server that you want to sync a directory you can use for example rsync, however rsync doesn’t offer the multi stream syncing and can be slower if you have many new files. The below solution uses lftp with 5 parallel streams to download and each stream is split into 10 separate segments. This is very helpful if you cannot max your download connection with regular methods and of course if the uploading server has enough bandwidth.

#!/bin/bash
login=your_username 
pass=your_password
host=sftp://download.redoem.com
remote_dir=/media/files/redoem/dir
local_dir=/home/redoer/downloads
 
trap "rm -f /tmp/syncserver.lock" SIGINT SIGTERM
if [ -e /tmp/syncserver.lock ]
then
 echo "Syncserver is running already."
 exit 1
else
 touch /tmp/syncserver.lock
 lftp -u $login,$pass $host << EOF
 set ftp:ssl-allow yes
 set mirror:use-pget-n 10 
 mirror -c -P5 --newer-than=now-15min --log=syncserver.log $remote_dir $local_dir
 quit
EOF
 rm -f /tmp/syncserver.lock
 trap - SIGINT SIGTERM
 exit 0
fi

The above sample should work for most cases – you just need to change the values of the variables (highlighted in red).

This part (–newer-than=now-15min) can be changed based on how old files you want to sync, for example if you synced the files 4 hours and you have new files in the last hour, you want to change this part to ONLY download the new files then change this part to be now-60min or more (but less than 4 hours in this example).

If you want to share from your local machine to the remote server, replace this line as (changed parts highlighted in red):

 mirror -R -c -P5 --newer-than=now-15min --log=syncserver.log $local_dir $remote_dir

Now save the above as a .sh file, example syncserver.sh and you can add it as a cronjob to run every 15 minutes or run it when you have new files.

*/15 * * * * /home/redoer/syncserver.sh >> /home/redoer/sync_cron.log 2>&1