Sync servers using multiple streams with LFTP
If you have a remote server that you want to sync a directory you can use for example rsync, however rsync doesn’t offer the multi stream syncing and can be slower if you have many new files. The below solution uses lftp with 5 parallel streams to download and each stream is split into 10 separate segments. This is very helpful if you cannot max your download connection with regular methods and of course if the uploading server has enough bandwidth.
#!/bin/bash login=your_username pass=your_password host=sftp://download.redoem.com remote_dir=/media/files/redoem/dir local_dir=/home/redoer/downloads trap "rm -f /tmp/syncserver.lock" SIGINT SIGTERM if [ -e /tmp/syncserver.lock ] then echo "Syncserver is running already." exit 1 else touch /tmp/syncserver.lock lftp -u $login,$pass $host << EOF set ftp:ssl-allow yes set mirror:use-pget-n 10 mirror -c -P5 --newer-than=now-15min --log=syncserver.log $remote_dir $local_dir quit EOF rm -f /tmp/syncserver.lock trap - SIGINT SIGTERM exit 0 fi
The above sample should work for most cases – you just need to change the values of the variables (highlighted in red).
This part (–newer-than=now-15min) can be changed based on how old files you want to sync, for example if you synced the files 4 hours and you have new files in the last hour, you want to change this part to ONLY download the new files then change this part to be now-60min or more (but less than 4 hours in this example).
If you want to share from your local machine to the remote server, replace this line as (changed parts highlighted in red):
mirror -R -c -P5 --newer-than=now-15min --log=syncserver.log $local_dir $remote_dir
Now save the above as a .sh file, example syncserver.sh and you can add it as a cronjob to run every 15 minutes or run it when you have new files.
*/15 * * * * /home/redoer/syncserver.sh >> /home/redoer/sync_cron.log 2>&1
Comments are closed.