Rsync is an awesome tool that I have used extensively. However, I think it is time to explore other tools that can replace some of what rsync has been doing for so many years. I’m not saying that you should never use rsysnc. What I am saying is there are other options and in some circumstances rsync isn’t the default choice any more(at least for me).
I was on the hunt for an rsync replacement because I was seeing extremely slow transfer times when moving large data files. Rysnc didn’t seem to take full advantage of all the bandwidth on a 1GB link. Transfers were taking much longer then what I thought they should. Rsysnc also does not run parallel syncs across the wire. In the event there are multiple files or one huge file, you have to figure out a work around, which makes writing scripts rather cumbersome. Here is an example http://stackoverflow.com/questions/24058544/speed-up-rsync-with-simultaneous-concurrent-file-transfers.
While on the hunt, lftp came into view so I took a look at it and eventually it is what I settled on. Another added benefit; lftp can be used as a standard ftp client as well. If you have legacy apps that expect /usr/bin/ftp to exist, you can create a symlink from /usr/bin/ftp to /usr/bin/lftp and your done. While I was researching the options for lftp I found there are about 6 dozen different ways to do something so make sure to read the man page carefully. There are so many options they can become confusing.
lftp has shell-like command syntax allowing you to launch several commands in parallel in background (&). It is also possible to group commands within () and execute them in back‐ ground. All background jobs are executed in the same single process. You can bring a fore‐ ground job to background with ^Z (c-z) and back with command `wait' (or `fg' which is alias to `wait'). To list running jobs, use command `jobs'. Some commands allow redirecting their output (cat, ls, ...) to file or via pipe to external command. Commands can be executed con‐ ditionally based on termination status of previous command (&&, ||).
An example lftp command that will use sftp to mirror a remote directory using shared keys
lftp -c "open sftp://email@example.com:/path/to/data/ && mirror --parallel=5 --verbose --delete && exit"
There are enough examples on the web so I won’t go into a great deal of detail on the specific command line usages. If you have never heard of lftp before go ahead and check it out.