Rsync is an awesome tool that I have used extensively. However, I think it is time to explore other tools that can replace some of what rsync has been doing for so many years. I’m not saying that you should never use rsysnc. What I am saying is there are other options and in some circumstances rsync isn’t the default choice any more(at least for me).

I was on the hunt for an rsync replacement because I was seeing extremely slow transfer times when moving large data files. Rysnc didn’t seem to take full advantage of all the bandwidth on a 1GB link. Transfers were taking much longer then what I thought they should. Rsysnc also does not run parallel syncs across the wire. In the event there are multiple files or one huge file, you have to figure out a work around, which makes writing scripts rather cumbersome. Here is an example http://stackoverflow.com/questions/24058544/speed-up-rsync-with-simultaneous-concurrent-file-transfers.

While on the hunt, lftp came into view so I took a look at it and eventually it is what I settled on. Another added benefit; lftp can be used as a standard ftp client as well. If you have legacy apps that expect /usr/bin/ftp to exist, you can create a symlink from /usr/bin/ftp to /usr/bin/lftp and your done. While I was researching the options for lftp I found there are about 6 dozen different ways to do something so make sure to read the man page carefully. There are so many options they can become confusing.

lftp  has  shell-like  command  syntax allowing you to launch several commands in parallel in
background (&). It is also possible to group commands within () and  execute  them  in  back‐
ground.  All  background  jobs are executed in the same single process. You can bring a fore‐
ground job to background with ^Z (c-z) and back with command `wait' (or `fg' which  is  alias
to  `wait').  To list running jobs, use command `jobs'. Some commands allow redirecting their
output (cat, ls, ...) to file or via pipe to external command. Commands can be executed  con‐
ditionally based on termination status of previous command (&&, ||).

An example lftp command that will use sftp to mirror a remote directory using shared keys

lftp -c "open sftp://username@server.domain.tld:/path/to/data/ && mirror --parallel=5 --verbose --delete && exit"

There are enough examples on the web so I won’t go into a great deal of detail on the specific command line usages. If you have never heard of lftp before go ahead and check it out.