[thelist] ftp synchronization tool

[Peter Smulders]

Hi all, Sarah,

Sarah wrote:
> Hello, all!
> 
> I am looking for recommendations for an FTP synchronization tool. I am 
> trying to make sure that a site I am working on is synched up between a 
> local server here at work and a client's remote server (where the site 
> will be hosted). Here's what I'm looking for:
> (1) works on Win2k
> (2) will preserve the modified time of files when they are transferred
> (3) will allow you to specify what to do with files based on if they:
>     - exist only on my machine
>     - are newer on my machine
>     - are the same on both machines
>     - exist only on the remote server
>     - are newer on the remote server
> I've tried WS_FTP Pro, which doesn't do (2) (at least not when using the 
> synchronization tool), and FTP Voyager, which doesn't do (3).

Depending on:
- Skills available to you
- Networking setup between the two sites

It might be a very good idea to use 'rsync'[1]. This is a UNIX-flavoured 
tool that is quite adept at figuring out the differences between two 
sets of files and pumping over those bits that are needed to make one 
site match another one.

The array of options is quite staggering; from (my) memory, the demands 
you have are all covered.

Rsync has the advantage of working incrementally: if you would, say, add 
one line to a 150MB file, it would essentially send over that one line, 
which theoretically is as efficient as you can get.

To cover the platforms you need (Win2K to Win2K, if I understand 
correctly), you will need an extra bit of tooling. I can heartily 
reccommend Cygwin[2], which includes a version of rsync.

Cygwin allows you to run UNIX-style tools on the Win32 family of 
operating systems.

The upside of doing this is that it works very, very well, especially on 
large amounts of files.

War story: I have used rsync to 'precopy' file sets as big as 200GB, 
which were still being worked on at the time (a CVS repository, if you 
really want to know). This took upwards of 8 hours over a slowish line. 
When the work was finished, some days later, I did a final rsync run, 
which picked up all the latest changes and thus did a data transfer that 
was scheduled to take 'one day after the work is finished' in about ten 
minutes after that time.


Cygwin is a world apart: I have seen people run Apache (the UNIX 
version, not the native Win32 compile), sshd, inetd, telnetd, XFree86 
(with rootless X-Windows, if that means anything to you), (La)TeX, 
rsyncd and a host of other programs, some even as Windows services on 
standard issue Windows machines. Suffice to say that is pretty powerful 
if you know how to take advantage of it. Lots of reading manuals has 
done me good, in any case.

Hope this helps,

Schmolle

[1] http://samba.anu.edu.au/rsync/
[2] http://www.cygwin.com/


PS: if you decide to go this route, feel free to ask for help here or in 
private; your problem is not difficult to solve, but requires lots of 
knowledge about the exact details of your setup.