Distributed bandwidth (was RE: [Theforum] Re: [Content] Mail Alert! - Sat Jun 22 31226)

[Beau Hartshorne]

> I would expect as many people as possible to use the most efficient
> mirroring tool available - that is rsync.  Whether you are mirroring 2
> megabytes, or 2 terabytes, rsync just rocks.  It can be secured over
ssh
> too.  I would much rather see people using rsync if they can, as
opposed to
> constantly wget'ing stuff, or - god forbid - manually downloading
files.
>
> Automating the mirroring process also helps us on the backend - we
only need
> to add a file once to beo, and the curator(s) will automatically get
that
> file they next time they connect.  If there are no changes, they don't
fetch
> anything.  They don't even do anything like HEAD checks to see if the
data
> has changed - rsync tells them so.
>
> I know i'm coming off as a bit arrogant sounding saying that, but
rsync
> really really is the way to go imho.  I maintain a few mirrors here at
home,
> and one of them is an FTP mirror, maintained over FTP.  To fetch the
changes
> on a 2.2gig dataset takes a minute or so to download.  When I rsync
the same
> dataset (albeit locally), I can see the amount of change data changing
hands
> is minimal compared to that transferred during the FTP session.  (did
that
> make sense? :)
>
> > [possible bandwidth misuse of mirrors by bad people snippage]
> >
> > The only solution I can think of is this: get volunteers to check
> > referrers, and only send files to anyone referred by evolt.org. Is
> > something like that possible?
>
> Yup, but it really does depend on the config of the curators kit -
once we
> start getting loads of people mirroring bits or all of beo, we're
moving
> away from the controlled environment that beo represents, and letting
loose
> a large amount of heterogenous services.  Some may be on Linux, some
NT,
> win2K, Tru64, Solaris, etc etc, some may be Apache, others IIS, Roxen,
etc
> etc.
>
> So YMMV.

Yeah, so how many of these platforms can run rsync?

At: ftp://rsync.samba.org/pub/rsync/binaries/

I see: AIX, BSDi, debian-i386, HPUX, IRIX, linuxppc, redhad, SCO,
Solaris, Sunos, SVR4.

You've got me sold on rsync, but if *anyone* can download and host these
browsers, will their configuration support rsync? I want to donate
bandwidth, but I'm on a shared hosting plan. My plan includes ftp, but
not rsync.

> > Does anyone else think this is necessary?
>
> possibly ...

It might not just be "bad people" either -- a robot might index the
link, for example. (Well, maybe a bad robot that ignores robots.txt.)

Beau