Distributed bandwidth (was RE: [Theforum] Re: [Content] Mail Alert! - Sat Jun 22 31226)

Beau Hartshorne beau at members.evolt.org
Mon Jun 24 11:12:03 CDT 2002

> I would expect as many people as possible to use the most efficient
> mirroring tool available - that is rsync.  Whether you are mirroring 2
> megabytes, or 2 terabytes, rsync just rocks.  It can be secured over
> too.  I would much rather see people using rsync if they can, as
opposed to
> constantly wget'ing stuff, or - god forbid - manually downloading
> Automating the mirroring process also helps us on the backend - we
only need
> to add a file once to beo, and the curator(s) will automatically get
> file they next time they connect.  If there are no changes, they don't
> anything.  They don't even do anything like HEAD checks to see if the
> has changed - rsync tells them so.
> I know i'm coming off as a bit arrogant sounding saying that, but
> really really is the way to go imho.  I maintain a few mirrors here at
> and one of them is an FTP mirror, maintained over FTP.  To fetch the
> on a 2.2gig dataset takes a minute or so to download.  When I rsync
the same
> dataset (albeit locally), I can see the amount of change data changing
> is minimal compared to that transferred during the FTP session.  (did
> make sense? :)
> > [possible bandwidth misuse of mirrors by bad people snippage]
> >
> > The only solution I can think of is this: get volunteers to check
> > referrers, and only send files to anyone referred by evolt.org. Is
> > something like that possible?
> Yup, but it really does depend on the config of the curators kit -
once we
> start getting loads of people mirroring bits or all of beo, we're
> away from the controlled environment that beo represents, and letting
> a large amount of heterogenous services.  Some may be on Linux, some
> win2K, Tru64, Solaris, etc etc, some may be Apache, others IIS, Roxen,
> etc.
> So YMMV.

Yeah, so how many of these platforms can run rsync?

At: ftp://rsync.samba.org/pub/rsync/binaries/

I see: AIX, BSDi, debian-i386, HPUX, IRIX, linuxppc, redhad, SCO,
Solaris, Sunos, SVR4.

You've got me sold on rsync, but if *anyone* can download and host these
browsers, will their configuration support rsync? I want to donate
bandwidth, but I'm on a shared hosting plan. My plan includes ftp, but
not rsync.

> > Does anyone else think this is necessary?
> possibly ...

It might not just be "bad people" either -- a robot might index the
link, for example. (Well, maybe a bad robot that ignores robots.txt.)


More information about the theforum mailing list